The Confluent Serverless project shows how to integrate various cloud provider Lambda functions to build serverless data streaming applications using Confluent and Apache Kafka®.
Serverless applications represent a new way to develop powerful applications, letting you focus on creating the business logic instead of the infrastructure and resource concerns. You just write, build, and deploy it, leaving the rest to the serverless platform.
Serverless code is invoked when an event occurs—such as when consuming from a Kafka topic. The primary concern is with the development of the application and not on the deployment, hosting, and infrastructure. These variables must still be considered to ensure sufficient processing capabilities but are minimal in comparison to traditional on-premises and self-hosted applications.
Function as a Service (FaaS) solutions enable you to write code contained within a single function and register it as a listener on a specific Kafka topic. Upon consuming a new event, the FaaS provider starts up a function execution to process it and with any other events that may arrive during the function's execution window. FaaS providers traditionally pair this execution model with a “pay-as-you-go” billing model, making it even more attractive to workloads that run only intermittently.
Stateless operations are one form of serverless business logic and are usually fairly simple as they don't need to maintain any state beyond the lifespan of the function call. All data necessary to process the event is contained within the event itself. There are, however, use cases that require information about previous events—for these, you will need state.
Stateful processing involves maintaining state between function calls, including previously processed events and ongoing aggregations. Maintaining state becomes much more common as the complexity of the application grows, so it’s important to use a serverless platform that makes this as simple as possible.
This page contains resources for creating serverless event streaming applications with Confluent Cloud and Lambda functions from separate cloud providers.
For questions, feedback, or to open a discussion on serverless, join the friendly Confluent Community. For more resources like this, be sure to explore all of Confluent Developer, which contains courses, examples, and more.