100 Days of Code with Apache Kafka®
About the 100-Day Challenge
100 Days Of Code is a challenge that helps developers build strong coding habits with a self-directed learning path. This page is tailored to developers interested in event streaming who want to use the 100 Days Of Code model as a way to learn about Apache Kafka®.
The rules are:
- Get hands-on learning Kafka for at least one hour, every day, for the next 100 days
- Share your progress every day on social media (Twitter, LinkedIn—wherever you want) and include
#100DaysOfCode @ApacheKafka
Where to Start
Tweet your commitment to the challenge, making yourself accountable to the community.
Then setup a sandbox environment for learning Kafka. You don't have to run your own Kafka cluster, instead get started with Confluent Cloud (get $100 of free usage with the promo code 100DAYSKAFKA).
Define your goals: what you want to learn and what you want to build. If you are driven to learn new technologies, you could still choose something to build so you have a forcing function to use those technologies. If you are driven to work on a particular project, figure out what you need to learn to accomplish it and then go learn it.
Then create your 100-day plan. If you would like suggestions on Kafka educational material, see the 100-day learning journey at the bottom of this page.
During the Challenge
Join the Confluent community, ask all your Kafka-related questions, find answers, and encourage others who are doing the 100 Days Of Code challenge for Apache Kafka.
Tweet your progress daily and include links to any related resources. As you develop your own applications, upload your code (to GitHub for example) and include a link in your tweet.
100-Day Learning Journey
We absolutely encourage you to follow your own, self-directed 100-day learning journey in such a way that it covers the technologies you want to learn and helps you be successful with your projects. For Kafka-specific material, refer to Confluent Developer, search online, or pull ideas from the example 100-day journey below. Tip: we do not recommend making a rigid schedule—just use the quick starts, tutorials, courses, and other resources as you need them, and let your 100 Days Of Code challenge be a creative journey. Kafka may be your entire focus for the 100 days, but more likely than not, you also want to become more proficient in the broader ecosystem: microservices, or serverless, or event sourcing, or programming languages like Python, Go, or .NET, or analytics, or web development, or databases, SQL or GraphQL—so feel free to augment your learning with the other technical areas that you are interested in.
Introduction
| Day (Click to tweet) | Resource | Description |
|---|---|---|
| 1 | Quick Start | The simplest way to get started with Apache Kafka: Write your first data record and read it in real time. |
| 2 | Events | We live in a world of events—payments, website clicks, temperature readings, and more. Learn how to process those events in an event streaming platform. |
| 3 | Topics | In Kafka, all events are stored in topics—ordered, immutable, append-only logs. Learn how to create and work with Kafka topics. |
| 4 | Partitions | Kafka topics are made up of partitions, which can be spread across multiple brokers in a cluster. Learn how partitions can help you to increase throughput and scale your applications. |
| 5 | Brokers | Kafka brokers are fast and easy to scale, due in large part to their simplicity. Learn a few basics about brokers to see why they scale so well. |
| 6 | Producers | Producers, part of the Kafka client library, make it easy to send events to Kafka topics. They handle things like partition assignment, serialization, compression, and more. |
| 7 | Consumers | Consumers allow us to read events from Kafka topics so we can use them in our applications. Consumer groups make it easy to share the workload across multiple instances of our application. |
| 8 | Kafka Connect | Kafka Connect is a no-code integration API provided as part of Kafka. Stream data out of other systems like databases and MQs into Kafka, and from Kafka down to other systems like NoSQL stores and more! |
| 9 | Kafka Streams | Kafka Streams is a core part of the Kafka ecosystem and provides a Java library for doing stateful stream processing. (Java not your thing? Check out ksqlDB!) |
| 10 | ksqlDB | ksqlDB is a specialized kind of database that is optimized for stream processing applications, uses SQL, and exposes a REST interface to applications, which can submit new stream processing jobs to run and query the results. |
| 11 | KRaft | Kafka is replacing ZooKeeper with KRaft, which will make Kafka lighter, faster, and much more scalable. It's still in preview, but you can take it for a spin. |
Languages and CLIs
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day (Click to tweet) | Resource | Description |
|---|---|---|
| 12 | Python | Python has become the lingua franca for machine learning applications, and Kafka is being used in a growing number of ML pipelines. See how you can build Kafka clients with Python. |
| 13 | Go | Docker, Kubernetes, and other household names are built with Go, but Go is also a great language for building event-driven microservices. Get started with this helpful guide. |
| 14 | .NET | A rich .NET client library allows you to build Kafka clients in your favorite .NET language. This C# guide will help you get started. |
| 15 | Node.js | Node.js opened the backend world to JavaScript developers, and this Node quick start guide will help you use your mad JS skills to build powerful Kafka clients. |
| 16 | First Producer Application (Java) | Dig in and build a producer application in Java. This tutorial provides everything you need and some great learning along the way. |
| 17 | Producer Callbacks | The Producer.send() method returns a Future, which you can use to asynchrously respond to completed send requests, but you can also use a callback function. This tutorial shows you how. |
| 18 | First Consumer Application (Java) | Now that you have events being produced to Kafka, this step-by-step tutorial shows how to build a consumer application to process these events. |
| 19 | Spring Boot | If you love developing with Spring Boot, run this Hello World Spring Boot client example that writes messages to Kafka and reads them back. |
| 20 | C/C++ | Feeling bold? This C/C++ quick start guide will show you how to build Kafka clients using the librdkafka library. |
| 21 | CLI Basics | The command line is a developer's best friend, even when using Kafka! Today, use the Confluent CLI to produce and consume JSON events from a new topic, and learn about record keys and values. |
| 22 | CLI with (De)serializers | Use the Confluent CLI to produce and consume records formatted in Apache Avro. Using the Schema Registry, register a new schema, then read and write records directly from the terminal. |
| 23 | Reading from Specific Offset | Records in Kafka are immutable and stored at specific offsets. Using CLI commands, you can read records from a specific topic partition and offset. |
| 24 | kcat | kcat (formerly known as kafkacat) is a popular utility for working with Kafka. Use kcat to produce and consume records with keys and values directly on the command line. |
| 25 | REST Proxy | HTTP is a popular way of interacting with remote services. Confluent's REST API provides an interface for working with Kafka from any language or platform, and this quick start will help you get going. |
Schemas
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day (Click to tweet) | Resource | Description |
|---|---|---|
| 26 | Schema Registry | What are schemas in a Kafka application? Why should you care? How do you work with schemas and manage them within your system? Schema Registry has all the answers. |
| 27 | Why Avro? | Apache Avro is the de facto standard serialization format for Kafka. But what is it, how does it work, and what features make it ideal for evolving event systems? |
| 28 | Avro and Schema Registry: Java | Developing a Kafka client application in Java? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers. |
| 29 | Avro and Schema Registry: Python | Developing a Kafka client application in Python? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers. |
| 30 | Avro and Schema Registry: REST Proxy | Developing a Kafka REST client? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers. |
| 31 | Avro and Schema Registry: Spring Boot | Developing a Kafka client application with Spring Boot? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers. |
| 32 | Protobuf | Apache Avro is the standard serialization format for Kafka, but it's not the only one. Fans of Protobuf are equally well supported. Here's a walkthrough using Google's favorite serializer. |
| 33 | Broker-Side Schema Validation | Having a schema for your data isn't always mandatory, but you may want to guarantee that new data adheres to a certain format. With broker-side schema validation, you can enforce schema-on-write guarantees for your system. |
| 34 | Schema Compatibility Evolution | Times change, systems grow, and schemas must evolve to work with data old and new. Look at how Avro helps you describe data over its lifetime and how to safely migrate from one schema to another. |
| 35 | Schema Evolution with ksqlDB Migrations | Schema migrations are a standard part of the modern database-development toolkit, and ksqlDB is no exception. Learn how to create, evolve, and manage schemas in this walkthrough guide. |
Stream Processing with Kafka Streams
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 36 | Diving into Kafka Streams | So you've developed applications with Kafka producer and consumer clients, but you're ready to get into more sophisticated microservices? Kick off with this in-depth Kafka Streams course. |
| 37 | Basic Operations | Manipulate data in real time with Kafka Streams. Learn about stream processing topologies—directed acyclic graphs (DAGs)—and get hands on with basic operations to filter and modify values. |
| 38 | KTable | Learn about KTables, for which newly arriving records are updates to previous ones with the same key. In this exercise, you'll learn how KTables only maintain the latest version for any given key. |
| 39 | Serialization | Kafka brokers store keys and values as byte arrays, but you need to work with concrete objects. Learn how to use SerDes in Kafka Streams to convert bytes into objects and back into bytes. |
| 40 | Joining Data | Sometimes you need to combine event streams to get the complete answer your business problem demands. In today's lesson, you'll get hands on with stream-stream and stream-table joins. |
| 41 | Stateful Operations | Learn about the importance of stateful operations in Kafka Streams. State is necessary for computing results that take previous events into account. |
| 42 | Counting a Stream of Events | The count() operator is a simple yet powerful stateful operation in Kafka Streams. Learn how to put it into action. |
| 43 | Merging Many Streams into One | Learn how to take multiple event streams and merge them into one stream so you can apply the common logic to all events. |
| 44 | Choosing an Output Topic Dynamically | Learn how to route records to different topics at runtime in a Kafka Streams application, similar to a "topic exchange." |
| 45 | Transforming a Stream of Events | Learn how to transform objects in an event stream into different types in this Kafka Streams lesson. |
| 46 | Windowing | Stateful operations are a powerful tool in Kafka Streams, but sometimes you'd like to have the results segmented by time. See how to achieve just that by applying windowing to your stateful processors. |
| 47 | Session Windows | Most of the windowing options in ksqlDB and Kafka Streams are time driven. Learn about session windows, which can grow to an arbitrary size determined by user activity. |
| 48 | Time Concepts | Record timestamps drive the behavior of a ksqlDB and Kafka Streams application. Learn how this works in this lesson. |
| 49 | Testing | Testing is an important part of development, and Kafka Streams is no exception. You can test your Kafka Streams application using the Topology Test Driver, which obviates the need for a real Kafka broker. |
Stream Processing with ksqlDB
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 50 | Diving into ksqlDB | Find out why SQL + Apache Kafka = 🥰. With ksqlDB, you can use SQL to create stream processing pipelines and materialized views on data in Kafka. |
| 51 | Administering ksqlDB | You can work with ksqlDB however you prefer—whether using a web interface, a CLI, or the REST API directly from your application code. |
| 52 | Creating, Importing, and Exporting | You can create ksqlDB objects on existing Kafka topics, populate new ones with INSERT syntax, or configure connectors to directly stream data into and out of other systems. |
| 53 | Filtering Data | With ksqlDB, you can easily filter streams of data in Kafka using SQL predicates. For example, SELECT * FROM ORDERS WHERE STATUS='SHIPPED' creates a new stream of only shipped orders. |
| 54 | Lookups and Joins | Streams and tables don't live in isolation—they often need to be referenced and joined with others. ksqlDB supports joins using the standard SQL JOIN syntax. |
| 55 | ksqlDB Java Client | Harness the power of ksqlDB from your Java application with the lightweight Java client. This is a great option if you don't need the full power of Kafka Streams. |
| 56 | Materialized Cache | Events in Kafka topics can be materialized into state, and using ksqlDB, you can build materialized views to query directly from your application. |
| 57 | Converting Formats | If you need to convert messages in Kafka from one serialization format to another, you can do this in ksqlDB with a single SQL statement—CSV to Avro, JSON to Protobuf, or any other permutation you need. |
| 58 | Advanced Querying with Lambda Functions | Lambda functions in ksqlDB allow you to use filters, transformations, and reductions on events in ARRAY and MAP fields in Kafka topics. |
| 59 | Splitting Streams into Substreams | When you want to split a stream of events in a Kafka topic, ksqlDB is the perfect tool for the job. With the WHERE clause, you can use standard SQL to route events to different streams as needed. |
| 60 | GraphQL and ksqlDB | How do GraphQL and ksqlDB intersect? Learn how they can work together, and get started quickly with a REST API on ksqlDB to dynamically create the GraphQL schema and the resolvers. |
Microservices
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 61 | Persisting Events | In the first exercise of this demo, kick off a microservices implementation in Java. First, you'll write a service that produces HTTP requests to Apache Kafka as events. |
| 62 | Event-Driven Applications | Event-driven microservices coordinate by exchanging events asyncronously. Continuing the exercise from yesterday, write a Java service that validates customer order events from a topic. |
| 63 | Enriching Streams with Joins | Streams can be enriched with data from other streams or tables through joins. Learn how to enrich data by performing joins with data that is updated continuously and concurrently. |
| 64 | Filtering and Branching | Filtering and branching event streams can be used to create new event streams with logically consistent data. Learn how with this exercise in the microservices demo. |
| 65 | Stateful Operations | Stateful aggregations compute single values from multiple inputs. Use Kafka Streams to create insightful aggregations on continuous event streams. |
| 66 | State Stores | If you're processing streams of data in real time, explore using in-app state stores to remember recently received input records, track rolling aggregates, de-duplicate input records, etc. |
| 67 | Event-Driven Microservices | See how ksqlDB can greatly simplify the building of a microservices architecture by isolating complex stateful operations inside a managed runtime environment. |
Data Pipelines
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 68 | Introduction to Data Pipelines | ETL is dead…long live ETL! With streaming data pipelines, you can have scalable data integration with low latency, getting your data where you need it when you want it. |
| 69 | Integrating Kafka with External Systems | Kafka Connect provides scalable streaming integration into and out of Kafka and other systems—RDBMS, MQs, flat files, NoSQL stores, and more! |
| 70 | Creating a Data Generator with Kafka Connect | A nice example of a Kafka Connect source is the Datagen connector. You can use it to generate test data in various schemas directly into Kafka topics. |
| 71 | Kafka Connect SerDes | Kafka Connect handles the (de)serialization of your messages using pluggable converters for Avro, Protobuf, JSON Schema, and more. |
| 72 | Ingesting Data from Databases with Change Data Capture (CDC) | One of the most common integrations in Kafka is with an RDBMS, which you can easily do with Kafka Connect. With CDC, you can capture all of the events from a database as they happen. |
| 73 | Troubleshooting Kafka Connect | Make sure your data pipelines are flowing freely with this essential Kafka Connect troubleshooting guide. |
| 74 | Single Message Transforms (SMTs) | Use a Single Message Transforms (SMT) to transform messages as they pass through Kafka Connect. Drop fields, rename objects, change data types, and more using this flexible and powerful component of Kafka. |
| 75 | JDBC Sink | One of the most popular uses of Kafka Connect is to stream data from Kafka to an RDBMS. Learn more in this video. |
| 76 | Building a Streaming ETL Solution with Rail Data | Here's a real-world data pipeline in action using real-time data feeds about train movements. It includes integration with ActiveMQ, S3, Elasticsearch, PostgreSQL, and Neo4j. |
| 77 | Building a Data Pipeline for Analyzing Clickstream Data | Run a recipe that demonstrates how to optimize webpages and determine the effectiveness of web presence by better understanding users’ click activity and navigation patterns |
Event Sourcing
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 78 | Messaging as the Single Source of Truth | Look deeper into the motivations for choosing an event-sourced architecture, beginning with this 10,000-foot view of event streaming. |
| 79 | Thinking in Events | One of the most important skills you'll need to be successful with Kafka is analyzing problems and solutions in terms of events. Begin the event sourcing course to crack open that mindset. |
| 80 | Storing Data as Events | Does event sourcing support CRUD, or is it in conflict with it? Continuing from yesterday's shift in mindset, we'll look at how to manage state in a world without updates or deletes. |
| 81 | Why Store Events? | The big question: Why store data as events? What are the benefits? |
| 82 | Command Query Responsibility Segregation (CQRS) | No event sourcing journey is complete without looking at CQRS. In this lesson, we'll break down this core pattern. |
| 83 | Trying Out Event Sourcing | To wrap up the event sourcing course, write some code in a simple, hands-on walkthrough. |
Performance and Observability
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 84 | Explore Kafka Internals | Latency is the amount of time it takes to produce and consume a message with Confluent Cloud. Learn about what you can do to optimize latency for your Kafka application. |
| 85 | Producer Scenarios | The Kafka producer is responsible for sending records to the broker. See how to monitor the producer client for optimal performance and troubleshooting. |
| 86 | Consumer Scenarios | Continue to learn about monitoring clients. Here the focus is on the KafkaConsumer. |
| 87 | Failure to Create a New Partition | Especially useful if your Kafka cluster has limits on the number of partitions you can create, here you'll learn how to monitor partition counts for your cluster. |
| 88 | Request Rate Limits | Certain cluster types in Confluent Cloud have limits on client requests per second. Today, you'll learn how to monitor the request rate of your application. |
| 89 | Availability | Learn about various settings you can use to optimize your Kafka application for high availability and quick recovery from failures. |
| 90 | Durability | Data durability means you have confidence that your messages won't be lost. Learn about the important settings for ensuring high data durability. |
| 91 | Latency | Latency is the amount of time that it takes to produce and consume a message with Confluent Cloud. Learn about what you can do to optimize latency for your Kafka application. |
| 92 | Throughput | Throughput is about moving as much data as you can in a given amount of time. In today's lesson, you'll learn concepts that enable you to maximize the throughput of your Kafka applications. |
| 93 | Parallel Consumer | Learn how to process messages in parallel with the Parallel Apache Kafka client wrapper, which increases throughput without having to increase the number of partitions. |
| 94 | Metrics API | Access to actionable metrics is critical for getting ahead of any performance issues. Learn about the Confluent Cloud Metrics API and practice running some queries. |
Cloud Operations
Prerequisite: Complete the day 1 quick start to create a sandbox environment for learning Kafka.
| Day | Resource | Description |
|---|---|---|
| 95 | Using API Keys | Confluent Cloud uses API keys to manage access and authentication to cloud services. Use the Confluent CLI and Confluent Cloud Console to manage credentials and associate them with specific services. |
| 96 | Confluent CLI | The Confluent CLI allows you to control your environment from the command line. Manage environments, clusters, topics, ACLs, and more, all from a single downloadable, standalone binary. |
| 97 | Confluent Cloud API | HTTP APIs are useful building blocks for automating critical cloud services. Use the growing set of Confluent Cloud management APIs to write a simple procedure. |
| 98 | Audit Logging | Data governance is critical to ensuring secure, available, and accurate data systems. Learn how to use Confluent Audit Logging to capture authentication and authorization activity with familiar tools. |
| 99 | Cluster Linking for Topic Data Sharing | Multi-cluster strategies can help address a variety of complex architectural needs. Learn how Cluster Linking simplifies these designs with byte-for-byte topic replication for your Confluent Cloud clusters. |
| 100 | Stream Lineage | Build some event streaming components, and learn how Stream Lineage provides an interactive graphical UI describing the relationships between them. |