100 Days of Code with Apache Kafka®

About the 100-Day Challenge

100 Days Of Code is a challenge that helps developers build strong coding habits with a self-directed learning path. This page is tailored to developers interested in event streaming who want to use the 100 Days Of Code model as a way to learn about Apache Kafka®.

The rules are:

  1. Start hands-on learning Kafka for at least one hour every day for the next 100 days
  2. Share your progress every day on social media (Twitter, LinkedIn—wherever you want) and include #100DaysOfCode @ApacheKafka

Commit to the Challenge

If you decide to commit to the 100 Days Of Code challenge for Apache Kafka, share your commitment to the world, making yourself accountable to the community. Tweet to commit to the challenge!

#100DaysOfCode--graphic

Where to Start

Define your goals: what you want to learn and what you want to build. If you are driven to learn new technologies, you could still choose something to build so you have a forcing function to use those technologies. If you are driven to work on a particular project, figure out what you need to learn to accomplish it and then go learn it.

Then create your 100-day plan. If you would like suggestions on Kafka educational material, see the 100-day learning journey at the bottom of this page.

For a sandbox environment for learning Kafka, without having to run your own Kafka cluster, check out Confluent Cloud (get $100 of free usage with the promo code 100DAYSKAFKA).

During the Challenge

Join the Confluent community, ask all your Kafka-related questions, find answers, and encourage others who are doing the 100 Days Of Code challenge for Apache Kafka.

Tweet your progress daily. As you develop your own applications, upload your code (to GitHub for example) and include a link in your tweet.

100-Day Learning Journey

We absolutely encourage you to follow your own, self-directed 100-day learning journey in such a way that it covers the technologies you want to learn and helps you be successful with your projects. For Kafka-specific material, refer to Confluent Developer, search online, or pull ideas from the example 100-day journey below. Tip: we do not recommend making a rigid schedule—just use the quick starts, tutorials, courses, and other resources as you need them, and let your 100 Days Of Code challenge be a creative journey. Kafka may be your entire focus for the 100 days, but more likely than not, you also want to become more proficient in the broader ecosystem: microservices, or serverless, or event sourcing, or programming languages like Python, Go, or .NET, or analytics, or web development, or databases, SQL or GraphQL—so feel free to augment your learning with the other technical areas that you are interested in.

Introduction

DayResourceDescription
1Quick StartThe simplest way to get started with Apache Kafka: Write your first data record and read it in real time.
2EventsWe live in a world of events—payments, website clicks, temperature readings, and more. Learn how to process those events in an event streaming platform.
3TopicsIn Kafka, all events are stored in topics—ordered, immutable, append-only logs. Learn how to create and work with Kafka topics.
4PartitionsKafka topics are made up of partitions, which can be spread across multiple brokers in a cluster. Learn how partitions can help you to increase throughput and scale your applications.
5BrokersKafka brokers are fast and easy to scale, due in large part to their simplicity. Learn a few basics about brokers to see why they scale so well.
6ProducersProducers, part of the Kafka client library, make it easy to send events to Kafka topics. They handle things like partition assignment, serialization, compression, and more.
7ConsumersConsumers allow us to read events from Kafka topics so we can use them in our applications. Consumer groups make it easy to share the workload across multiple instances of our application.
8Kafka ConnectKafka Connect is a no-code integration API provided as part of Kafka. Stream data out of other systems like databases and MQs into Kafka, and from Kafka down to other systems like NoSQL stores and more!
9Kafka StreamsKafka Streams is a core part of the Kafka ecosystem and provides a Java library for doing stateful stream processing. (Java not your thing? Check out ksqlDB!)
10ksqlDBksqlDB is a specialized kind of database that is optimized for stream processing applications, uses SQL, and exposes a REST interface to applications, which can submit new stream processing jobs to run and query the results.
11KRaftKafka is replacing ZooKeeper with KRaft, which will make Kafka lighter, faster, and much more scalable. It's still in preview, but you can take it for a spin.

Languages and CLIs

DayResourceDescription
12PythonPython has become the lingua franca for machine learning applications, and Kafka is being used in a growing number of ML pipelines. See how you can build Kafka clients with Python.
13GoDocker, Kubernetes, and other household names are built with Go, but Go is also a great language for building event-driven microservices. Get started with this helpful guide.
14.NETA rich .NET client library allows you to build Kafka clients in your favorite .NET language. This C# guide will help you get started.
15Node.jsNode.js opened the backend world to JavaScript developers, and this Node quick start guide will help you use your mad JS skills to build powerful Kafka clients.
16First Producer Application (Java)Dig in and build a producer application in Java. This tutorial provides everything you need and some great learning along the way.
17Producer CallbacksThe Producer.send() method returns a Future, which you can use to asynchrously respond to completed send requests, but you can also use a callback function. This tutorial shows you how.
18First Consumer Application (Java)Now that you have events being produced to Kafka, this step-by-step tutorial shows how to build a consumer application to process these events.
19Spring BootIf you love developing with Spring Boot, run this Hello World Spring Boot client example that writes messages to Kafka and reads them back.
20C/C++Feeling bold? This C/C++ quick start guide will show you how to build Kafka clients using the librdkafka library.
21CLI BasicsThe command line is a developer's best friend, even when using Kafka! Today, use the Confluent Cloud CLI to produce and consume JSON events from a new topic, and learn about record keys and values.
22CLI with (De)serializersUse the Confluent Cloud CLI to produce and consume records formatted in Apache Avro. Using the Schema Registry, register a new schema, then read and write records directly from the terminal.
23Reading from Specific OffsetRecords in Kafka are immutable and stored at specific offsets. Using CLI commands, you can read records from a specific topic partition and offset.
24kcatkcat (formerly known as kafkacat) is a popular utility for working with Kafka. Use kcat to produce and consume records with keys and values directly on the command line.
25REST ProxyHTTP is a popular way of interacting with remote services. Confluent's REST API provides an interface for working with Kafka from any language or platform, and this quick start will help you get going.

Schemas

DayResourceDescription
26Schema RegistryWhat are schemas in a Kafka application? Why should you care? How do you work with schemas and manage them within your system? Schema Registry has all the answers...
27Why Avro?Apache Avro is the de facto standard serialization format for Kafka. But what is it, how does it work, and what features make it ideal for evolving event systems?
28Avro and Schema Registry: JavaDeveloping a Kafka client application in Java? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers.
29Avro and Schema Registry: PythonDeveloping a Kafka client application in Python? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers.
30Avro and Schema Registry: REST ProxyDeveloping a Kafka REST client? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers.
31Avro and Schema Registry: Spring BootDeveloping a Kafka client application with Spring Boot? You don't have to include a data schema with all your messages. Use Schema Registry and ensure compatibility between your readers and writers.
32ProtobufApache Avro is the standard serialization format for Kafka, but it's not the only one. Fans of Protobuf are equally well supported. Here's a walkthrough using Google's favorite serializer.
33Broker-Side Schema ValidationHaving a schema for your data isn't always mandatory, but you may want to guarantee that new data adheres to a certain format. With broker-side schema validation, you can enforce schema-on-write guarantees for your system.
34Schema Compatibility EvolutionTimes change, systems grow, and schemas must evolve to work with data old and new. Look at how Avro helps you describe data over its lifetime and how to safely migrate from one schema to another.
35Schema Evolution with ksqlDB MigrationsSchema migrations are a standard part of the modern database-development toolkit, and ksqlDB is no exception. Learn how to create, evolve, and manage schemas in this walkthrough guide.

Stream Processing with Kafka Streams

DayResourceDescription
36Diving into Kafka StreamsSo you've developed applications with Kafka producer and consumer clients, but you're ready to get into more sophisticated microservices? Kick off with this in-depth Kafka Streams course.
37Basic OperationsManipulate data in real time with Kafka Streams. Learn about stream processing topologies—directed acyclic graphs (DAGs)—and get hands on with basic operations to filter and modify values.
38KTableLearn about KTables, for which newly arriving records are updates to previous ones with the same key. In this exercise, you'll learn how KTables only maintain the latest version for any given key.
39SerializationKafka brokers store keys and values as byte arrays, but you need to work with concrete objects. Learn how to use SerDes in Kafka Streams to convert bytes into objects and back into bytes.
40Joining DataSometimes you need to combine event streams to get the complete answer your business problem demands. In today's lesson, you'll get hands on with stream-stream and stream-table joins.
41Stateful OperationsLearn about the importance of stateful operations in Kafka Streams. State is necessary for computing results that take previous events into account.
42Counting a Stream of EventsThe count() operator is a simple yet powerful stateful operation in Kafka Streams. Learn how to put it into action.
43Merging Many Streams into OneLearn how to take multiple event streams and merge them into one stream so you can apply the common logic to all events.
44Choosing an Output Topic DynamicallyLearn how to route records to different topics at runtime in a Kafka Streams application, similar to a "topic exchange."
45Transforming a Stream of EventsLearn how to transform objects in an event stream into different types in this Kafka Streams lesson.
46WindowingStateful operations are a powerful tool in Kafka Streams, but sometimes you'd like to have the results segmented by time. See how to achieve just that by applying windowing to your stateful processors.
47Session WindowsMost of the windowing options in ksqlDB and Kafka Streams are time driven. Learn about session windows, which can grow to an arbitrary size determined by user activity.
48Time ConceptsRecord timestamps drive the behavior of a ksqlDB and Kafka Streams application. Learn how this works in this lesson.
49TestingTesting is an important part of development, and Kafka Streams is no exception. You can test your Kafka Streams application using the Topology Test Driver, which obviates the need for a real Kafka broker.

Stream Processing with ksqlDB

DayResourceDescription
50Diving into ksqlDBFind out why SQL + Apache Kafka = 🥰. With ksqlDB, you can use SQL to create stream processing pipelines and materialized views on data in Kafka.
51Administering ksqlDBYou can work with ksqlDB however you prefer—whether using a web interface, a CLI, or the REST API directly from your application code.
52Creating, Importing, and ExportingYou can create ksqlDB objects on existing Kafka topics, populate new ones with INSERT syntax, or configure connectors to directly stream data into and out of other systems.
53Filtering DataWith ksqlDB, you can easily filter streams of data in Kafka using SQL predicates. For example, SELECT * FROM ORDERS WHERE STATUS='SHIPPED' creates a new stream of only shipped orders.
54Lookups and JoinsStreams and tables don't live in isolation—they often need to be referenced and joined with others. ksqlDB supports joins using the standard SQL JOIN syntax.
55ksqlDB Java ClientHarness the power of ksqlDB from your Java application with the lightweight Java client. This is a great option if you don't need the full power of Kafka Streams.
56Materialized CacheEvents in Kafka topics can be materialized into state, and using ksqlDB, you can build materialized views to query directly from your application.
57Converting FormatsIf you need to convert messages in Kafka from one serialization format to another, you can do this in ksqlDB with a single SQL statement—CSV to Avro, JSON to Protobuf, or any other permutation you need.
58Advanced Querying with Lambda FunctionsLambda functions in ksqlDB allow you to use filters, transformations, and reductions on events in ARRAY and MAP fields in Kafka topics.
59Splitting Streams into SubstreamsWhen you want to split a stream of events in a Kafka topic, ksqlDB is the perfect tool for the job. With the WHERE clause, you can use standard SQL to route events to different streams as needed.
60GraphQL and ksqlDBHow do GraphQL and ksqlDB intersect? Learn how they can work together, and get started quickly with a REST API on ksqlDB to dynamically create the GraphQL schema and the resolvers.

Microservices

DayResourceDescription
61Persisting EventsIn the first exercise of this demo, kick off a microservices implementation in Java. First, you'll write a service that produces HTTP requests to Apache Kafka as events.
62Event-Driven ApplicationsEvent-driven microservices coordinate by exchanging events asyncronously. Continuing the exercise from yesterday, write a Java service that validates customer order events from a topic.
63Enriching Streams with JoinsStreams can be enriched with data from other streams or tables through joins. Learn how to enrich data by performing joins with data that is updated continuously and concurrently.
64Filtering and BranchingFiltering and branching event streams can be used to create new event streams with logically consistent data. Learn how with this exercise in the microservices demo.
65Stateful OperationsStateful aggregations compute single values from multiple inputs. Use Kafka Streams to create insightful aggregations on continuous event streams.
66State StoresIf you're processing streams of data in real time, explore using in-app state stores to remember recently received input records, track rolling aggregates, de-duplicate input records, etc.
67Event-Driven MicroservicesSee how ksqlDB can greatly simplify the building of a microservices architecture by isolating complex stateful operations inside a managed runtime environment.

Data Pipelines

DayResourceDescription
68Introduction to Data PipelinesETL is dead…long live ETL! With streaming data pipelines, you can have scalable data integration with low latency, getting your data where you need it when you want it.
69Integrating Kafka with External SystemsKafka Connect provides scalable streaming integration into and out of Kafka and other systems—RDBMS, MQs, flat files, NoSQL stores, and more!
70Creating a Data Generator with Kafka ConnectA nice example of a Kafka Connect source is the Datagen connector. You can use it to generate test data in various schemas directly into Kafka topics.
71Kafka Connect SerDesKafka Connect handles the (de)serialization of your messages using pluggable converters for Avro, Protobuf, JSON Schema, and more.
72Ingesting Data from Databases with Change Data Capture (CDC)One of the most common integrations in Kafka is with an RDBMS, which you can easily do with Kafka Connect. With CDC, you can capture all of the events from a database as they happen.
73Troubleshooting Kafka ConnectMake sure your data pipelines are flowing freely with this essential Kafka Connect troubleshooting guide.
74Single Message Transforms (SMTs)Use a Single Message Transforms (SMT) to transform messages as they pass through Kafka Connect. Drop fields, rename objects, change data types, and more using this flexible and powerful component of Kafka.
75JDBC SinkOne of the most popular uses of Kafka Connect is to stream data from Kafka to an RDBMS. Learn more in this video.
76Building a Streaming ETL Solution with Rail DataHere's a real-world data pipeline in action using real-time data feeds about train movements. It includes integration with ActiveMQ, S3, Elasticsearch, PostgreSQL, and Neo4j.
77Building a Streaming ETL Pipeline from MongoDB to SnowflakeCheck out this demo of getting data from MongoDB to Snowflake using Apache Kafka and easy-to-run source and sink connectors.

Event Sourcing

DayResourceDescription
78Messaging as the Single Source of TruthLook deeper into the motivations for choosing an event-sourced architecture, beginning with this 10,000-foot view of event streaming.
79Thinking in EventsOne of the most important skills you'll need to be successful with Kafka is analyzing problems and solutions in terms of events. Begin the event sourcing course to crack open that mindset.
80Storing Data as EventsDoes event sourcing support CRUD, or is it in conflict with it? Continuing from yesterday's shift in mindset, we'll look at how to manage state in a world without updates or deletes.
81Why Store Events?The big question: Why store data as events? What are the benefits?
82Command Query Responsibility Segregation (CQRS)No event sourcing journey is complete without looking at CQRS. In this lesson, we'll break down this core pattern.
83Trying Out Event SourcingTo wrap up the event sourcing course, write some code in a simple, hands-on walkthrough.

Performance and Observability

DayResourceDescription
84Explore Kafka InternalsLatency is the amount of time it takes to produce and consume a message with Confluent Cloud. Learn about what you can do to optimize latency for your Kafka application.
85Producer ScenariosThe Kafka producer is responsible for sending records to the broker. See how to monitor the producer client for optimal performance and troubleshooting.
86Consumer ScenariosContinue to learn about monitoring clients. Here the focus is on the KafkaConsumer.
87Failure to Create a New PartitionEspecially useful if your Kafka cluster has limits on the number of partitions you can create, here you'll learn how to monitor partition counts for your cluster.
88Request Rate LimitsCertain cluster types in Confluent Cloud have limits on client requests per second. Today, you'll learn how to monitor the request rate of your application.
89AvailabilityLearn about various settings you can use to optimize your Kafka application for high availability and quick recovery from failures.
90DurabilityData durability means you have confidence that your messages won't be lost. Learn about the important settings for ensuring high data durability.
91LatencyLatency is the amount of time that it takes to produce and consume a message with Confluent Cloud. Learn about what you can do to optimize latency for your Kafka application.
92ThroughputThroughput is about moving as much data as you can in a given amount of time. In today's lesson, you'll learn concepts that enable you to maximize the throughput of your Kafka applications.
93Parallel ConsumerLearn how to process messages in parallel with the Parallel Apache Kafka client wrapper, which increases throughput without having to increase the number of partitions.
94Metrics APIAccess to actionable metrics is critical for getting ahead of any performance issues. Learn about the Confluent Cloud Metrics API and practice running some queries.

Cloud Operations

DayResourceDescription
95Using API KeysConfluent Cloud uses API keys to manage access and authentication to cloud services. Use the ccloud CLI and Confluent Cloud Console to manage credentials and associate them with specific services.
96Confluent Cloud CLIThe Confluent Cloud CLI allows you to control your environment from the command line. Manage environments, clusters, topics, ACLs, and more, all from a single downloadable, standalone binary.
97Confluent Cloud APIHTTP APIs are useful building blocks for automating critical cloud services. Use the growing set of Confluent Cloud management APIs to write a simple procedure.
98Audit LoggingData governance is critical to ensuring secure, available, and accurate data systems. Learn how to use Confluent Audit Logging to capture authentication and authorization activity with familiar tools.
99Cluster Linking for Topic Data SharingMulti-cluster strategies can help address a variety of complex architectural needs. Learn how Cluster Linking simplifies these designs with byte-for-byte topic replication for your Confluent Cloud clusters.
100Stream LineageBuild some event streaming components, and learn how Stream Lineage provides an interactive graphical UI describing the relationships between them.

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.