Introduction

Hi, I'm Tim Berglund with Confluent, and I want to tell you all about Apache Kafka. Kafka has become the universal foundation on which data systems are built. Whether you care about data pipelines and analytics, or you write applications and you've got microservices to connect, or you're just trying to get data from some other system to where you are, Kafka is going to be a foundational part of your world. If you're like me, when you think about data, you're probably inclined to think about data as tables, right? And tables are very good at storing representations of things, things in the world that our software deals with, items in inventory, internet-connected smart cars, users who have signed up for a site, like all kinds of different things.

You're going to see that Kafka encourages you not to think first of things, but of events, things that happen in the world at a specific point in time. Like an item getting sold, that's an event. A driver in a connected smart car using their turn signal. A user clicking somewhere on your site. Those are all events, and events form the backbone of contemporary data systems, and Kafka is the infrastructure the world uses to manage them. And because an event is, by its very nature, a thing that happens at a particular time, Kafka is very focused on processing events in real time. This means that systems built on Kafka tend to do their computation on events as soon as they occur. They don't store them up in tables, or files, or some other structure to be processed later on in a batch. You never put the data over there promising to get back to it later. No, an event happens, you do the work on the event right now.

That's not to say that Kafka can't remember events that happen. It can, and we'll look at how. And this tension between thing and event, well, that's not absolute either. We'll see how Kafka can help you manage the schema of events that you're storing in it. That's very much a thing way of thinking. And we'll see how to put things into Kafka, how to read them out, how events are structured, how Kafka manages storage and replication, all of those mechanics. And we'll look at the differences between Apache Kafka and modern cloud-native Kafka services like Confluent Cloud. And of course, this is just the beginning. I said Kafka is the foundation of modern data systems, and I meant it. After this, there is so much more for you to learn. You're really just getting started.

Many of the world's largest companies use Kafka. Some handle millions of events per second, billions per hour, trillions per day. There are just crazy numbers of companies, like these logos that you see here, using Kafka in the wild. And an entire platform is emerging on top of it. For example, we'll want to perform real-time computation on all these events or do what we call stream processing. We'll need to impose governance on all this event data. We'll need to connect Kafka with other non-Kafka systems. There's an entire ecosystem of tools and components emerging on top of Kafka, one we like to call a data streaming platform. It's taken shape in a real way since Kafka got its start, and it's continuing to grow as we move forward.

And to give you an idea of what you have ahead of you in this course, if you're taking it on Confluent Developer, you should see the list of modules off to the side. But if not, after this, there are 10 more modules covering important topics that get you introduced to this space. Let's get started.