Introduction
David Anderson
Software Practice Lead
Apache Flink® 101
About This Course
This course is an introduction to Apache Flink, focusing on its core concepts and architecture. Learn what makes Flink tick, and how it handles some common use cases.
What is Flink?
Today's consumers have come to expect timely and accurate information from the companies they do business with. Whether it's being alerted that someone just used your credit card to rent a car in Prague, or checking on the balance of your mobile data plan, it's not good enough to learn about yesterday's information today. We all expect the companies managing our data to provide fully up-to-the-moment reporting.
Apache Flink is a battle-hardened stream processor widely used for demanding applications like these. Its performance and robustness are the result of a handful of core design principles, including a share-nothing architecture with local state, event-time processing, and state snapshots (for recovery). Through a combination of videos and hands-on exercises, this course brings these core principles to life.
Common Flink Use Cases
Common use cases include data analytics, fraud detection, billing, business process monitoring, rule-based alerting, etc.
How Flink Works
Flink is a powerful system with many components, but once you understand the fundamentals presented in this course, and how they fit together – streams, state, time, and snapshots – learning the details will become much easier.
The hands-on exercises in this course use Flink SQL to illustrate and clarify how Flink works. The focus is on learning about Flink, using the SQL you already know.
What You’ll Learn in This Course
- What Apache Flink is, and why you might use it
- What stream processing is, and how it differs from batch processing
- Flink’s runtime architecture
- How to use Flink and Kafka together
- How to use Flink SQL: tables, windows, event time, watermarks, and more
- Stateful stream processing
- How watermarks support event time operations
- How Flink uses snapshots (checkpoints) for fault tolerance
Intended Audience
Anyone who knows the basics of Kafka and SQL who wants to understand what Flink is and how it works.
Prerequisites
- Required knowledge:
- This course assumes some basic familiarity with Kafka and SQL. If you understand what producers and consumers are, and can explain what GROUP BY does, that’s good enough.
- Required setup:
- A local Docker installation.
Resources
To learn more about Kafka, see Kafka 101.
Building Flink Applications in Java is a companion course to this one, and a great way to learn more about the practical side of Flink application development.
Length
- Approximately 2-3 hours
Staff
David Anderson (Course Author)
David has been working as a data engineer since long before that job title was invented. He has worked on recommender systems, search engines, machine learning pipelines, and BI tools, and has been helping companies adopt stream processing and Apache Flink since 2016. David is an Apache Flink committer, and works at Confluent as a Software Practice Lead.
- Previous
- Next
Use the promo codes FLINK101 & CONFLUENTDEV1 to get $25 of free Confluent Cloud usage and skip credit card entry.
Be the first to get updates and new content
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.
Introduction
Hi, I'm David Anderson with Confluent, here to tell you all about Apache Flink. We'll look together at why Flink is interesting, and how you can use it to build real-time data products. Along the way, I'll explain the big ideas on which Flink is based, and show you around under the hood so you'll understand how Flink works. Apache Flink is a battle-hardened stream processor widely used for demanding real-time applications. Its performance and robustness are the result of a handful of core design principles, including a share-nothing architecture featuring local state, event-time processing, and state snapshots for recovery. Through a combination of videos and hands-on exercises, this course brings these core principles to life. The focus of this course is going to be on the 4 Big Ideas that form the foundation for Apache Flink, which are streaming, state, time, and the use of state snapshots for fault tolerance and failure recovery. Understanding how Flink's runtime is organized around these four concepts, and how they are interrelated, is the key that unlocks Apache Flink. In the first four sections, we'll look together at streams and stream processing, and do so from several different perspectives. In the second half, we will look at each of the other three big ideas. By the end of the course you'll know enough to be able to implement some common use cases, and you'll understand what's going on inside of Flink when it is running your applications. Most of the modules in this course have a hands-on exercise that reinforces and expands upon the information in these videos. You'll find these exercises and other materials for this course on developer.confluent.io. These hands-on exercises all use Flink SQL. The focus will always be on learning about the concepts and architecture of Flink, while taking advantage of the SQL that you already know. And don't worry, you're not expected to be a SQL expert for this course. We're not going to do anything more complicated than aggregating data with GROUP BY. In several of the hands-on exercises, you will be using Flink SQL together with Apache Kafka to produce and consume data on Confluent Cloud. If you haven't already signed up for Confluent Cloud, sign up now so when you need it for the exercises, you'll be ready. And be sure to use the promo code in the description: it provides enough free credit to do all of the exercises for this course.