Apache Flink with Java - An Introduction
Wade Waldron
Staff Software Practice Lead
Building Apache Flink Applications in Java
About This Course
Peel away the surface of a modern system and you’ll often find a mountain of data being processed. It wasn’t always this way. Not long ago applications were smaller and the data tended to be static. Processing was performed on demand whenever a query was made. If the application required upfront computation, it was done with a batch job running against a relatively small data set.
Today, the data sets have grown to staggering sizes. They are too large for a simple batch job to handle. Meanwhile, users are no longer content to wait hours or even minutes for a batch job to process their data. They want results now. As a result, developers are increasingly turning to distributed streaming solutions to process data in real time.
Apache Flink is a powerful engine built for processing streaming data flows in a distributed environment. Rather than accumulating data into batches to be processed later, Apache Flink allows us to process the data as it happens, applying stateful transformations along the way. This makes it an invaluable tool for today’s streaming needs.
This course will introduce students to Apache Flink through a series of hands-on exercises. Students will build a basic application in Java that will consume a collection of Apache Kafka data streams. The data will be transformed using Flink and pushed back into new Kafka topics. This simple use case will give students many of the tools they need to start building production-grade Apache Flink applications.
Intended Audience
- You are an experienced Java developer who is new to Apache Flink.
- You are curious about real-time data streaming systems.
Course Outline
- Overview
- Datastream Programming
- Setup your Flink environment (Exercise)
- The Flink Job Lifecycle
- Running a Flink Job (Exercise)
- Anatomy of a Stream
- Flink Data Sources
- Creating a Flink Data Source (Exercise)
- Serializers & Deserializers
- Deserializing Messages in Flink (Exercise)
- Transforming Data in Flink
- Flink Data Transformations (Exercise)
- Flink Data Sinks
- Creating a Flink Data Sink (Exercise)
- Closing Remarks
Prerequisites
- You will need a decent knowledge of the Java programming language (Java 11), including classes, functions, lambdas.
- You will need a Java development environment.
- You can use your local development environment if you have one.
- A Gitpod configuration will be supplied to create a temporary environment.
- You will need a Confluent Cloud account.
- If you don't already have one, you will be instructed to create it early in the course.
Length
- 2-3 hours
Staff
Wade Waldron (Course Author)
Wade has been a Software Developer since 2005. He has worked on video games, backend microservices, ETL Pipelines, IoT systems, and more. He is an advocate for Test-Driven Development, Domain-Driven Design, Microservice Architecture, and Event-Driven Systems. Today, Wade works as a Staff Software Practice Lead at Confluent, showing people how to build modern data streaming applications.
- Previous
- Next
Use the promo codes FLINKJAVA101 & CONFLUENTDEV1 to get $25 of free Confluent Cloud usage and skip credit card entry.
Be the first to get updates and new content
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.
Apache Flink with Java - An Introduction
Hi, I'm Wade from Confluent. In this course, I'm going to show you how to create a series of small Flink Jobs using Java. Flink can be a very powerful tool for helping you build real-time data streams. But, if you aren't sure where to begin, it can seem scary at first. This course will take some of the guesswork out of the process by walking you through building Flink applications from the ground up. We'll be taking an iterative approach so that each step feels small and manageable. By the end, you will have three fully functioning Flink jobs that cover many of the basics you will need. Throughout this course, we'll be teaching you how to: Run a job Consume data from a Kafka topic Serialize and deserialize Java objects, or POJOs as they are often called. Transform and filter data using a variety of operators. Produce data to a Kafka topic. Combine multiple incoming data sources Aggregate data streams using time windows And manage stateful operations. The course will introduce you to the Flink Data Stream API in Java through a series of hands-on exercises. These exercises will allow you to produce and consume data through Confluent Cloud using Flink. If you haven't already signed up for Confluent Cloud, sign up now so when your first exercise asks you to log in, you're ready. Be sure to use the promo code when signing up to get the free usage it provides. If you aren't already on Confluent Developer, head there now using the link in the video description to access the rest of this course and its hands-on exercises.