Building Apache Flink Applications in Java

About This Course

Peel away the surface of a modern system and you’ll often find a mountain of data being processed. It wasn’t always this way. Not long ago applications were smaller and the data tended to be static. Processing was performed on demand whenever a query was made. If the application required upfront computation, it was done with a batch job running against a relatively small data set.

Today, the data sets have grown to staggering sizes. They are too large for a simple batch job to handle. Meanwhile, users are no longer content to wait hours or even minutes for a batch job to process their data. They want results now. As a result, developers are increasingly turning to distributed streaming solutions to process data in real time.

Apache Flink is a powerful engine built for processing streaming data flows in a distributed environment. Rather than accumulating data into batches to be processed later, Apache Flink allows us to process the data as it happens, applying stateful transformations along the way. This makes it an invaluable tool for today’s streaming needs.

This course will introduce students to Apache Flink through a series of hands-on exercises. Students will build a basic application in Java that will consume a collection of Apache Kafka data streams. The data will be transformed using Flink and pushed back into new Kafka topics. This simple use case will give students many of the tools they need to start building production-grade Apache Flink applications.

Intended Audience

• You are an experienced Java developer who is new to Apache Flink.
• You are curious about real-time data streaming systems.

Course Outline

• Overview
• Datastream Programming
• Setup your Flink environment (Exercise)
• The Flink Job Lifecycle
• Running a Flink Job (Exercise)
• Anatomy of a Stream
• Flink Data Sources
• Creating a Flink Data Source (Exercise)
• Serializers & Deserializers
• Deserializing Messages in Flink (Exercise)
• Transforming Data in Flink
• Flink Data Transformations (Exercise)
• Flink Data Sinks
• Creating a Flink Data Sink (Exercise)
• Closing Remarks

Prerequisites

• You will need a decent knowledge of the Java programming language (Java 11), including classes, functions, lambdas.
• You will need a Java development environment.
  • You can use your local development environment if you have one.
  • A Gitpod configuration will be supplied to create a temporary environment.
• You will need a Confluent Cloud account.
  • If you don't already have one, you will be instructed to create it early in the course.

Length

• 2-3 hours

Staff

Wade Waldron (Course Author)

Wade has been a Software Developer since 2005. He has worked on video games, backend microservices, ETL Pipelines, IoT systems, and more. He is an advocate for Test-Driven Development, Domain-Driven Design, Microservice Architecture, and Event-Driven Systems. Today, Wade works as a Staff Software Practice Lead at Confluent, showing people how to build modern data streaming applications.

LinkedIn