Course overview

The evolution of data engineering

The future of data engineering is in streaming! This may seem counterintuitive. Traditionally, data engineering has focused on supplying business data to data scientists. A data scientist might use it to generate analytics for reports that cover an extended period. Why should that report be created in real-time when it can be executed as needed?

The need for real-time processing

However, the world is changing faster every day. Businesses must be prepared to make rapid shifts when necessary to adapt to the new environment. These adaptations will happen in real-time and faster than a human being can achieve. AI models will perform adaptations in real-time as needs arise. Imagine financial systems that can react to changes in the market as they happen, rather than waiting minutes, hours, or even longer to adjust.

The shift to streaming systems

To meet these urgent needs, Data Engineers will transition from batch-driven processes to streaming systems that can adapt more quickly. That's not to suggest that there won't be a need for monthly or quarterly reports, but the major business drivers will be real-time data streams.

Course focus

This course will guide you in shifting workloads from a more traditional batch-driven architecture into a real-time system based on technologies like Apache Kafka® and Apache Flink®. You'll learn how these streaming systems can improve not just the speed at which insights can be delivered but also the reliability.

The future of data engineering

The role of a Data Engineer is evolving. In the future, the Data Stream Engineer will be critical to ensuring that businesses can remain agile, adapt, and even thrive in a rapidly changing world.

Audience

• Data engineers who are experienced in batch processing but want to move into stream processing.
• Data engineers who are new to their careers and need to understand how streaming could impact them.

Prerequisites

Required knowledge:

• Data Stores (Databases, Object Stores like S3)
• SQL

Required setup:

• Local Docker
• Confluent account

Learning objectives

Students will:

• Gain an awareness of the current Data Engineering landscape
• Understand the fundamentals of streaming data and how it differs from traditional batch processing.
• Gain practical skills transitioning from batch-driven to streaming data architectures.
• Explore strategies for implementing streaming systems that improve the speed and reliability of data pipelines.
• Recognize the evolving role of Data Engineers in the context of streaming data and real-time system adaptations.

Staff

Wade Waldron (Course Author)

Wade has been a Software Developer since 2005. He has worked on video games, backend microservices, ETL Pipelines, IoT systems, and more. He is an advocate for Test-Driven Development, Domain-Driven Design, Microservice Architecture, and Event-Driven Systems. Today, Wade works at Confluent, showing people how to build modern data streaming applications.

LinkedIn

Afzal Mazhar (Course Author)

Afzal Mazhar is a technologist with a passion for helping developers solve complex problems. His extensive experience allows him to appreciate the challenges being faced and provide the right insight to get things done. He has worked in finance, big data, medical informatics and nuclear medicine; his current passion is leveraging data for social good.

LinkedIn

Peter Moskovits (Course Author)

Peter Moskovits leads the Developer Education team at Confluent, focusing on delivering high-quality educational content to developers on YouTube and Confluent Developer. Peter has worked in a diverse range of environments, from 70-person startups in Silicon Valley to multinational corporations with over 300,000 employees. His decades-long experience includes roles in developer training, creating both in-person and online educational programs, and positions in developer relations and product management. Peter is the co-author of multiple books and is the creator and maintainer of the oh-so-popular Apache Kafka, Apache Flink, and Apache Iceberg Learning Resources github repos.

LinkedIn