Get Started Free
August 31, 2021 | Episode 174

Multi-Cluster Apache Kafka with Cluster Linking ft. Nikhil Bhatia

  • Transcript
  • Notes
undefined

Note: This episode was recorded when Cluster Linking was in preview mode. It’s now generally available as part of the Confluent Q3 ‘21 release on August 17, 2021. 

Infrastructure needs to react in real time to support globally distributed events, such as cloud migration, IoT, edge data collection, and disaster recovery. To provide a seamless yet cloud-native, cross-cluster topic replication experience, Nikhil Bhatia (Principal Engineer I, Product Infrastructure, Confluent) and the team engineered a solution called Cluster Linking. Available on Confluent Cloud, Cluster Linking is an API that enables Apache Kafka® to work across multi-datacenters, making it possible to design globally available distributed systems. 

As industries adopt multi-cloud usage and depart from on-premises and single cluster operations, we need to rethink how clusters operate across regions in the cloud. Cluster Linking as an inter-cluster replication layer into Confluent Server, allowing you to connect clusters together and replicate topics asynchronously without the need for Connect. 

Cluster Linking requires zero external components when moving messages from one cluster to another. It replicates data into its destination by partition and byte for byte, preserving offsets from the source cluster. Different from Confluent Replicator and MirrorMaker2, Cluster Linking simplifies failover in high availability and disaster recovery scenarios, improving overall efficiency by avoiding recompression. As a great cost-effective alternative to Multi-Region Cluster, Cluster Linking reduces traffic between data centers and enables inter-cluster replication without the need to deploy and manage a separate Connect cluster. 

With low recovery point objective (RPO) and recovery time objective (RTO), Cluster Linking enables scenarios such as: 

  • Migration to cloud: Remove the complexity layer of self-run datacenters with fully managed cloud services. 
  • Global reads: Enable users to connect to Kafka from around the globe and consume data locally. Empowering better performance and improving cost effectiveness. 
  • Disaster recovery: Prepare your system for fault tolerance, from datacenter, regional, or cloud-level disasters, ensuring zero data loss and high availability. 

Find out more about Cluster Linking architecture and set your data in motion with global Kafka.

Continue Listening

Episode 175September 9, 2021 | 34 min

What Is Data Mesh, and How Does it Work? ft. Zhamak Dehghani

The data mesh architectural paradigm shift is all about moving analytical data away from a monolithic data warehouse or data lake into a distributed architecture—allowing data to be shared for analytical purposes in real time, right at the point of origin. The idea of data mesh was introduced by Zhamak Dehghani (Director of Emerging Technologies, Thoughtworks) in 2019. Here, she provides an introduction to data mesh and the fundamental problems that it’s trying to solve.

Episode 176September 14, 2021 | 35 min

How to Build a Strong Developer Community with Global Engagement ft. Robin Moffatt and Ale Murray

A developer community brings people with shared interests and purpose together. The fundamental elements of a community are to gather, learn, support, and create opportunities for collaboration. A developer community is also an effective and efficient instrument for exploring and solving problems together.

Episode 177September 21, 2021 | 15 min

Apache Kafka 3.0 - Improving KRaft and an Overview of New Features

Apache Kafka 3.0 is out! To spotlight major enhancements in this release, Tim Berglund (Apache Kafka Developer Advocate) provides a summary of what’s new in the Kafka 3.0 release from Krakow, Poland, including API changes and improvements to the early-access Kafka Raft (KRaft).

Got questions?

If there's something you want to know about Apache Kafka, Confluent or event streaming, please send us an email with your question and we'll hope to answer it on the next episode of Ask Confluent.

Email Us

Never miss an episode!

Confluent Cloud is a fully managed Apache Kafka service available on all three major clouds. Try it for free today.

Try it for free