Cluster Linking enables you to directly connect clusters and mirror topics from one cluster to another. Cluster Linking makes it easy to build multi-datacenter, multi-region, and hybrid cloud deployments. It is secure, performant, tolerant of network latency, and built into Confluent Server and Confluent Cloud.
In this hands-on exercise, we'll configure Cluster Linking between two cloud clusters located in different regions and on different cloud providers, Amazon AWS and Google Cloud.
Since we'll be using Confluent Cloud to perform this hands-on, make sure that you've gone through the first hands-on in the course, or that you have an account that allows you to set up and manage environments. For our source cluster, you can use any cluster type, Basic, Standard, or Dedicated. However, for the destination cluster, you will need to have a Dedicated instance.
The first step is making sure that you have an Environment set up on Confluent Cloud that you can use for this hands-on. You'll want to create one using the Essentials Stream Governance Package on AWS near where you are located. I'll name mine
Next, to create a cluster click
Create cluster on my own
Begin configuration for a Basic cluster.
Select where you want your cluster to be located. I'm going to select Google Cloud, Las Vegas, with a single zone of availability.
Since this will be my source cluster, I'll use a name that makes following this hands-on easy.
Orders_USWest_Source. You will most likely already have a naming convention you'd like to follow or clusters that have already been named.
Since we will need another cluster to set up Cluster Linking, navigate back out into your environment and add a new cluster.
As mentioned above, this one needs to be a Dedicated cluster.
This time I'll select AWS, North Virginia, with a single zone for my availability.
Internet and then
Automatic and then
Name your new cluster. Similar to our source, I'll name this one
Orders_USEast_Destination. Again, you'll want to name it something that fits in with your setup and naming convention.
Review the summary and click
While we wait for the dedicated cluster to be created, let's get the data flowing into our source cluster.
Click on the
Orders_USWest_Source cluster and navigate into
Connectors on the left of the screen.
Select the Datagen Source connector if it appears. If you don't see it in the list, type
Datagen in the search box at the top of the screen.
Add new topic.
Pick a name and set up the number of partitions you'd like. In this case, I'll name the topic
Orders_West with 3 partitions.
Create with defaults
Select the topic we just created and click
Global access and click
Generate API key & download.
Give the API key a description and click
Avro and the
Orders dataset and click
Leave the Connector sizing at
1 and click
Name the Datagen connector, or accept the default name and click
Once the Datagen connector has been provisioned you can go into your Topics,
Orders_West topic, and click on
Messages to see the data flowing in.
We now have two clusters, one basic that will serve as our source cluster that has our topic and data flowing in, and one dedicated that will be our destination cluster.
We'll now set up our cluster link from the source to the destination cluster.
Make sure your dedicated cluster is provisioned before starting the next step. Go to your
ClusterLinking Environment. It should look similar to the image below. You should be able to see that your source cluster has data flowing through it, but the destination cluster does not as we haven't configured anything yet.
Let's create the cluster link by clicking on the
Cluster Link menu on the left of the screen.
Create cluster link
This screen allows you to configure three types of links, in your organization, in another organization, or from either Confluent Platform or Apache Kafka. We will select
Confluent Cloud, our
ClusterLinking Environment, and the source,
Orders_USWest_Source. We will also provide this cluster link with read-all access. You could also configure specific access by selecting
Now we will select our destination environment and cluster.
There are a few different configuration options that need to be selected. In this case, I want the cluster link to
Auto-create mirror topics,
Add a prefix to my mirror topics and
Sync consumer offsets as well.
Name the cluster link, in this case, I'll name it
Launch cluster link.
Now we have to set up which topics we want to be mirrored. Click
Add mirror topic.
Orders_West from the dropdown and click
You can now navigate to your destination cluster and see the newly created mirror topic with all the messages coming in.
Congratulations! You have successfully set up a cluster link.
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.