MirrorMaker 2 for Hybrid Clouds

MirrorMaker 2 is the only open-source option that we’ll discuss today. So, if your goal is to run Kafka and stick with open-source products MirrorMaker 2 is your only choice.

There are several benefits to using MirrorMaker 2:

First:MirrorMaker 2 leverages the Kafka Connect framework and ecosystem. Includes both source and sink connectors.

It automatically detects new topics, and partitions.

Automatically syncs topic configuration between clusters.

Manages downstream topic ACLs.

It supports "active/active" cluster pairs and any number of active clusters.

It supports cross-datacenter replication, aggregation, and other complex topologies, as we discussed earlier in the course.

It provides metrics including end-to-end replication latency across multiple data centers/clusters.

And MirrorMaker 2 supports virtually all consumer types for consumer offset sync

However, there are a few considerations that we need to keep in mind when selecting MirrorMaker 2:

While MirrorMaker 2 exposes both JMX and the standard Kafka Connect metrics, it doesn’t expose replication lag or throughput;

these must be derived by the operator through custom tooling.

While MirrorMaker 2 will automatically sync your offsets, you will still need to configure a system to translate those offsets.

Offset translation is still required.

There is limited documentation for monitoring, tuning, and securing your MirrorMaker 2 configuration.

When we talk about failover logic, or what happens when your cluster goes down, this is application-specific and can be time-consuming and labor-intensive to set up and maintain.

Any changes to MirrorMaker 2 must be made on the properties file for each instance and requires a restart of the connect cluster.

If you plan on scaling your clusters or replicating to new instances, there is a significant amount of overhead as:

Each MirrorMaker 2 instance requires 4 internal connectors and 3 internal topics on your Kafka cluster and

each destination cluster needs to have its own MirrorMaker 2 connector.

While we covered the replication patterns in an earlier module, there are a couple of considerations you should keep in mind when it comes to scaling your MirrorMaker 2 clusters.

For each target, you will need to configure a MirrorMaker 2 connect cluster.

So that means if you have two Kafka clusters you will only need one MirrorMaker 2 connect cluster located on the target.

If you decide to scale your architecture to have one source, and two targets, or destinations, you will need two MirrorMaker 2 clusters.

This is also true if you are looking to have a bidirectional flow with each cluster replicating each other.

You will need a MirrorMaker 2 cluster for each, as each serves as a destination cluster.

No matter the size of your cluster you will need a MirrorMaker 2 connect cluster for each destination.

This, combined with each MirrorMaker 2 instance requiring 4 connectors and 3 internal topics can create a significant amount of overhead and cost to your Kafka clusters.

MirrorMaker 2 is a great open-source solution to configure your hybrid cloud architecture,

however, as mentioned earlier in this module, it has some significant considerations that need to be addressed and manually configured for your solution to be robust and trustworthy.

The official Apache documentation has more details on how to configure MirrorMaker 2.

I also suggest sticking around and completing MirrorMaker 2 hands-on module later in the course.

The biggest difficulty in working with MirrorMaker 2 is the lack of official documentation and support.

I don’t want to downplay the amount of time and effort it can take to set up and maintain MirrorMaker 2 instance on top of the ongoing maintenance of Kafka.

While there is some documentation out there, in most cases, you will be on your own to troubleshoot and configure.

I highly recommend you try out and test MirrorMaker 2 in your current setup before trying to scale it up or build your entire architecture on top of it.

Part of my reasoning is the additional overhead of having to set up and configure a separate instance of the MirrorMaker 2 connect cluster for each destination cluster.

In the end, MirrorMaker 2 is a great solution for replication if you are willing to devote the teams and resources to the configuration and management of the architecture.

If you aren't already on Confluent Developer head there now using the link in the video description to access the rest of this course, the hands-on exercises, and additional resources.