Course: Building Data Pipelines with Apache Kafka® and Confluent

Integrating Kafka with External Systems

5 min
Tim BerglundSr. Director, Developer Advocacy (Course Presenter)
Robin MoffattStaff Developer Advocate (Course Author)

Integrating Kafka with external systems

As we saw in the previous lesson, streaming pipelines are made up of at least two—and often three—components: ingest, egress, and optionally processing. Streaming ingest and egress between Kafka and external systems is usually performed using an Apache Kafka component called Kafka Connect.

Using Kafka Connect, you can create streaming integration with numerous different technologies, including:

There are lots of connectors available on Confluent Cloud. These connectors are fully managed, which means you have zero infrastructure to operate, monitor, and upgrade; you only need to configure them with the necessary settings, using the graphical interface, API, or command line.

mongodb-kafka-connector

kafka-connect-web-ui

You can also run Kafka Connect yourself. It's a fully distributed system, making it scalable and resilient. You might choose to run your own Kafka Connect cluster if you also run your own Kafka brokers, or if you are using Confluent Cloud but need a particular connector that's not offered on it yet (such as a custom connector you have built yourself).

kafka-connect-architecture

You can use the Confluent UI to configure Kafka Connect, and you can also use the Kafka Connect REST API to send it configuration in JSON.

{
  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
  "connection.url" : "jdbc:mysql://asgard:3306/demo",
  "table.whitelist": "sales,orders,customers"
  []
}

Whichever way you configure Kafka Connect, and whether you use fully managed connectors or self-managed, there is no coding required to integrate between Kafka and these other systems—it's just configuration!

Use the promo code PIPELINES101 to receive $101 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.