Integrating Kafka with External Systems

Integrating Kafka with external systems

As we saw in the previous lesson, streaming pipelines are made up of at least two—and often three—components: ingest, egress, and optionally processing. Streaming ingest and egress between Kafka and external systems is usually performed using an Apache Kafka component called Kafka Connect.

Using Kafka Connect, you can create streaming integration with numerous different technologies, including:

• Cloud data warehouses, such as BigQuery and Snowflake

• Relational databases, like Oracle, Postgres, MySQL, etc.

• Cloud object stores, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage

• Message queues, like ActiveMQ, IBM MQ, JMS, and others

• NoSQL stores, such as Cassandra and MongoDB

• Document stores like Elasticsearch

• SaaS platforms such as Salesforce

• …and so many more

There are lots of connectors available on Confluent Cloud. These connectors are fully managed, which means you have zero infrastructure to operate, monitor, and upgrade; you only need to configure them with the necessary settings, using the graphical interface, API, or command line.

You can also run Kafka Connect yourself. It's a fully distributed system, making it scalable and resilient. You might choose to run your own Kafka Connect cluster if you also run your own Kafka brokers, or if you are using Confluent Cloud but need a particular connector that's not offered on it yet (such as a custom connector you have built yourself).

You can use the Confluent UI to configure Kafka Connect, and you can also use the Kafka Connect REST API to send it configuration in JSON.

{
  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
  "connection.url" : "jdbc:mysql://asgard:3306/demo",
  "table.whitelist": "sales,orders,customers"
  […]
}

Whichever way you configure Kafka Connect, and whether you use fully managed connectors or self-managed, there is no coding required to integrate between Kafka and these other systems—it's just configuration!