Install and Run Apache Kafka®

Confluent Platform includes Apache Kafka. You can download and unpack the community edition by running this command:

wget https://packages.confluent.io/archive/7.0/confluent-community-7.0.0.tar.gz
tar -xf confluent-community-7.0.0.tar.gz

Confluent provides multiple ways to install and run Kafka, including Docker, ZIP, TAR, Kubernetes, and Ansible.

A great alternative to having to install and run Kafka yourself is to use the fully managed cloud service provided by Confluent Cloud.

The recommended approach for running Kafka on Windows is to run it under WSL 2 (Windows Subsystem for Linux). This blog post provides step-by-instructions showing you how to run Kafka on Windows.

Note that while this is fine for trying out Kafka, Windows isn’t a recommended platform for running Kafka with production workloads. If you are using Windows and want to run Kafka in production, then a great option is to use Confluent Cloud along with the provided Confluent CLI (which is supported on Microsoft Windows) to interact with it.

If you install Kafka using the .tar.gz package from the Kafka website, it will be in whichever folder you unpack the contents into. The same applies if you install Confluent Platform manually.

If you install Confluent Platform using the RPM or Debian packages, then you will by default find the data files in /var/lib and configuration files in /etc (e.g. /etc/kafka).

For more details on specific paths and installation methods see the documentation.

1. Paste the Docker Compose file below into a file called docker-compose.yml.

   ---
   version: '3'
   services:
     zookeeper:
       image: confluentinc/cp-zookeeper:7.0.0
       container_name: zookeeper
       environment:
         ZOOKEEPER_CLIENT_PORT: 2181
         ZOOKEEPER_TICK_TIME: 2000
   
     broker:
       image: confluentinc/cp-kafka:7.0.0
       container_name: broker
       ports:
       # To learn about configuring Kafka for access across networks see
       # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
         - "9092:9092"
       depends_on:
         - zookeeper
       environment:
         KAFKA_BROKER_ID: 1
         KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
         KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
         KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
         KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
         KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
         KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

2. Run docker-compose up -d.

For more details see the Docker quick start.

There are many other Docker images you can run alongside the broker including one for a Kafka Connect worker, a ksqlDB instance, a REST proxy, Schema Registry, etc. For a Confluent Platform demo that includes those services along with a greater set of features, including security, Role-Based Access Control, Kafka clients, and Confluent Replicator, see cp-demo.

Confluent for Kubernetes (CFK) runs on Kubernetes and provides a cloud-native control plane with a declarative Kubernetes-native API approach to configure, deploy, and manage Apache Kafka®, Connect workers, ksqlDB, Schema Registry, Confluent Control Center, and resources like topics and rolebindings, through Infrastructure as Code (IaC).

To install CFK, run the following:

helm repo add confluentinc https://packages.confluent.io/helm
helm repo update
helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes

For an overview of Kafka on Kubernetes, see this talk from Kafka Summit.

Once you have downloaded Confluent Platform, start ZooKeeper using the Confluent CLI.

confluent local services zookeeper start

Or you can start the whole platform all at once:

confluent local services start

Keep in mind that you can also launch the Apache Kafka broker in KRaft mode (which is experimental as of Confluent Platform 7.0.0), which means that it runs without ZooKeeper. See this page for more details.