Hands On: Use the Python Producer Class

Use Producer to Send Events to Kafka

In this exercise, you will use the Producer class to write events to a Kafka topic.

Create Topics

1. In the Confluent Cloud Console, navigate to the Topics page for the kafka-python cluster in the learn-kafka-python environment.
2. Create a new topic called hello_topic, with 6 partitions and defaults settings.

Project Setup

3. Open a terminal window and navigate to the kafka-python directory that you created in the previous exercise.

4. If you are not currently using the kafka-env environment that was created in the last exercise, switch to it with the following command:

   source kafka-env/bin/activate

5. Create a file called producer.py.

6. Open producer.py in an IDE of your choice.

Add Required Imports

7. Add the following import statements to the top of the producer.py file:

   from confluent_kafka import Producer
   from config import config

Create callback function

8. Create a function called callback() that can be passed to the produce() method.

   def callback(err, event):
       if err:
           print(f'Produce to topic {event.topic()} failed for event: {event.key()}')
       else:
           val = event.value().decode('utf8')
           print(f'{val} sent to partition {event.partition()}.')

Create function to produce to hello_topic

9. Create a function called say_hello() that takes a producer and a key.

   def say_hello(producer, key):
       value = f'Hello {key}!'
       producer.produce('hello_topic', value, key, on_delivery=callback)

Add Main Block

10. Add the following main block that will pull this all together

    if __name__ == '__main__':
        producer = Producer(config)
        keys = ['Amy', 'Brenda', 'Cindy', 'Derrick', 'Elaine', 'Fred']
        [say_hello(producer, key) for key in keys]
        producer.flush()

Run the Program

11. Execute the program by running the following command

    python producer.py

    Notice how the different names, which we are using for keys, result in specific partition assignments. To get a better idea of how this works, you can try changing some of the names and see how the partition assignment changes.

    For a good explanation of partitions and partition assignments, please refer to the Kafka 101 course on Confluent Developer.