Get Started Free
Tutorial

How to handle multiple event types in a Kafka topic with Protobuf

Kafka
‹ Back to Tutorials

How to handle multiple event types in a Kafka topic with Protobuf

It's sometimes advantageous to produce distinct but related event types to the same topic, e.g., to guarantee the exact order of different events for the same key. For example, consider pageview and purchase records associated with the same customer ID. In order to properly attribute purchases to preceding pageviews, these distinct events must be sent to the same topic so that the order is preserved in one Kafka topic partition. But, let's say we also need to maintain the topic-name subject constraints with Schema Registry.

To accomplish this with Protobuf-formatted events, we can use schema references, where a schema contains a field whose type is a reference to another schema.

Example Protobuf schema with references

The example in this tutorial uses a top-level Protobuf schema specifying that a record is either a purchase or a pageview:

message CustomerEvent {
  oneof action {
    Purchase purchase = 1;
    Pageview pageview = 2;
  }
  string id = 3;
}

Where these references are defined as follows:

message Pageview {
  string url = 1;
  bool is_special = 2;
  string customer_id = 3;
}
message Purchase {
  string item = 1;
  double amount = 2;
  string customer_id = 3;
}

Now, if you use the top-level schema for a topic, then you can produce either PageviewProto.Pageview or PurchaseProto.Purchase records to the topic.

Running the example

In order to run this example, first clone the confluentinc/tutorials GitHub repository (if you haven't already) and navigate to the tutorials directory:

git clone git@github.com:confluentinc/tutorials.git
cd tutorials

Now you can either execute the unit test included with the example, or run the example in Docker or in Confluent Cloud.

Execute the unit tests

To run the unit tests, use the provided Gradle Wrapper:

./gradlew clean :multiple-event-types-protobuf:kafka:test --info
Run in Docker

Prerequisites

Start Kafka and Schema Registry

Start Kafka by running:

docker compose -f ./docker/docker-compose-ksqldb.yml up -d

Create topic

Create the proto-events topic:

docker exec -t broker kafka-topics --create --topic proto-events --bootstrap-server broker:9092

Build the application

Build the application uberjar:

./gradlew :multiple-event-types-protobuf:kafka:shadowJar

Run the application

Run the application, which produces and consumes pageview and purchase events, with the following command:

java -jar multiple-event-types-protobuf/kafka/build/libs/multiple-event-types-protobuf-standalone-0.0.1.jar multiple-event-types-protobuf/kafka/local.properties

Cleanup

Stop Kafka and Schema Registry:

docker compose -f ./docker/docker-compose-ksqldb.yml down
Run in Confluent Cloud

Prerequisites

Create topic

Using the Confluent Cloud Console, create a topic with default settings called proto-events.

Generate client configuration

In the Confluent Cloud Console, navigate to the Cluster Overview page. Select Clients in the lefthand navigation and create a new Java client. Generate API keys during this step, and download the generated client configuration. Place it at multiple-event-types-protobuf/kafka/cloud.properties.

Register schemas

Run the following task to register the schemas in Schema Registry:

./gradlew :multiple-event-types-protobuf:kafka:registerSchemasTask

In the Confluent Cloud Console, navigate to Topics in the lefthand navigation, select the proto-events topic, and click Schema. Validate that a Value schema has been set.

Build the application

Build the application uberjar:

./gradlew :multiple-event-types-protobuf:kafka:shadowJar

Run the application

Run the application, which produces and consumes pageview and purchase events, with the following command. Note that we are passing the client configuration as an argument:

java -jar multiple-event-types-protobuf/kafka/build/libs/multiple-event-types-protobuf-standalone-0.0.1.jar multiple-event-types-protobuf/kafka/cloud.properties

In the Confluent Cloud Console, select the Messages tab for the proto-events topic and view the messages that are produced.

Cleanup

Delete the cluster used for this tutorial if you no longer need it.