Course: ksqlDB 101

Converting Data Formats with ksqlDB

2 min
Allison WaltherIntegration Architect (Course Presenter)
Robin MoffattStaff Developer Advocate (Course Author)

Converting Data Formats with ksqlDB

There are different ways to serialize data written to Apache Kafka topics. Common options include Avro, Protobuf, and JSON.

You can use ksqlDB to create a new stream of data identical to the source but serialized differently. This can be useful in several cases:

  • If the source stream is JSON or delimited (CSV), then it does not have an explicit schema, which can make it more difficult for consumers to work with. Using ksqlDB, you can apply a schema to the data and write it in a format more suitable for all consumers.
  • You may be using a format, such as Avro, but a consuming application may insist on another format, such as delimited (CSV).

To write a stream of data from its CSV source to a stream using Protobuf, you would first declare the schema of the CSV data:

CREATE STREAM source_csv_stream (ITEM_ID INT, 
                                 DESCRIPTION VARCHAR, 
                                 UNIT_COST DOUBLE, 
                                 COLOUR VARCHAR, 
                                 HEIGHT_CM INT, 
                                 WIDTH_CM INT, 
                                 DEPTH_CM INT) 
                          WITH (KAFKA_TOPIC ='source_topic', 
                                VALUE_FORMAT='DELIMITED');

Then you would use a continuous query to write all of these events to a new ksqlDB stream serialized as Protobuf:

CREATE STREAM target_proto_stream 
WITH (VALUE_FORMAT='PROTOBUF') 
AS SELECT * FROM source_csv_stream

Use the promo code KSQLDB101 to get $101 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.