Converting Data Formats with ksqlDB

Converting Data Formats with ksqlDB

There are different ways to serialize data written to Apache Kafka topics. Common options include Avro, Protobuf, and JSON.

You can use ksqlDB to create a new stream of data identical to the source but serialized differently. This can be useful in several cases:

• If the source stream is JSON or delimited (CSV), then it does not have an explicit schema, which can make it more difficult for consumers to work with. Using ksqlDB, you can apply a schema to the data and write it in a format more suitable for all consumers.
• You may be using a format, such as Avro, but a consuming application may insist on another format, such as delimited (CSV).

To write a stream of data from its CSV source to a stream using Protobuf, you would first declare the schema of the CSV data:

CREATE STREAM source_csv_stream (ITEM_ID INT, 
                                 DESCRIPTION VARCHAR, 
                                 UNIT_COST DOUBLE, 
                                 COLOUR VARCHAR, 
                                 HEIGHT_CM INT, 
                                 WIDTH_CM INT, 
                                 DEPTH_CM INT) 
                          WITH (KAFKA_TOPIC ='source_topic', 
                                VALUE_FORMAT='DELIMITED');

Then you would use a continuous query to write all of these events to a new ksqlDB stream serialized as Protobuf:

CREATE STREAM target_proto_stream 
WITH (VALUE_FORMAT='PROTOBUF') 
AS SELECT * FROM source_csv_stream