There are different ways to serialize data written to Apache Kafka topics. Common options include Avro, Protobuf, and JSON.
You can use ksqlDB to create a new stream of data identical to the source but serialized differently. This can be useful in several cases:
To write a stream of data from its CSV source to a stream using Protobuf, you would first declare the schema of the CSV data:
CREATE STREAM source_csv_stream (ITEM_ID INT, DESCRIPTION VARCHAR, UNIT_COST DOUBLE, COLOUR VARCHAR, HEIGHT_CM INT, WIDTH_CM INT, DEPTH_CM INT) WITH (KAFKA_TOPIC ='source_topic', VALUE_FORMAT='DELIMITED');
Then you would use a continuous query to write all of these events to a new ksqlDB stream serialized as Protobuf:
CREATE STREAM target_proto_stream WITH (VALUE_FORMAT='PROTOBUF') AS SELECT * FROM source_csv_stream
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.