Staff Software Practice Lead
Dealing with basic strings in our Kafka messages is convenient, but usually not what we want. Typically, we want to convert from a C# class or struct into a string or binary representation of the object. To do this, we can use Kafka Serializers. They allow us to convert a complex object into a wire-compatible format such as JSON, Avro, or Protobuf. This video will demonstrate how to create Kafka serializers and deserializers and connect them to your producer or consumer.
Topics:
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.
Hi, I'm Wade from Confluent. Let's take a moment to learn about serialization and deserialization when working with Kafka in .NET. When we create a message producer, we provide it with two types. The first type represents the Key of the message, while the second type represents the Value. These types can be almost anything. They can be simple types, such as strings or integers but they can also be more complex types, such as a custom class or struct. However, Kafka itself doesn't actually understand these types. It just works with blobs of data in the form of bytes. This means that if we want to send the data to Kafka, we first need to convert it from a complex object into an array of bytes. If we use simple types, such as strings or integers, this conversion is easy. However, for more complex types, we're going to need to provide rules about how to convert from our object to a set of bytes. This is known as serialization. The Kafka client provides several built-in serializers for some of the basic types, such as strings, integers, floats and doubles, byte arrays and more. It also includes support for more complex types of serialization, including Google Protobuf, Apache Avro, and JSON. When using serializers for the basic types, we don't need to do anything. They should just work right out of the box. However, if we are using one of the more complex serializers, we'll need to register it with our producer. To register a serializer for the message value, we use the SetValueSerializer method on the ProducerBuilder. In this example, we're using a JsonSerializer to convert our Biometrics object. We can also apply a serializer for the message key. Here we're registering a JsonSerializer for a DeviceId object. However, it is often preferable to represent a key using a simple object, such as a string or an integer. In that case, registering a key serializer is unnecessary. Once we have registered the serializer with the producer, all we need to do is send it a message of the appropriate type. The producer will accept the message, execute the necessary serializers to convert it to an array of bytes, and then send the results to Kafka. Sometimes we need to serialize an object where none of the built-in serializers is appropriate. In this case, we can create our own serializer by implementing the ISerializer interface we will then be able to provide our own set of instructions for how to convert our object into an array of bytes. For complex types such as JSON or Avro, it's often desirable to perform the serialization in an asynchronous fashion. To do this, we can leverage the IAsyncSerializer interface instead. This may allow our system to make better use of resources, such as threads in CPU. The built-in serializers for JSON, Avro and Protobuf all use this interface under the hood. Converting to an array of bytes is only half the battle. Once we have the data in Kafka, we need to get it out again. This means we need to deserialize the data. Thankfully, the process for deserializing is largely the same as serializing. We have the same set of built-in deserializers for basic types and more complex types, and if necessary, we can leverage custom deserializers by implementing the IDeserializer or IAsyncDeserializer interfaces. The combination of serialization and deserialization is sometimes called SerDes for short. When used together, they provide a convenient way to convert from the domain objects in our code to the wire format required by Kafka. If you aren't already on Confluent Developer, head there now using the link in the video description to access the rest of this course and its hands-on exercises.