Hi, I'm Wade from Confluent. In this video, we're going to see how to create a Kafka producer in .NET. Before we start looking at code, let's take a moment to talk about what a producer is. Systems based on Kafka are focused on communicating using asynchronous messages but those messages need to come from somewhere. This is where the producer comes in. When we want to publish messages in .NET for any reason, we will do it using a producer. The responsibility of the producer is to publish messages that will be of interest to downstream consumers. This can be a little tricky because in a system based on Kafka, you don't necessarily know who the consumers will be. However, this isn't really any different than designing an API. When you build an API, you don't always know how it will be used. You try to expose the information you think will be valuable, and then you plan to evolve the API as necessary. In fact, you can consider the messages being produced by your system to be just another type of API. Some APIs will be consumed through HTTP while others might be consumed through Kafka. It's a little bit like traditional publishing. If I write a book, I don't know who will read it or what they are interested in. Instead, I have to make educated guesses about who the readers will be, and then try to include content in the book that will be interesting to them. But once the book hits the shelves, who reads it is out of my control. Similarly, our system can only control messages until they've been produced. Once the messages hit Kafka, we don't really know where they'll end up or what will be done with them. As a result, we should put the same level of care into our messages as we do any API that we build. So how do we actually create and use a producer in .NET? The first thing we need is some configuration. We can create the configuration in code simply by constructing a new ProducerConfig. Then it's just a matter of populating the appropriate fields. There are many configuration settings that can be tuned through the ProducerConfig class. Let's go through some of the important ones. The BootstrapServers field identifies one or more Kafka brokers each specified by a host and port if necessary. It will be used to establish the initial connection to the Kafka cluster. Once connected, additional brokers may become available. The ClientId is used to identify the producer. In other words, to give it a name. Although it's not strictly required, providing a ClientId will make debugging a lot easier. Usually, we need some way of authenticating our producer. We do this by specifying the SecurityProtocol. There are a variety of different authentication methods, each of which has different settings. We'll be using SASL-SSL in this course. Acks can be used to to tune whether the producer will wait for a response from the broker. The default behavior is to wait for an acknowledgement that all brokers have received the message before continuing. This can be adjusted to only wait for the leader or to not wait at all. This can speed up publishing time but carries risks that the messages might be lost. MessageTimeoutMilliseconds will determine how long the producer should wait for the broker before it gives up. Changing this can have an impact on the reliability of your message delivery. There are also settings to control how messages are batched by the producer. BatchNumMessages controls the size of each batch and LingerMs controls how long the producer should wait to accumulate messages in a batch. Changing these can impact the bandwidth, throughput and latency of the system. If your messages are large, you may benefit from compression. Compressing the messages prior to sending them can reduce your bandwidth, and improve your latency. However, it's not a guarantee. It depends on the size and shape of your messages. Some data will compress better than others, and small messages may not see a significant benefit. You'll want to experiment with different CompressionTypes to find the one that fits your use case. There are many other configuration values that can be adjusted in the ProducerConfig. In most cases, it's probably better to leave them with the default values unless you have a specific problem you're trying to solve. Changing them prematurely can impact the reliability and performance of your system. Now, these configuration values could be hard coded but they're usually pulled from a configuration file somewhere. When working with ASP.NET, we can leverage the configure method to simplify the process. If we provide it with an appropriate JSON configuration, it can automatically deserialize it to a ProducerConfig object. This makes loading the configuration file trivial. Once we have a configuration, the next step is to construct our producer. We do that using a ProducerBuilder. The ProducerBuilder accepts two generic types, representing the key and value of the messages that will be produced. It also takes our ProducerConfig that we created previously. For example, if we wanted to produce a message where the Key and Value were both strings, then we might create our builder like this. Our builder contains some additional options that we'll cover later in the course. In the meantime, our final step for creating the producer is to call the Build method. Once we have our producer in place, we can produce a message using the ProduceAsync method. Be careful when calling ProduceAsync. Because it's an async method, you may need to await the result. The most basic form of this method takes a name for the topic where the messages will be produced, and the actual message. However, the messages aren't necessarily sent immediately. They may be buffered in memory so that multiple messages can be sent as a batch. Once we're sure we want the messages to be sent, it's a good idea to call the Flush method. However, be aware that Flush is a synchronous method. It will wait for an acknowledgement from the broker before continuing. If you flush every message, you are essentially creating a synchronous producer and that can have consequences on your performance. It's often better to produce multiple messages into a batch prior to calling Flush. If you aren't already on Confluent Developer, head there now using the link in the video description to access the rest of this course and its hands-on exercises.