Hands On: Confluent Cloud Schema Registry and Spring Boot

Hi, this is Viktor Gamov with Confluent. And in this exercise, I'm gonna show you how we can use Cloud Schema Registry together in your Spring Boot application. Let's get to it. And we're back to our application that we developed on the previous lesson. And in order to use Schema Registry, we need to make sure that the Schema Registry is activated in our Confluent Cloud. So I'm going Confluent Cloud in my environment and I find the Schema Registry. I should be able to see something like this, like end point that I will be able to connect to, and some of the keys that I will be able to connect my client application. We'll get back to those in a few seconds. Next thing is that we also need to use a plugin that allows us to generate Java POJOs out of Avro schema. So in this case, we are gonna be using this Avro Gradle plugin. We're gonna copy this piece in our build script. So we go into the build.gradel, find the section about plugins. In order for this to be activated and used correctly with my IDE, I also need to enable intelligent idea plugin. So I will refresh the configuration. And how I would know if this actually works, I should be able to see a new task here very soon. Like after a few seconds when the Gradle will be updated called SOR generation. All right, in order to generate our Java POJOs we need to create Avro schema, which will be... We will be using the Json format for the schema. Going here in src/main. And we can create new folder called avro. And inside this avro folder I will create a file called hobbit.avsc so in this case, we're going to be using this schema that represents this Hobbit object that will be holding one property called quote. So we're going to store our string that was generated using the faker library. And after that, we'll wrap this into Hobbit object. So in order to get this working, I need to go and execute task Gradle tasks called generate Avro Java. And what I will see here in my build directory I will see this generated main Avro folder with my object hobbit. And because I enable intelliJ idea Gradle integration it was able to modify intelliJ idea project and also include generated sources in the class path. Now let's modify our application code as well to in order to, in order to support this. Also, couple dependencies that we're missing here. So, first of all, we see that it was new some code but some of the things are missing. So this is why we don't have our avro dependencies. We'll fix that in a second. Now we're going to be using confluent provided a serializer for our application that we'll be using this Avro object, then serialize it and the store information, the schema registry. So in this case they're going to be using Kafka Avro serializer in the version of confluent platform, also aligned with the version of the Kafka that spring brings. So in a, in application.properties what we need to change now and we need to specify this special serializer called Kafka serializers and our Kafka Avro serializer. And at this point we'd done with our producer let's fix our consumer. So in our consumer code. So first of all, we're going to be using different topics. So let's call it topic Avro, Hobbit Avro and we're going to be listening topic called Hobbits Avro. In this case, our string becomes integer because our key is integer type and our value is a Hobbit here as well. So that's our consumer. We don't need to use a processor. So that's why I just commented out this, this annotations. So those methods will not wouldn't be created and the rest controller as well. So just in case, because we don't need this for now. Now our producer now supposed to be creating messages in Avro format and write these messages into Kafka topics. So let's create this new, new Kafka topic. We don't need this guy actually here let's call it Hobbit Avro and we will provide a name Hobbit Avro here as well. Before I run this application I need to copy one important thing from our Confluent Cloud config. So let's get back to Confluent Cloud and switch into these clusters. My cluster, we go on into clients and Spring boot and now what we can see here we can generate our scheme registry key and the password let's call it a spring Kafka lesson. We're gonna save this continue. So in this case my place holder was replaced with actual values, that's something that I will copy in my in my code, in my configuration. So I'm going to application.properties and somewhere here around these lines I should put this configuration for scheme registry couple things that we put this extra. It's a scheme registry URL. This is the location of our endpoint that is connected to our Confluent Cloud cluster and the some of the authentication properties, basic auth user and the type of credential source. After I modify the code for my Kafka consumer here I also need to change a configuration for my consumer. So in this particular case, my consumer was using the long in string serializer deserializer. So in this case it's going to be using integer deserializer and it will going to be using io.com Confluent.Kafka.seriailizers and the Kafka Avro deserializers. So in this case, my producer will produce data in our format and my consumer will consume data in Avro format. So what I will do here don't forget to disable Kafka streams here because we don't need this for now. We don't, modify this code. And once again, our producer writes data into topic in a Hobbit format, and our consumer also does the same. Alright, so let's run this. And what we start seeing here is our system out printlns, start writing data in Avro format. So let's see if we can see the same data in our topic. I'm going here, topics, Hobbit Avro messages. And voila, now we are able to see something like this. So value would be stored in Avro format. Again, UI is not fully show this. However, we now can see inside our topic in the schema tab we finally can see our schema. We can see the version of the schema. We can defy the schema and so forth and so on. And now you have it in this lesson. You learn how we can use Avro format and how you can connect the cloud schema registry in order to produce and consume some of the binary data and use this for more complex objects that you might have in your applications.