Managing Schemas

Hi, I'm Danica Fine, thanks for joining me for another Schema Registry module. In this module, we'll see how to manage both Protobuf and Avro schema files. Schema management starts with registering schemas, but further in the life cycle, it also includes updating schemas and viewing or downloading them. Later in this course, we'll also cover evolving and testing a schema. For now, let's see how to register a schema. Suppose we've already created some schema and we want to register it. Before we get into the ways that we can register a schema, let's first talk about what happens during registration. When you register a schema, you provide the subject name in addition to the schema itself. The subject name is what we'll use to identify the schema, and it's usually just the namespace for the schema, almost like a key when you use a HashMap. The standard naming convention is the topic name followed by key or the topic name followed by value, depending on whether you're defining a schema to be used for a key or a value. It doesn't just end there, though, there are some other strategies for subject names that we'll cover in a later session. Once Schema Registry receives the schema, it assigns it a unique ID number and a version. Unsurprisingly, the first time you register a schema for a given subject name, it's assigned a version of 1. After this happens, we're done, the schema has been registered. All right, let's get back to the registration process. Like I said, there are several methods at your disposal for registering a schema. You can use the Confluent CLI, the Schema Registry REST API, the UI on Confluent Cloud, or the Maven and Gradle plugins we discussed earlier. The world is your oyster. The example seen here uses the Confluent CLI, take note of the subject name, "purchases-value". This indicates that this schema represents the value part of the key-value records within the "purchases" topic. If we also wanted to have a schema to define the key, the subject name of that schema would be "purchases-key". And as a side note, while you can have a schema for the key, it's worth mentioning that scalar values, like string and integer, are better suited for the key. With that in mind, we'll only discuss using schemas for the value for the rest of this course. Going back to the command here, note that when using Protobuf, you need to explicitly specify the schema type as PROTOBUF. For Avro, you don't need to provide the type, Schema Registry uses Avro by default. Let's see the registration process using the Schema Registry REST API. In this example, we use the command line JSON tooL, JQ, in conjunction with the REST API. Since Avro schemas are defined in JSON, you can use them as-is in the command. Protobuf schema definitions are not defined in JSON, so you need to first get them into JSON format before you can register them using the REST API. In this example, we use a helper script to parse and format the schema into JSON. The curl command then uses this JSON-formatted output from the script. The Gradle and Maven plugins we discussed earlier are also capable of registering a schema. This example illustrates registering two schemas using the Gradle Schema Registry plugin. In the Gradle build script, you need to provide a subject entry in the register block which includes the path to the schema file and the schema type. This block can include one or multiple subject entries, and each entry can be either the AVRO, PROTOBUF, or JSON_SR schema type. Simple! Using a plugin with your development environment could potentially be the easiest approach to registering schemas. Just the one registerSchemasTask command will register all of the entries in the register block. And finally, when you're using a Kafka Producer, you can also enable it to auto-register a schema. Although, while this is very handy for development, it's not something I'd recommend doing in production. All right, so we've created and registered schemas, what do we do when we need to update a schema? Well, you can actually use any of these register methods to update the schema as well. Provided your making compatible changes , Schema Registry will simply assign a new ID to the schema and increment the version number. While the new ID is not guaranteed to be in sequential order, the version number will always be incremented by one. And as you'll see in a second, the version number is actually more important than the ID when viewing the schema. So, we've updated schemas, but how do we view or download a schema? Again, you have a couple options. All you need to pull a schema from Schema Registry is a subject name, and surprise surprise, a version number. If you don't know which version number to use, you can also just use the keyword "latest" which simply gives you the latest version available for the schema. The Gradle plugin also has an additional downloadSchemasTask command to view the schemas. And with that, you have a better idea of how to register and evolve your schemas over time. See you in the next course module where we'll explore how to apply our knowledge to client applications.