The Schema Workflow

Understanding the Schema Registry Workflow

The workflow of Schema Registry includes:

• Writing a schema file
• Adding schema files to a project
• Leveraging schema tools and plugins—Maven and Gradle
• Configuring Schema Registry plugins
• Generating the model objects from a schema
• Locating and exploring the generated files

In this module you will learn about the workflow of using schemas. This workflow starts with writing schema files, adding them to a project, and leveraging tools such as Maven and Gradle to generate the model objects that schemas represent as well as register and update them in a schema registry.

Writing a Schema File – Protobuf

Let’s first look at a schema defined using the Protobuf format.

• The first line declares the version of Protocol Buffers to use. This course uses version 3.
• The package declaration creates a name space for proto files to prevent name clashes. It ends up being the package name in the generated Java class unless you specify an option java_package in the proto file.
• The option java_outer_classname field describes the name of the generated java file
• Protocol Buffers define the message in the schema as an inner class in the generated file

More detail of how Protobuf works is in the next module.

Writing a Schema File – Avro

Next, let’s examine what the same schema looks like using the Avro format. Notice that it’s defined in JSON format.

• The namespace field serves the same purpose as in the Protobuf file, preventing name collisions. The namespace also becomes the package name for the generated Java file.
• Fields are declared in a JSON array.

More detail about Avro is in the next module as well.

Placing Schema Files in a Project

The schema files you write need to go in your project somewhere. In this course the Avro schema files go in the src/main/avro directory and Protobuf files land in src/main/proto.

Available Plugins

Avro – Gradle and Maven plugins

• https://github.com/davidmc24/gradle-avro-plugin
• https://mvnrepository.com/artifact/org.apache.avro/avro-maven-plugin

Protobuf – Gradle and Maven plugins

• https://www.xolstice.org/protobuf-maven-plugin/
• https://github.com/google/protobuf-gradle-plugin

Once you’ve written your schema files you will want to generate the model/data objects from the schema files. Fortunately, there are plugins available, either Maven or Gradle, that can integrate into your local build.

This course focuses on how to configure and use the Gradle plugin. In the upcoming exercise you will configure a build.gradle file to use with Avro and Protobuf schemas.

We should note that Schema Registry supports JSON Schema as well, but that’s more of a specification while Avro and Protobuf are full serialization platforms. As a result the tooling is much better for these two versus JSON Schema. JSON Schema will be covered when we go in-depth on the schema types, but the rest of the course sticks to either Avro or Protocol Buffers.

Plugin Configuration

Let’s now do a quick walkthrough of a Gradle configuration file. The plugins section is where you identify the plugins that will be used when running Gradle scripts. This example includes plugins required for working with Schema Registry.

Protobuf Plugin Configuration

Here’s how the Protobuf plugin is set up. The generatedFilesBaseDir points to the directory where the generated files land.

The upcoming exercise will use the Avro plugin defaults so the gradle.build file doesn’t contain configuration settings for it.

Schema Registry Configuration

Here you have the schemaRegistry configuration block for the Gradle schema registry plugin.

• The url field is set to the endpoint address for the Scheme Registry
• The credentials block includes authentication details that allow connectivity with Schema Registry in Confluent Cloud. The credentials included in requests sent to Schema Registry need to be in the form API Key:API Secret. The Gradle schema registry plugin will form these credentials using the values set for <username> and <password>. This will result in <username>:<password> being included in the request.
• The register block contains all the information the plugin uses to register a schema.

Generated Files in a Project

To generate the objects from schema files, run the ./gradlew clean build command. This will generate the files seen in this illustration. Note that the files shown here don’t necessarily reflect what you will see when you go through the exercise that follows.