Hands On: Configure, Build and Register Protobuf and Avro Schemas

In this exercise you are going to complete the following tasks:

• Examine the Schema Registry settings in the gradle.build file
• Configure Protobuf and Avro schema definitions
• Generate model objects from the schema definitions using Gradle
• Register the Protobuf and Avro schemas in Confluent Cloud Schema Registry

1. In your editor/IDE, navigate to the schema-registry project, open the build.gradle file, and review its contents beginning near the top.

2. In the plugins section, take note of the protobuf, schema-registry and avro plugins.
3. In the dependencies section, take note of the protobuf and avro required libraries.
4. In the protobuf section, take note of where the generated model objects end up in the project directory.
5. In the schemaRegistry section, take note of the configuration blocks for different commands offered by the plugin.
   

The confluent.properties file you created before is used here for the API key and secret to connect to Schema Registry in Confluent Cloud.

6. The register section contains the information required to register schemas with the registerSchemasTask that you will run later in this exercise.
7. The download section contains the required information for downloading schemas.

In this exercise you will create a schema locally, but in a later exercise you will download one.

8. The compatibility section contains what’s needed to test if changes you make to a schema are compatible.
   

With the latest version of that schema—you will test the compatibility of a schema in the last exercise.

Now that you have reviewed the contents of build.gradle, you are now ready to create a new schema.

9. Open the purchase.proto file contained in the src/main/proto directory.

Some of the details of the schema are already completed. Let’s quickly review them.

10. First notice that you are going to use version 3 of the Protocol Buffer specification.
11. The package (the namespace for this schema) is defined as io.confluent.developer.proto.

This setting will also be the Java package for the generated objects.

12. The java_outer_class-name option is set to PurchaseProto.

Now let’s complete the schema.

13. First, create the message type named Purchase.

    syntax = "proto3";
    
    package io.confluent.developer.proto;
    option java_outer_classname = "PurchaseProto";
    
    // Complete this file by adding the message implementation as directed in the course
    
    message Purchase {
    
    }

14. Add the item field with a type of string and a tag of 1. This field will represent the name of the purchased item.

    syntax = "proto3";
    
    package io.confluent.developer.proto;
    option java_outer_classname = "PurchaseProto";
    
    // Complete this file by adding the message implementation as directed in the course
    
    message Purchase {
      string item = 1;
    }

15. Add the total_cost field with a type of double and give it a tag of 2. This field represents the price of the purchased item.

16. Add the customer_id field with a type of string and a tag of 3.

    Note: Remember the “tags” are the position of the field in the binary payload.

17. Save the purchase.proto file.

Now let’s create an Avro schema.

18. Open up the purchase.avsc file contained in the src/main/avro directory.

There are also some fields that have been pre-filled for you. Let’s review them now.

• The type of the schema is set to record.
• The namespace for the schema is set to io.confluent.developer.avro.

This will be the package name for Java objects generated from the schema as well.

Now let’s complete the schema.

19. First, add the name of the record Purchase.

    {
      "type":"record",
      "namespace": "io.confluent.developer.avro",
      // Complete the JSON for the Avro record as directed in the course
      "name":"Purchase",
    }

20. Now add a fields key to the JSON—this will point to a JSON array containing the record’s fields.

21. Add a JSON object with the name of item and a type of string.

      "name":"Purchase",
      "fields": [
        {"name": "item", "type":"string"},

    Note: Remember to add a comma after each field object, excluding the last one.

22. Add a second field with the name of total_cost and a type of double.

23. Add the final field with a name of customer_id and a type of string.

24. To complete the fields array, add a closing bracket.

    {
      "type":"record",
      "namespace": "io.confluent.developer.avro",
      "name":"Purchase",
      "fields": [
        {"name": "item", "type":"string"},
        {"name": "total_cost", "type": "double" },
        {"name": "customer_id", "type": "string"}
      ]
    }

25. Save the purchase.avsc file.

Now let’s generate the objects from our schemas.

26. Open a terminal window and navigate to the root of the schema-registry project.
27. From your terminal window, run the command ./gradlew clean build.

You will see the output of the build in the terminal and it should end with a Build Successful message.

Now let’s go take a look at where the generated files end up. Note that you won’t need to touch the generated files directly. You need them when you build your app. The generated objects are automatically added to the classpath. Referencing them from your app is no issue.

28. In your IDE, navigate to the build directory and expand the directory structure.

• The generated proto objects end up nested under the generated-main-proto-java directory.
• The generated avro objects end up nested under the generated-main-avro-java directory.

To register a schema, the topic it will be associated with needs to exist. Let’s create the required topics now.

29. Go to your schema-registry-101 Kafka cluster in Confluent Cloud.
30. Click on Topics on the left hand side of the UI.
31. Click on Create topic.
32. In the Topic name box type avro-purchase.

33. Keep the default Partitions setting and click on Create with defaults.
34. Repeat the previous steps to create a new topic named proto-purchase.
35. Click on Topics again to review the topics you created.

You are now ready to create the new schemas.

36. Return to your schema-registry course project.
37. From your terminal window, run the command ./gradlew registerSchemasTask.

When the task completes you should see a BUILD SUCCESSFUL message.

Now go back to the Confluent Cloud Console to inspect the new schemas.

38. Once back in the Confluent Cloud Console click on Schema Registry at bottom on the left side of the Confluent Cloud Console.
39. When the Schema Registry view loads take note of the listed schemas under the Subject name column.

40. Click on the View and manage schemas button.
41. Click on avro-purchase-value and take note of the version and schema ID.
42. Expand the schema fields and verify it matches the purchase.avsc definition.

43. Return to the general Schemas view by clicking SCHEMAS in the breadcrumb at the top of the page.
44. Examine the proto-purchase-value schema using the previous steps taking note of the version, schema ID, and the fields.

This concludes the exercise.