Hands On: Configuring a PrivateLink Cluster

Welcome to the final exercise of the Confluent Cloud networking course. In this exercise, we'll be provisioning a PrivateLink cluster inside of Confluent Cloud. We'll do this by provisioning a network, some PrivateLink access, provisioning the actual cluster itself. We will then go into our AWS console, create some VPC endpoints and a private hosted zone, and then we'll go ahead and produce to the cluster using a fully managed Datagen connector and then consume using the Confluent CLI from our EC2 instance inside of our AWS VPC. So let's go ahead and get started. From the Confluent Cloud UI, let's click on the cloud networking environment and then click on the Network Management tab. We'll create a network and provision it in AWS us-east-2 region. We'll then select the PrivateLink networking type with all availability zones and give the network a name. We'll click Create and wait for the network to be provisioned. This may take a few moments. Once it's ready, it'll turn orange. We can click into it and provision a PrivateLink access. We'll give the PrivateLink access a name and specify our AWS account number where our VPC resides. So we'll copy and paste the account number. And then we'll copy the PrivateLink service address for use in a later step. Again, provisioning this PrivateLink access may take a couple of minutes. Once it shows a status of ready, we can go back to the AWS console, go to Endpoints, and create an endpoint. We'll give the endpoint a name. And select the service category as a PrivateLink Ready partner service. Then, paste the service name from the previous step and verify that it's valid, which it is. And then we'll put this endpoint inside of our networking course VPC. Next, we'll select all availability zones for this VPC endpoint, and we'll select all of the public subnets for each zone. We'll set the IP address type as IPV4. We'll select the default security group and edit its inbound rules and add rules for traffic over Port 9092. Which is the Kafka protocol port for Confluent Cloud. And we'll allow the source range as the CIDR range of our AWS VPC. We'll also add a rule for HTTPS. This will allow for UI management traffic to traverse the private link. And then finally, we'll add a rule for HTTP as well. Next, we'll save the rules and go back to the VPC endpoint wizard and click Create. Now that the endpoint is provisioned, we can go ahead and provision a cluster. We'll go to the Cluster tab, click Add Cluster, and we'll go ahead and select a dedicated cluster and then select the availability type as multi-zone. We'll set security as automatic. And then give the cluster a name. We're provisioning this cluster as a multi availability zone cluster to show some of the advanced features and some of the more advanced setup that's required for these types of clusters. This cluster provisioning may take a while, so we'll go ahead and configure some of the other items needed for a PrivateLink connection while the cluster's provisioning. So we'll go ahead and copy the network overview section and put it in a notepad for reference a little bit later. And now we'll go take a look at our VPC endpoint. Copy the Subnets section and also put it in that notepad for reference. This will allow us to relate the availability zones to the canonical names. So here you can see us-east-2c relates to use2-az3. That'll be useful when we're creating our zonal endpoints later. Next, we'll copy the DNS names for each of the VPC endpoints, the main endpoint as well as the zonal endpoints, and put it on our notepad again for when we need to create our DNS entries. So now we'll go and switch over to route 53 and create our private hosted zone. We'll go ahead and click on Hosted Zones. Click Create Hosted Zone. And enter the domain name as the DNS domain from our network overview. Next, we'll create a description for the hosted zone. We'll call it the Networking Course Zone and configure it as a private hosted zone. We'll choose our region, us-east-2, and the VPC ID of our networking course VPC and hit Create. Now that the zone is created, we'll add some CNAME records. First, we'll add a record for the main VPC endpoint DNS. So we'll grab that from our notepad and paste that in. And then we'll set the TTL to one minute. And we'll add another record. The next record will be the zonal endpoint record for our first availability zone. So we'll put in *.use2-az1. And then based on the reference between the canonical names and the AZs that we got from the AWS console previously, that we put in our notepad, we can see that the use2-az1 corresponds to the VPC endpoint for us-east-2a. Again, we'll configure the TTL and then add an additional record for the second AZ. So this will be *.use2-az2. Make sure it's a CNAME record. And we'll put in the VPC zonal endpoint for US-east-2-B. These steps can be tricky, so be sure to double check your work and make sure that all of the AZs that you are specifying here match up to the correct zonal endpoint. So az3 corresponds to east-2c. And we've set our TTL. So now we can create the records, and we can review them here. So at this point, we will go back to the Confluent Cloud UI and wait for the PrivateLink cluster to complete provisioning. Now that the cluster is finished provisioning, let's go ahead and click into it. And at the cluster overview, we should get another error very similar to what we got for the VPC peer cluster. So we'll go ahead and fix this error. We'll go back to our Amazon EC2 instance. Go ahead and SSH. And then we will edit our Nginx config file. And we only need to change one setting in here. Under the server block, we need to change the resolver from 1.1.1.1 to 127.0.0.53. And this will basically allow Nginx to resolve the VPC endpoints that we created in our private hosted zone in route 53 earlier. So now that we've updated that configuration, we'll go ahead and restart Nginx and then take a look at its status and make sure that it's running. And everything looks good, so now we just need to update our local machine hosts file. So we'll go to the AWS console. Jump back over to EC2. Get the public IP address of our EC2 instance. And copy it here. And then we'll go back. Edit our hosts file. Again, on Mac, this is under etc/hosts. And we'll add a line for the public IP address of our EC2 instance and the endpoint URL that is displayed in the error message. We'll save this. And now once we go back to Confluent Cloud and refresh the page, the error will disappear. So, awesome. Now that we've configured the UI access, now we can go and try to test it out by creating a topic. So we'll click on Topics. Click Create Topic. And as usual, we'll give it a name of "clickstream." And create with the defaults. Next, we'll create a connector. So we'll go ahead, add a Datagen connector. Use the clickstream topic. Generate a global API key secret pair. Select the Clickstream template and the JSON output format. Leave it at one task; hit Continue. And we'll give the connector a name. And we'll give this a couple minutes to provision. Now that the provisioning is complete, we'll go back over to our terminal, log into our EC2 instance, and try to consume some records. So we'll log to the Confluent CLI. Again, using our email address and password for the credentials. And then set the default environment and default cluster, like we have in the previous exercises using the environment list, environment use, and cluster list and cluster use commands. Next, we'll store the API key that we generated when we created the connector. So we'll do the api-key store command. Give the resource ID of the cluster. Add our key and secret. And now we'll set it as the default, so we'll use the api-key use command. Give the key and the resource ID. And now we should be ready to consume messages over our PrivateLink connection, so we'll go ahead and issue a topic consume on the clickstream topic, and we can see messages flowing through. So at this point, we have successfully completed the exercise. Now, let's review what we completed during the exercise. First, we created a network of type PrivateLink inside of Confluent Cloud. We then created a PrivateLink access to our AWS account. Inside of AWS, we then provisioned VPC endpoints to tie back to the PrivateLink access on the Confluent side. Next, we provisioned a multi-availability zone PrivateLink dedicated Confluent Cloud cluster. While the cluster was provisioning, we created a private hosted zone in Amazon route 53 to resolve the DNS records for our VPC endpoints. Next, we configured UI access for the Confluent Cloud console using Nginx and our hosts file. Finally, we created a fully managed Datagen Source connector and then consumed the records over the PrivateLink connection using our Confluent CLI from our EC2 instance in AWS. So now that we've completed all of the exercises let's go ahead and clean up all of the resources. First, let's clean up the Datagen connector by clicking on the connector name and deleting it, and then entering its name to confirm the deletion. We'll then delete the API key. Again, confirming the deletion. And next, we'll delete our dedicated cluster. We'll click Delete cluster, enter its name to confirm the deletion. Next, we'll delete the actual network and the PrivateLink Access. So we'll go to Network Management. We'll click in our network. We'll then delete the PrivateLink Access, confirming with the PrivateLink Access ID. And then delete the network. Now that we've deleted all the Confluent Cloud resources, we'll switch over to our AWS console. And we'll first start off by deleting our private hosted that we created in route 53. So we'll go to Hosted Zones, find the hosted zone for the networking course. We'll delete all of the CNAME records that we created. Just validate that's correct. And then click Delete Zone. We'll confirm the deletion. And the zone was successfully deleted. Next, we'll go ahead and clean up our elastic IP. So we'll look up elastic IPs. And go ahead and disassociate the IP address. And then release it. Now that we've released the IP, we can go ahead and terminate our EC2 instance. And next, we'll go ahead and clean up some of the VPC resources. So we'll go ahead and delete our VPC endpoints. Confirm the deletion. And this deletion process may take a couple of minutes, so feel free to stretch your legs for a second and come back. Now that the endpoints are successfully deleted, we can go ahead and delete the VPC itself. And confirm the deletion. And that's all there is to it. We have successfully completed the exercises for this course. If you have any additional questions or concerns, I encourage you to consult the Confluent documentation as well as our knowledge base. Both of these resources are a great source of information for troubleshooting and configuring private networking in Confluent Cloud. Happy streaming.