Hands On: Configuring a VPC Peered Cluster

Welcome back to the exercise section of the Confluent Cloud networking course. In this exercise, we'll be provisioning a VPC peered cluster in Confluent Cloud and peering it to our VPC in AWS. We'll then configure UI access for the cluster. We will provision a Confluent Cloud Datagen connector. And then we'll consume records over the VPC peering from our EC2 instance inside of the AWS VPC. (bright music) So let's go ahead and get started. We'll click on the cloud networking environment inside of the Confluent Cloud UI and click on the Network Management tab. We'll create a network and specify it in the AWS us-east-2 region, which corresponds to the VPC from the previous exercise that we created. We'll then enter the CIDR for Confluent Cloud to use. We'll make it 10.1.0.0/16. And hit Continue. We'll then give the network a name. We'll just call it the VPC peer network. And hit Create. The network will take a couple of minutes to provision. Once it's done being provisioned, it will turn orange. It'll say there are no active connections. We'll go ahead and add a connection in the form of a VPC peering. We'll give it a name. We'll call it the Network Course Peering connection. We'll get the AWS account number from the AWS management console. And copy it back into Confluent Cloud. And we'll grab the AWS VPC ID of the VPC that we created in the previous exercise, the networking-course-vpc. So we'll go ahead select it, view its details, and grab the ID there. And paste it back into the cloud UI. And then we'll also grab the CIDR from AWS. And paste it in. And we'll click Add. Now, the peering itself is being provisioned. Once it's ready to be accepted from the AWS side, we'll go ahead and click on Peering Connections. You may have to refresh this for the peering connection to show up. And there it is. We can see that it's pending acceptance. So we'll go ahead and select it. And accept the request. And confirm. So we are now peered with our Confluent Cloud network that we provisioned with our VPC that we provisioned in AWS. So the next step is to set up some routes from our AWS VPC back to Confluent Cloud. So we'll search for the networking course route tables. And we'll pick the public route table. And we'll go ahead and add a route. So we'll click on Routes, Edit Route. And then add a route to our Confluent Cloud network CIDR, which is 10.1.0.0/16. And we'll route all traffic for those IP addresses to the peering that we just accepted. And then hit Save. Now that we've updated the routes and successfully completed the peering process, we should see that the network course peering is showing its status as ready, which means we're good to go. So now we can go ahead and provision our cluster. We'll click on the Clusters tab and hit Add Cluster, and then we'll provision a dedicated cluster. We can leave it single zone for now, and we won't deal with any of the security options. We'll just give it a name of VPC Peered Cluster. And hit Launch. This provisioning process might take a little while, so this is a good opportunity to take a break, grab a cup of coffee, and come back once the cluster is provisioned. Once the cluster is provisioned, we'll go ahead and click into it. And once we click into it, we'll notice that we'll get an error message saying that we're unable to access this cluster. So we'll go ahead and fix this now. We're going to set up a proxy inside of our EC2 instance to be able to proxy traffic from our local computer to the cluster. So we'll start by installing an Nginx proxy to our EC2 instance using apt. And now that Nginx is installed, we'll go ahead and configure it. So we edit the configuration file. We'll go ahead and add a couple of lines to configure a proxy. You can find this information in the Confluent Cloud documentation, and that documentation is linked in the error message that we saw in the Confluent Cloud UI. Once we've added our configuration to Nginx, we just need to restart it and then validate the status. We'll go ahead and issue a sudo systemctl restart of Nginx. And then view the status, also using systemctl. And we can see that everything's running, good to go. Next, we'll need to edit the security group for our EC2 instance to allow 443 traffic from our local machine to the proxy. So we'll switch over to the AWS management console. Go to the EC2 dashboard. Select our instance. And then go to Security tab. Look at our security groups. Open it up and edit the inbound rules. And we'll add a rule for HTTPS 443 and from our IP. And save the rules. After we've saved the rules, now we need to go ahead and edit our local hosts file to be able to resolve the IP address of our proxy whenever our computer tries to request the DNS name of the Confluent Cloud cluster. So we'll go in the EC2 console and actually grab the public IP address that we had attached to the instance earlier. So we'll grab that under the instance summary. And we'll go over to our terminal. We'll be doing this on our local machines. Be sure not to do this inside of your EC2 instance, but rather on your local laptop. So on Mac, this is under etc/hosts. And we'll add a line in this file with the IP address that we copied and the URL from the error message in Confluent Cloud. Paste that. And we'll go ahead and save changes. And now that we've done these steps, we should be able to access the cluster through the Confluent Cloud UI. This may take a couple of minutes to work. You may wanna try to refresh the Confluent Cloud UI a couple times. But generally, the error message should go away pretty quickly, as you can see here. So now we can access things like the Topics UI and the Connectors UI. So we'll go ahead and create a topic. So we'll go ahead and click Create Topic. Give the topic a name of "clickstream" and create with defaults. Next, we'll go ahead and provision the connector. Go to Data Integration, Connectors, select our Datagen Source connector again. Select our clickstream topic, hit Continue. Again, provision a global access API key and secret pair and give it a quick description. We hit Continue. Select the Clickstream template and the JSON output format and leave it at one task. Go ahead and give the connector a name. Hit Continue. And again, this may take a couple minutes to provision. Once the provisioning is complete, we'll go back to our terminal. Log back into our EC2 instance, and now we'll consume the records over our VPC peered connection. So we'll go ahead, issue a confluent login. Log into the CLI using our email and password. And we'll go ahead and issue a confluent environment list and a confluent environment use to set the Confluent Cloud networking course environment as the default. And we'll then set the VPC peer cluster as the default, so we'll do a cluster list and a cluster use with our newly created VPC peer cluster. And then we'll store the credentials that we generated in the connector creation step here in the CLI. So we'll do a confluent API key store, give the resource as the Kafka cluster ID, and then we'll paste in the key and secret here to the terminal. And now that we've stored the key, we'll set the key as the default using the confluent API key use command. Again, specifying the cluster ID as the resource. And now, we're ready to consume. It's the moment of truth. We'll issue a confluent kafka topic consume command to our clickstream topic. And we should see messages coming across the pipe. So we've successfully completed another Confluent Cloud networking course exercise. In this exercise, we went ahead and created a VPC peered network. We created a peering on that network. We then provisioned a cluster on the network. We configured all of the appropriate routes. We set up a proxy for the UI. And we then consumed data that was being produced by a Datagen connector through the Confluent Cloud CLI over our peering connection from our EC2 instance that we provisioned. So the only thing left to do is to go ahead and clean up. So we'll go ahead into the Confluent Cloud UI. Delete the connector first by going to Data Integration, Connectors, selecting our connector, and hitting Delete. We'll confirm the deletion. We'll then go ahead and delete the API key, and confirm that as well. And finally, we'll go delete the cluster by going to Cluster Overview, Cluster Settings, and delete the cluster. Next up, we'll need to go and clean up our network, so we'll go to the Network Management tab. Go to our VPC peer network. Go ahead and delete the peering connection itself first. And then we can delete the network by hitting Delete Network and entering the network ID. Now that all the Confluent Cloud resources have been deleted, we can go ahead and clean up in the AWS console. We'll go ahead and go to the VPC dashboard, and we will see that the peering connection here should show as deleted now. It'll disappear after a couple hours on its own. There's no action for you to take. What action you do have to take is you need to fix the route table that now has a route that goes to nowhere. So we'll click on Route Tables, search for the route table for the public part of our VPC, edit the routes, remove the black hole, and save it. Last but not least, we'll revert the changes that we made to our hosts file earlier by deleting the single line that we added with the IP and host name of our Confluent Cloud cluster. And that's all there is to it. In the next exercise, we'll be provisioning a private link cluster over multiple availability zones. See you there.