Get Started Free
‹ Back to courses
course: Confluent Cloud Networking

Hands On: Configuring a PrivateLink Cluster

18 min

Dennis Wittekind

Customer Success Technical Architect (Presenter)


  • AWS login
    • Permissions to create VPCs
    • Permissions to create EC2 instances
    • Permission to create VPC endpoints
    • Permission to create Route 53 records and hosted zones
  • Confluent Cloud login
  • Confluent Cloud CLI

Create a Confluent Cloud Private Link network.

  1. Log in to Confluent Cloud, and navigate to the Cloud-Networking Environment.


  1. Click Network Management -> Create your first network.


  1. Select AWS, and the cloud provider region where you created your VPC (us-east-2 in this example).


  1. Select PrivateLink, and leave the Zone Placement as is, and give the network a name.



  1. The network provisioning will take a few minutes to complete.


  1. Once the network is ready, click on its tile to manage it.


  1. Create a PrivateLink Access.


  1. Complete the details, using the account information for the VPC that you created earlier, and save the VPC Service Endpoint ID for future use.


  1. Once the PrivateLink Access is provisioned, in the AWS console, configure the endpoint.




  1. Edit the security group to add three inbound rules from the VPC CIDR on TCP ports 80, 443, and 9092.


Configure DNS to resolve the PrivateLink endpoints.

  1. Create a Route53 Hosted Zone, using the “DNS Domain” value from the Confluent Cloud Network Management UI.



  1. Create the record for all Zones endpoint; you can get the endpoint DNS names from the AWS console under the endpoint created earlier.


  1. Now create records for each Zonal endpoint. This can be tricky/confusing, so take your time and make sure you have the canonical and aliased AZ ID’s somewhere handy. You can get this information from the Subnets tab of the Endpoint screen in the AWS console.



Provision a cluster and validate connectivity from your EC2 instance to Confluent Cloud.

  1. Provision a multi-zone Confluent Cloud cluster.






  1. While the cluster is provisioning, install the confluent CLI on your EC2 instance. You may have to configure an Elastic IP for the instance so you can access it from your workstation.


  1. Test connectivity and DNS setup by issuing an NSLOOKUP and openssl commands.


Next, get some data in your cluster by creating a topic and a connector. You can only do this via the CLI on your EC2 instance unless you have a way to VPN into your VPC containing the PrivateLink endpoint.

  1. First, set the environment and cluster, create a topic and API key.


  1. Next, create the JSON file for the connector, an example JSON can be found here.


  1. Now, create the connector using the JSON file you created as input. You can also check the status of the connector provisioning.


  1. Next, test consumption from your cluster using the CLI.


Congratulations! You’ve successfully consumed data over the PrivateLink connection!

Clean up.

  1. First, delete the connector.


  1. Next, delete the topic and API key.


  1. From the UI, delete the cluster.


  1. Delete the PrivateLink access.


  1. After the PrivateLink access is done deprovisioning, delete the network.

  2. Next, delete the resources created in the AWS console, starting with the DNS entries and hosted zone.



  1. Next, release any Elastic IPs you created, and terminate the EC2 instance as well.




  1. Finally, delete the PrivateLink endpoints and the VPC itself.



Use the promo code NETWORKING101 to get $25 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.

Hands On: Configuring a PrivateLink Cluster

Welcome to the final exercise of the Confluent Cloud networking course. In this exercise, we'll be provisioning a PrivateLink cluster inside of Confluent Cloud. We'll do this by provisioning a network, some PrivateLink access, provisioning the actual cluster itself. We will then go into our AWS console, create some VPC endpoints and a private hosted zone, and then we'll go ahead and produce to the cluster using a fully managed Datagen connector and then consume using the Confluent CLI from our EC2 instance inside of our AWS VPC. So let's go ahead and get started. From the Confluent Cloud UI, let's click on the cloud networking environment and then click on the Network Management tab. We'll create a network and provision it in AWS us-east-2 region. We'll then select the PrivateLink networking type with all availability zones and give the network a name. We'll click Create and wait for the network to be provisioned. This may take a few moments. Once it's ready, it'll turn orange. We can click into it and provision a PrivateLink access. We'll give the PrivateLink access a name and specify our AWS account number where our VPC resides. So we'll copy and paste the account number. And then we'll copy the PrivateLink service address for use in a later step. Again, provisioning this PrivateLink access may take a couple of minutes. Once it shows a status of ready, we can go back to the AWS console, go to Endpoints, and create an endpoint. We'll give the endpoint a name. And select the service category as a PrivateLink Ready partner service. Then, paste the service name from the previous step and verify that it's valid, which it is. And then we'll put this endpoint inside of our networking course VPC. Next, we'll select all availability zones for this VPC endpoint, and we'll select all of the public subnets for each zone. We'll set the IP address type as IPV4. We'll select the default security group and edit its inbound rules and add rules for traffic over Port 9092. Which is the Kafka protocol port for Confluent Cloud. And we'll allow the source range as the CIDR range of our AWS VPC. We'll also add a rule for HTTPS. This will allow for UI management traffic to traverse the private link. And then finally, we'll add a rule for HTTP as well. Next, we'll save the rules and go back to the VPC endpoint wizard and click Create. Now that the endpoint is provisioned, we can go ahead and provision a cluster. We'll go to the Cluster tab, click Add Cluster, and we'll go ahead and select a dedicated cluster and then select the availability type as multi-zone. We'll set security as automatic. And then give the cluster a name. We're provisioning this cluster as a multi availability zone cluster to show some of the advanced features and some of the more advanced setup that's required for these types of clusters. This cluster provisioning may take a while, so we'll go ahead and configure some of the other items needed for a PrivateLink connection while the cluster's provisioning. So we'll go ahead and copy the network overview section and put it in a notepad for reference a little bit later. And now we'll go take a look at our VPC endpoint. Copy the Subnets section and also put it in that notepad for reference. This will allow us to relate the availability zones to the canonical names. So here you can see us-east-2c relates to use2-az3. That'll be useful when we're creating our zonal endpoints later. Next, we'll copy the DNS names for each of the VPC endpoints, the main endpoint as well as the zonal endpoints, and put it on our notepad again for when we need to create our DNS entries. So now we'll go and switch over to route 53 and create our private hosted zone. We'll go ahead and click on Hosted Zones. Click Create Hosted Zone. And enter the domain name as the DNS domain from our network overview. Next, we'll create a description for the hosted zone. We'll call it the Networking Course Zone and configure it as a private hosted zone. We'll choose our region, us-east-2, and the VPC ID of our networking course VPC and hit Create. Now that the zone is created, we'll add some CNAME records. First, we'll add a record for the main VPC endpoint DNS. So we'll grab that from our notepad and paste that in. And then we'll set the TTL to one minute. And we'll add another record. The next record will be the zonal endpoint record for our first availability zone. So we'll put in *.use2-az1. And then based on the reference between the canonical names and the AZs that we got from the AWS console previously, that we put in our notepad, we can see that the use2-az1 corresponds to the VPC endpoint for us-east-2a. Again, we'll configure the TTL and then add an additional record for the second AZ. So this will be *.use2-az2. Make sure it's a CNAME record. And we'll put in the VPC zonal endpoint for US-east-2-B. These steps can be tricky, so be sure to double check your work and make sure that all of the AZs that you are specifying here match up to the correct zonal endpoint. So az3 corresponds to east-2c. And we've set our TTL. So now we can create the records, and we can review them here. So at this point, we will go back to the Confluent Cloud UI and wait for the PrivateLink cluster to complete provisioning. Now that the cluster is finished provisioning, let's go ahead and click into it. And at the cluster overview, we should get another error very similar to what we got for the VPC peer cluster. So we'll go ahead and fix this error. We'll go back to our Amazon EC2 instance. Go ahead and SSH. And then we will edit our Nginx config file. And we only need to change one setting in here. Under the server block, we need to change the resolver from to And this will basically allow Nginx to resolve the VPC endpoints that we created in our private hosted zone in route 53 earlier. So now that we've updated that configuration, we'll go ahead and restart Nginx and then take a look at its status and make sure that it's running. And everything looks good, so now we just need to update our local machine hosts file. So we'll go to the AWS console. Jump back over to EC2. Get the public IP address of our EC2 instance. And copy it here. And then we'll go back. Edit our hosts file. Again, on Mac, this is under etc/hosts. And we'll add a line for the public IP address of our EC2 instance and the endpoint URL that is displayed in the error message. We'll save this. And now once we go back to Confluent Cloud and refresh the page, the error will disappear. So, awesome. Now that we've configured the UI access, now we can go and try to test it out by creating a topic. So we'll click on Topics. Click Create Topic. And as usual, we'll give it a name of "clickstream." And create with the defaults. Next, we'll create a connector. So we'll go ahead, add a Datagen connector. Use the clickstream topic. Generate a global API key secret pair. Select the Clickstream template and the JSON output format. Leave it at one task; hit Continue. And we'll give the connector a name. And we'll give this a couple minutes to provision. Now that the provisioning is complete, we'll go back over to our terminal, log into our EC2 instance, and try to consume some records. So we'll log to the Confluent CLI. Again, using our email address and password for the credentials. And then set the default environment and default cluster, like we have in the previous exercises using the environment list, environment use, and cluster list and cluster use commands. Next, we'll store the API key that we generated when we created the connector. So we'll do the api-key store command. Give the resource ID of the cluster. Add our key and secret. And now we'll set it as the default, so we'll use the api-key use command. Give the key and the resource ID. And now we should be ready to consume messages over our PrivateLink connection, so we'll go ahead and issue a topic consume on the clickstream topic, and we can see messages flowing through. So at this point, we have successfully completed the exercise. Now, let's review what we completed during the exercise. First, we created a network of type PrivateLink inside of Confluent Cloud. We then created a PrivateLink access to our AWS account. Inside of AWS, we then provisioned VPC endpoints to tie back to the PrivateLink access on the Confluent side. Next, we provisioned a multi-availability zone PrivateLink dedicated Confluent Cloud cluster. While the cluster was provisioning, we created a private hosted zone in Amazon route 53 to resolve the DNS records for our VPC endpoints. Next, we configured UI access for the Confluent Cloud console using Nginx and our hosts file. Finally, we created a fully managed Datagen Source connector and then consumed the records over the PrivateLink connection using our Confluent CLI from our EC2 instance in AWS. So now that we've completed all of the exercises let's go ahead and clean up all of the resources. First, let's clean up the Datagen connector by clicking on the connector name and deleting it, and then entering its name to confirm the deletion. We'll then delete the API key. Again, confirming the deletion. And next, we'll delete our dedicated cluster. We'll click Delete cluster, enter its name to confirm the deletion. Next, we'll delete the actual network and the PrivateLink Access. So we'll go to Network Management. We'll click in our network. We'll then delete the PrivateLink Access, confirming with the PrivateLink Access ID. And then delete the network. Now that we've deleted all the Confluent Cloud resources, we'll switch over to our AWS console. And we'll first start off by deleting our private hosted that we created in route 53. So we'll go to Hosted Zones, find the hosted zone for the networking course. We'll delete all of the CNAME records that we created. Just validate that's correct. And then click Delete Zone. We'll confirm the deletion. And the zone was successfully deleted. Next, we'll go ahead and clean up our elastic IP. So we'll look up elastic IPs. And go ahead and disassociate the IP address. And then release it. Now that we've released the IP, we can go ahead and terminate our EC2 instance. And next, we'll go ahead and clean up some of the VPC resources. So we'll go ahead and delete our VPC endpoints. Confirm the deletion. And this deletion process may take a couple of minutes, so feel free to stretch your legs for a second and come back. Now that the endpoints are successfully deleted, we can go ahead and delete the VPC itself. And confirm the deletion. And that's all there is to it. We have successfully completed the exercises for this course. If you have any additional questions or concerns, I encourage you to consult the Confluent documentation as well as our knowledge base. Both of these resources are a great source of information for troubleshooting and configuring private networking in Confluent Cloud. Happy streaming.