Hands On: Configuring a PrivateLink Cluster

Configuring a PrivateLink Cluster

Prerequisites:

• AWS login
  • Permissions to create VPCs
  • Permissions to create EC2 instances
  • Permission to create VPC endpoints
  • Permission to create Route 53 records and hosted zones
• Confluent Cloud login
• Confluent Cloud CLI

Create a Confluent Cloud Private Link network.

1. Log in to Confluent Cloud, and navigate to the Cloud-Networking Environment.

2. Click Network Management -> Create your first network.

3. Select AWS, and the cloud provider region where you created your VPC (us-east-2 in this example).

4. Select PrivateLink, and leave the Zone Placement as is, and give the network a name.

5. The network provisioning will take a few minutes to complete.

6. Once the network is ready, click on its tile to manage it.

7. Create a PrivateLink Access.

8. Complete the details, using the account information for the VPC that you created earlier, and save the VPC Service Endpoint ID for future use.

9. Once the PrivateLink Access is provisioned, in the AWS console, configure the endpoint.

10. Edit the security group to add three inbound rules from the VPC CIDR on TCP ports 80, 443, and 9092.

Configure DNS to resolve the PrivateLink endpoints.

11. Create a Route53 Hosted Zone, using the “DNS Domain” value from the Confluent Cloud Network Management UI.

12. Create the record for all Zones endpoint; you can get the endpoint DNS names from the AWS console under the endpoint created earlier.

13. Now create records for each Zonal endpoint. This can be tricky/confusing, so take your time and make sure you have the canonical and aliased AZ ID’s somewhere handy. You can get this information from the Subnets tab of the Endpoint screen in the AWS console.

Provision a cluster and validate connectivity from your EC2 instance to Confluent Cloud.

14. Provision a multi-zone Confluent Cloud cluster.

15. While the cluster is provisioning, install the confluent CLI on your EC2 instance. You may have to configure an Elastic IP for the instance so you can access it from your workstation.

16. Test connectivity and DNS setup by issuing an NSLOOKUP and openssl commands.

Next, get some data in your cluster by creating a topic and a connector. You can only do this via the CLI on your EC2 instance unless you have a way to VPN into your VPC containing the PrivateLink endpoint.

17. First, set the environment and cluster, create a topic and API key.

18. Next, create the JSON file for the connector, an example JSON can be found here.

19. Now, create the connector using the JSON file you created as input. You can also check the status of the connector provisioning.

20. Next, test consumption from your cluster using the CLI.

Congratulations! You’ve successfully consumed data over the PrivateLink connection!

Clean up.

21. First, delete the connector.

22. Next, delete the topic and API key.

23. From the UI, delete the cluster.

24. Delete the PrivateLink access.

25. After the PrivateLink access is done deprovisioning, delete the network.

26. Next, delete the resources created in the AWS console, starting with the DNS entries and hosted zone.

27. Next, release any Elastic IPs you created, and terminate the EC2 instance as well.

28. Finally, delete the PrivateLink endpoints and the VPC itself.