Connect to Confluent Cloud with Secure Public Endpoints

Hi, I'm Justin Lee with Confluent. In this module, we're going to be looking at the ins and outs of connecting to Confluent Cloud using secure public endpoint. If you're just getting started with Confluent Cloud, you're probably going to start with a cluster that is accessed over a secure public endpoint. In fact, it's quite possible that this is the only network connectivity option you'll ever need. This is a secure production grade solution that greatly simplifies your architecture. At a very high level with a secure public endpoint, you can access your Confluent Cloud cluster from anywhere in a secure fashion. Let's talk about security. All communications from clients Security Features - Encryption in Transit whether that's Kafka applications and services, Kafka Connect Integrations, stream processing, or even just exploring Kafka with a UI or CLI is encrypted using industry standard TLS encryption. This is the same encryption used when you access your bank account or make a credit card payment online. Everything is secured in transit. Additionally, everything in Confluent Cloud requires authentication. Security Features - AuthN/AuthZ In order to talk to Kafka, you have to prove your identity using an API key and secret in order to access any data. We can also use your identity to validate that you should have access to a specific set of data. So unless you have a specific InfoSec or compliance requirement that requires complex or advanced network connectivity using a secure public endpoint may actually be both the easiest and best way to use Confluent Cloud. With a secure public endpoint, Benefits of Secure Public Endpoints we're standing up a Kafka cluster that can be accessed from anywhere on the internet. Kafka clients and services that are running on premise on a workstation or in the cloud can all access the Kafka cluster without any special network connectivity. For example, if I want to access data from an application running in AWS, I can do that regardless of where in AWS it's running, as long as my application can access the Confluent Cloud broker's public endpoint on the internet. There's several caveats to be aware of here. Because Kafka uses a TCP wire protocol, and not HTTP or HTTPS, access through an HTTPS proxy is not supported. If you're using a secure public endpoint Confluent Cloud cluster, your Kafka clients have to be able to access the internet without a proxy. At the same time, if I'm running services in my data center, I can also access Kafka as long as I have the ability to access the internet, again, without a proxy. Let's expand on this a little more. Architecture: Datacenter and/or Cloud With a secure public endpoint, your Confluent cluster is exposed on a set of public endpoints securely. You can access it from anywhere you need to access it from, whether from your cloud provider network or from your on-premise network. Secure public endpoints allow clients Architecture: Fully-Managed Connectors to connect to Confluent cloud, but there are also situations in which Confluent Cloud may need to connect back to your network. For example, if you're looking at fully managed connectors. When you're using a secure public endpoint, managed connectors run in the Confluent Cloud network. Because managed connectors are going to be accessing your data sources and your data sinks, your data sources and data sinks need to be accessible over the internet, just like Confluent Cloud is accessible over the internet. This can be done in one of two ways. If you're using a SaaS provider data source or data sink, your provider may support exposing your data source directly on the internet. For example, we see this a lot with customers using AWS S3 or Snowflake. If you're using self-managed data sources or data sinks, you'll need to work with your networking team to expose your data source to the internet in a secure way. For example, you may have a TLS-secured database that you expose on a public DNS name or IP address that can then be accessed from the Confluent network running on the internet. In addition, all of the managed connector considerations apply to all Confluent Cloud network architectures. If your data sources and data sinks can be exposed on internet endpoints, you can use fully managed connectors regardless of whether you're using public endpoints or one of the other options we'll discuss later in this course, such as a peering connection or a private linking network. If you can't expose your data sources or data sinks to the internet, another option is self-managed connectors. In this architecture, Architecture: Self-Managed Connectors you'll run a connector in your network premise. This is a data integration which lives in your network and sends data to Confluent Cloud. In the case of a self-managed source connector, you run a data integration in your network that reads from your data source and pushes or produces it up to Confluent Cloud using the Confluent Cloud secure public endpoint. In the case of a self-managed sink connector, you'll again run a data integration in your environment. This time one that consumes or reads from Confluent Cloud using the Confluent Cloud secure public endpoint and writes to your data sink. Because these self-managed connectors are running in your environment, this layout actually works regardless of your network architecture. You can do this with secure public endpoints, peered networks, or even private link. Here are a few reasons why you wouldn't be able to use secure public endpoints. In some environments services aren't allowed to access the internet directly. As a reminder, Kafka doesn't use HTTP and it doesn't use HTTPS, so it doesn't work through an HTTPS proxy. If you aren't able to access the internet directly, Why Not Secure Public Endpoints? you may end up needing to use a private endpoint. This often comes from an InfoSec requirement. Certain environments require private network connectivity for everything, and we support that in Confluent Cloud. We'll talk about this in the next few modules. Another situation we occasionally encounter is situations where customers want to use our fully managed connectors. These connectors run in the Confluent network, and when you're using a secure public endpoint Confluent Cloud cluster, our fully managed connectors can only access endpoints on the public internet. Note that as an alternative to this, if you're trying to read or write from a data source or data sink that's only accessible from within your network, you can choose to run a separate Kafka connect cluster in your data center or cloud account and configure it to read or write data to your Confluent Cloud cluster. Remember that because we're using secure public endpoints, your Confluent Cloud cluster can be accessed from anywhere that has access to the internet. With this solution, you have two clusters, a local Kafka connect cluster that's running your integrations, and a Kafka cluster that's running in Confluent Cloud. Thank you for joining us in this module. We'll see you in the next one.