Confluent Cloud Overview

Hi, I'm Justin Lee with Confluent. In this module, we're going to talk about components and services that are going to be connecting with Confluent Cloud as well as common deployment patterns that enterprises use when doing so. Let's talk about Confluent Cloud. There are two main ways to interact with Confluent Cloud. One is our data plane and the second is our control plane. The data plane is where your data lives in Confluent Cloud. The control plane is where a lot of management, maintenance, and operations take place. Let's look at the data plane first, the place where your data lives in Confluent Cloud. The data plane is made up of those services that actually store your business data and it falls into two categories. In the data plane, we run a fully managed schema registry which is accessed over the internet. Also in the data plane, we have a concept of a Confluent network. This consists of the Confluent cluster itself as well as any fully managed ksqlDB instances or connectors that you may be running. These are the services that are affected by your network configuration. Choosing one of the options we describe later in this course, such as secure public endpoint or a peering connection or private link connection all will affect these services. Most resources in Confluent Cloud are accessed by something in your environment. This is always a one-way communication. For example, your Kafka client apps will access the Confluent cluster using the Kafka wire protocol, which is TCP, and access the schema registry using HTTPS. Managed connectors are a little bit different. These are integrations that will need access to your data sources and data sinks. If you want to run fully managed connectors, the network will have to be configured so that the connectors running in the Confluent network can access your data services. Now let's look at the control plane. The control plane is where a lot of management, maintenance, and operations take place. Through these internet accessible endpoints, you can do the following. Provision and manage Confluent Cloud, including all the resources in the data plane. You can monitor the health and performance of your Confluent Cloud resources. Additionally, you can track access to Confluent Cloud resources via audit logs. One quick thing to call out. Data Plane and Control Plane In order to see data in the Confluent Cloud UI, you need access to both the data plane and the control plane. If you want to read or write data into a Kafka topic in the Confluent UI, for example, you'll need to run your client, which is usually a browser, somewhere that has access both to the Confluent data plane and the Confluent control plane. Topology Before we delve into the details of cloud networking options, it will help to go over some of the topologies that are seen as enterprises expand into the cloud. Your organization may have a topology similar to one of these. Topology A can be construed as a one-time migration or steady state condition. The steady state use case is where data from on-premises is continuously sent to the cloud. With topology B, data is sent in both directions between on-premises and cloud deployments. With topology C, we capture a scenario where data centers cater to local markets and the cloud installations are used to aggregate the data either for global views or global features. With topology D, we involve multi-region or multi-cloud deployments for business continuity, disaster recovery, or for cloud agnostic platform deployments. Avoiding cloud lock-in and vendor management are other reasons often cited for multi-cloud deployments. In addition, mergers and acquisitions may force the adoption and maintenance of infrastructures that span multiple clouds. In the modules that follow, we will examine several use cases that map to these patterns. Network Architecture As you make decisions around your network architecture, here are some things you should be thinking about. First, what are we connecting? What types of clients, applications and services are going to be interacting with Confluent Cloud? Do we have to worry about multi-cloud or multi-region architectures? How are we going to handle replication? What types of data infrastructure are we going to be integrating with Confluent? Do we have self-hosted data sources or self-hosted data sinks? How are we going to handle third-party data, SaaS or PaaS providers? And do we need to be concerned about that? Next, where are we connecting from? Are our clients running on premise? Are they running in the cloud? Do we have users that need to be able to access Confluent data from corporate networks or even from their home offices? Do we need to be able to access data from across different clouds? Last of all, how do we want to connect to Confluent Cloud? Do we have requirements for private networking? As you can see, there are a lot of different permutations of what, where, and how. The dotted lines here show one such permutation. You may have source data in a local data center that you want to connect to Confluent Cloud and Private Link is a preferred networking option after considering all factors. Connectivity Options In the rest of this course, you'll learn how to answer the how to connect question given your particular scenarios. So what are our connectivity options? At a high level, we have four options for connecting to Confluent Cloud. The first option is secure public endpoints. You can access Confluent Cloud via a secure public endpoint. These are public IP addresses that are accessed over the internet. When you're doing this, everything is encrypted in transit with TLS 1.2. The second option is VPC or VNet peering. You can directly connect or peer the Confluent network with your Cloud network. When you're doing this, traffic will never traverse the public internet and will use the cloud provider's private back plane. The third option is AWS Transit Gateway. In AWS, this is an extension of peering where the Confluent network is peered to one of your transit gateways. This effectively acts as a Cloud router. You can connect multiple VPCs and remote networks using a single Transit Gateway. And the fourth option is private link. This is a one-way communication between your network and Confluent Cloud and it only allows connections to be initiated from your network to us. Public Endpoints The modules that follow will look at each of these in more detail. Secure public endpoints are the default way of connecting and are the easiest to use. VPC peering or VNet peering is common across all public cloud providers but comes with several drawbacks. Private Link and Transit Gateway provide more powerful capabilities but aren't available in all providers. Outro After we've covered these connectivity options, the last module will revisit the what, where, and how questions from the previous slide and we will be in a much better position to evaluate them and choose the option that best fits our requirements. Thanks for listening, we'll see you in the next module.