AWS Transit Gateway

Hi, I'm Justin Lee with Confluent. In this module, we're going to look at how AWS Transit Gateway can be used to connect with Confluent Cloud. Why use AWS Transit Gateway A special case of peering connections in Confluent Cloud is with AWS Transit Gateway. The Transit Gateway makes network connectivity in AWS significantly more flexible and introduces a number of additional options. Remember that the cloud in general doesn't support transitive peering. Clients can access services directly across a single peering connection. But if they're peered to a VPC, they can't reach other VPCs that are connected to that VPC, specifically, they can't transit multiple peering connections. For example, if I have a peering connection established between VPC B and VPC A, and another one established between VPC B and the Confluent Cloud network, this does not allow Kafka clients in VPC A to reach the Kafka cluster in the Confluent Cloud network. For Kafka clients in VPC A to access Confluent Cloud I have to set up a direct peering connection from VPC A to Confluent Cloud. Basically, whenever I introduce a new VPC that needs access to Confluent Cloud, I need to peer that VPC to the Confluent Cloud VPC. Regular peering connections are one to one. Say for example, we have five VPCs. If we want them all to communicate with each other, we'll have 10 peering connections. One for each of them to talk to all the others. If we want to introduce a sixth VPC, then we need to add five additional peering connections. As we add additional networks to our infrastructure, our peering connection counts grow exponentially. Transit Gateway changes this. Every time you add an additional VPC to your infrastructure, you have to peer that VPC with every other VPC that you want it to be able to communicate with. As the network grows, this can become exponentially more complex. Each peering is often a multi-step process requiring you to initiate a peering connection from one VPC and accept it from the other. In the context of Confluent Cloud, for every VPC you want to communicate with your Confluent Cloud network, if you're using direct peering connections, you have to set up a peering connection between the Confluent Cloud network and your VPC. This requires configuration both on the Confluent side managed by you and on the AWS side managed by your infrastructure team. How to use AWS Transit Gateway AWS specifically has a mechanism to make this whole process easier. Enter the Transit Gateway. Transit Gateways help simplify the peering architecture. They allow you to connect a single Transit Gateway to other AWS VPCs as well as on-premise networks and other cloud providers directly. Rather than peering VPCs directly to each other, you can peer multiple AWS VPCs to a single Transit Gateway. Think of it as a cloud network router, any network connected to the Transit Gateway can access any other network also connected to the Transit Gateway. This eases our overall network architecture. Rather than peering every additional network to every other network in our environment including our Confluent Cloud network, we can peer all of them to a single Transit Gateway. With our earlier example of five cloud networks, we just connect all of them to the same Transit Gateway; five peering connections and they can all talk to each other through the Transit Gateway. To add a sixth network, we just peer that to the Transit Gateway and away we go. Additionally, a Transit Gateway supports other types of connections. It's much easier to connect a Transit Gateway to an on-premise network or a data center. You can also set up VPN connections between an AWS Transit Gateway and other VPN supporting networks. This means that you can connect an AWS Transit Gateway to Azure or Google directly. On the Confluent side, if your Confluent Cloud network is in AWS, in addition to peering that network with one or more VPCs, you can peer that same network with exactly one of your Transit Gateways. Assuming you have a Transit Gateway provisioned in your AWS infrastructure, you can connect it to your Confluent Cloud network. Because this is a special operation, this peering connection currently requires you to open a ticket. This capability is not yet self-service. You can go to the Confluent Support Portal and open a Transit Gateway peering ticket and then provide information about your Transit Gateway. From there, a Confluent network engineer will initiate a peering connection to your Transit Gateway. This has to be accepted by your networking team. Aside from the differences in provisioning, from the Confluent side, it's the same network as a peering network. Everything aside from the provisioning peering process is the same. We still require a non-shared /16 CIDR range to be allocated to the Confluent Cloud network. Again, the CIDR range must be either part of the RFC 1918 private address space, part of the 6598 shared address space, or part of the RFC 2544 benchmark address space. From a routing perspective, we'll set up a default route from the Confluent Cloud network to point at the Transit Gateway. All traffic initiating from the Confluent Cloud network that isn't peered directly with one of your VPCs will go through the Transit Gateway. Given that it's still a Confluent peering network, you follow the same process to provision the peering network in Confluent Cloud. Once that network is up, you can create one or more dedicated Confluent Cloud clusters in the network. Additionally, once the network is up, you can set up network connectivity. In this case, you're setting up a peering connection between the Confluent Cloud peering network to an AWS Transit Gateway that you own rather than to a VPC that you own. Because this is another peering connection, we support the same bidirectional capabilities available to standard peering connections. This has both benefits and drawbacks. On the plus side, this means that your clients can access Confluent Cloud over the peering connection and fully managed connectors can access IP addresses in your network. This allows you to, for example, use a fully managed connector to read data from a database in your network or sink data to a data sink in your network. We still don't have access to your DNS but we have access to your IP address space. As long as the data source or data sink is reachable by IP address, or is resolvable by public DNS, fully managed connectors running in Confluent Cloud can be integrated with your data sources and data sinks. Benefits of AWS Transit Gateway Think of a Transit Gateway as an enhancement to peering connections in Confluent Cloud. It has all of the same abilities, it's just more flexible. For services that need to read or write to Confluent Cloud, you have much more flexibility in where you run them. It's much easier to directly connect to on-premise networks or to networks in other clouds with the Transit Gateway. You also don't have to set up as many peering connections to Confluent Cloud. Rather you connect everything to the Transit Gateway which acts as a cloud network router between different networks. For situations where you want to utilize fully managed connectors, you also have more flexibility. With a Transit Gateway, you can grant fully managed connectors running in the Confluent Cloud network access to a larger part of your network, including data sources and data sinks running outside of AWS. If you're running in AWS and using Transit Gateway that provides all of the benefits of a standard peering connection with a lot more flexibility, but there are some drawbacks. The Confluent network has more access to your network than a standing peering connection. We can access anything that's connected to your Transit Gateway. We'll never initiate connections to your environment that you don't set up but your InfoSec team has to be okay with this. The last drawback with AWS Transit Gateway is that it requires a ticket to actually set up the peering connection. This is not yet self-service. Thank you for joining us, we'll see you in the next module.