AWS Transit Gateway

When Is AWS Transit Gateway Needed?

A special case of peering connections in Confluent Cloud is with AWS Transit Gateway. The Transit Gateway makes network connectivity in AWS significantly more flexible, and introduces a number of additional options.

• Remember that the cloud, in general, doesn’t support transitive peering. Clients can access services directly across a single peering connection, but if they’re peered to a VPC, they can’t reach other VPCs that are connected to that VPC; they can’t transit multiple peering connections.
  • For example, if you have a peering connection established between VPC B and VPC A and another established between VPC A and Confluent Cloud VPC, this does not allow Kafka clients in VPC B to reach the Kafka cluster in the Confluent Cloud VPC.
  • For them to do so, you have to set up an additional peering connection.
  • Basically, whenever you introduce a new VPC that needs access to Confluent Cloud, you need to peer that VPC to the Confluent Cloud VPC.
  • Say, for example, you had five VPCs; if you want all of them to be peered directly to each other, you’ll end up with ten peering connections. If you introduce a sixth network, you need an additional five peering connections (for fifteen total); basically, as you add additional networks to your infrastructure, your peering connections grow exponentially.
  • Every time you add an additional VPC to your infrastructure, you have to peer that VPC with every other VPC that you want it to be able to communicate with; as the network grows, this can become exponentially more complex. Each peering is a multi-step process, requiring you to initiate a peering connection from one VPC, and accept it from the other.
• In the context of Confluent Cloud—for every VPC you want to communicate with your Confluent Cloud network, you have to set up a peering connection between the Confluent Cloud network and your VPC, which requires configuration both on the Confluent side and on the AWS side.

What Are AWS Transit Gateway Benefits?

• AWS Transit Gateway helps you simplify the peering architecture, connect to on-premise networks, and connect to networks in other cloud providers, such as Google and Azure.
  • Rather than peering VPCs directly to each other, you can peer multiple AWS VPCs to an AWS Transit Gateway. Think of it as a cloud network router—any network connected to the Transit Gateway can access any other network connected to the Transit Gateway.
  • This eases your overall network architecture. Rather than peering every additional network to every other network in your environment (including your Confluent Cloud network), you can peer them to each other.
  • With the earlier example of five cloud networks, you just connect them all up to the same Transit Gateway; five peering connections, and they can all talk to each other. To add a sixth network, you just peer that to the Transit Gateway, and off you go!
  • Additionally, a Transit Gateway supports other types of connections. It’s much easier to connect a Transit Gateway to an on-premises network or datacenter.
  • You can even set up a VPN connection between an AWS Transit Gateway and other VPN-supporting networks—this mean you could connect an AWS Transit Gateway to Azure or Google.
• On the Confluent side, if your Confluent Cloud network is in AWS, in addition to peering that network with one or more of your VPCs, you can peer that same network with exactly one of your Transit Gateways.
  • Assuming you have a Transit Gateway provisioned in your AWS infrastructure, you can connect it to your Confluent Cloud network.
  • Because this is a special operation, this peering connection currently requires you to open a Confluent support ticket; this capability isn’t yet self-service. Simply go to the Confluent support portal and open a Transit Gateway peering ticket, and then provide information about your Transit Gateway. From there, a Confluent network engineer will initiate a peering connection with your Transit Gateway, which has to be accepted by your network team.
  • Aside from the provisioning peering step, on the Confluent side everything is the same network as a peering network.
  • We still require a non-shared /16 CIDR range to be allocated to the Confluent Cloud network. Again, the CIDR range must be part of either the RFC 1918 private address space—anything that falls under 10/8, 172.16/12, or 192.168/16; part of the RFC 6598 shared address space—that’s anything in the 100.64/10 IP range, which is usually used to connect customer networks to carrier networks; or anything in the RCF2544 address space—that’s 198.18/15, which is usually used for benchmark testing.
  • From a routing perspective, we’ll set up a default route in the Confluent Cloud network to point to the Transit Gateway—all traffic initiating from the Confluent Cloud network that isn’t peered directly with one of your VPCs will go through the Transit Gateway.
  • Given that it’s still a Confluent peering network, you follow the same process to provision the peering network in Confluent Cloud. Once the network is up, you can create one or more dedicated Confluent Cloud clusters in the network.
  • Additionally, once the network is up, you can set up network connectivity. In this case, you’re setting up a peering connection between the Confluent Cloud peering network to an AWS Transit Gateway that you own, rather than to a VPC (or VNet) that you own.
• Because this is another peering connection, Confluent supports the same bidirectional capabilities available to standard peering connection. This has many benefits as well as some drawbacks. On the plus side this means that your clients access Confluent Cloud over the peering connection, and fully managed connectors can access IP addresses in your network. This allows you to, for example, use a fully managed connector to read data from a database in your network, or sink data to a data sink in your network. We still don’t have access to your DNS, but we have access to your IP address space. As long as the data source or data sink is reachable by IP address, or is resolvable by public DNS (and as long as you set up proper firewall rules to allow the Confluent Cloud network to access your services), fully managed connectors running in Confluent Cloud can be integrated with your data sources and data sinks.
• Think of a Transit Gateway as an enhancement to peering connections in Confluent Cloud; it has all of the same abilities, it’s just more flexible.
  • For services that need to read or write to Confluent Cloud, you have more flexibility in where you run them—it’s much easier to directly connect on-premises networks, or networks in other clouds, to the Transit Gateway. You also don’t have to set up as many peering connections to Confluent Cloud; rather, you connect everything to a Transit Gateway, which acts as a cloud network router between different networks.
  • For situations where you want to utilize fully managed connectors, you also have more flexibility; with a Transit Gateway, you can grant fully managed connectors running in the Confluent Cloud cluster access to a larger part of your network, including data sources and data sinks running outside of AWS.
• If you’re running in AWS and using Transit Gateway, it provides all of the benefits of a standard peering connection, with a lot more flexibility; but there are some drawbacks:
  • The Confluent network has more access to your network than a standard peering connection. We’ll never initiate connections to your environment that you don’t set up, but in some situations this is a security concern.

Next we’ll talk about Private Link, and some of the key differences between a peering network and a Private Link network connection.