Get Started Free
‹ Back to courses
course: Confluent Cloud Networking

AWS Transit Gateway

8 min
justin-lee

Justin Lee

Staff Solutions Engineer (Presenter)

When Is AWS Transit Gateway Needed?

aws-transit-gateway A special case of peering connections in Confluent Cloud is with AWS Transit Gateway. The Transit Gateway makes network connectivity in AWS significantly more flexible, and introduces a number of additional options.

  • Remember that the cloud, in general, doesn’t support transitive peering. Clients can access services directly across a single peering connection, but if they’re peered to a VPC, they can’t reach other VPCs that are connected to that VPC; they can’t transit multiple peering connections.
    • For example, if you have a peering connection established between VPC B and VPC A and another established between VPC A and Confluent Cloud VPC, this does not allow Kafka clients in VPC B to reach the Kafka cluster in the Confluent Cloud VPC.
    • For them to do so, you have to set up an additional peering connection.
    • Basically, whenever you introduce a new VPC that needs access to Confluent Cloud, you need to peer that VPC to the Confluent Cloud VPC.
    • Say, for example, you had five VPCs; if you want all of them to be peered directly to each other, you’ll end up with ten peering connections. If you introduce a sixth network, you need an additional five peering connections (for fifteen total); basically, as you add additional networks to your infrastructure, your peering connections grow exponentially.
    • Every time you add an additional VPC to your infrastructure, you have to peer that VPC with every other VPC that you want it to be able to communicate with; as the network grows, this can become exponentially more complex. Each peering is a multi-step process, requiring you to initiate a peering connection from one VPC, and accept it from the other.
  • In the context of Confluent Cloud—for every VPC you want to communicate with your Confluent Cloud network, you have to set up a peering connection between the Confluent Cloud network and your VPC, which requires configuration both on the Confluent side and on the AWS side.

What Are AWS Transit Gateway Benefits?

transit-gateway-benefits

  • AWS Transit Gateway helps you simplify the peering architecture, connect to on-premise networks, and connect to networks in other cloud providers, such as Google and Azure.
    • Rather than peering VPCs directly to each other, you can peer multiple AWS VPCs to an AWS Transit Gateway. Think of it as a cloud network router—any network connected to the Transit Gateway can access any other network connected to the Transit Gateway.
    • This eases your overall network architecture. Rather than peering every additional network to every other network in your environment (including your Confluent Cloud network), you can peer them to each other.
    • With the earlier example of five cloud networks, you just connect them all up to the same Transit Gateway; five peering connections, and they can all talk to each other. To add a sixth network, you just peer that to the Transit Gateway, and off you go!
    • Additionally, a Transit Gateway supports other types of connections. It’s much easier to connect a Transit Gateway to an on-premises network or datacenter.
    • You can even set up a VPN connection between an AWS Transit Gateway and other VPN-supporting networks—this mean you could connect an AWS Transit Gateway to Azure or Google.
  • On the Confluent side, if your Confluent Cloud network is in AWS, in addition to peering that network with one or more of your VPCs, you can peer that same network with exactly one of your Transit Gateways.
    • Assuming you have a Transit Gateway provisioned in your AWS infrastructure, you can connect it to your Confluent Cloud network.
    • Because this is a special operation, this peering connection currently requires you to open a Confluent support ticket; this capability isn’t yet self-service. Simply go to the Confluent support portal and open a Transit Gateway peering ticket, and then provide information about your Transit Gateway. From there, a Confluent network engineer will initiate a peering connection with your Transit Gateway, which has to be accepted by your network team.
    • Aside from the provisioning peering step, on the Confluent side everything is the same network as a peering network.
    • We still require a non-shared /16 CIDR range to be allocated to the Confluent Cloud network. Again, the CIDR range must be part of either the RFC 1918 private address space—anything that falls under 10/8, 172.16/12, or 192.168/16; part of the RFC 6598 shared address space—that’s anything in the 100.64/10 IP range, which is usually used to connect customer networks to carrier networks; or anything in the RCF2544 address space—that’s 198.18/15, which is usually used for benchmark testing.
    • From a routing perspective, we’ll set up a default route in the Confluent Cloud network to point to the Transit Gateway—all traffic initiating from the Confluent Cloud network that isn’t peered directly with one of your VPCs will go through the Transit Gateway.
    • Given that it’s still a Confluent peering network, you follow the same process to provision the peering network in Confluent Cloud. Once the network is up, you can create one or more dedicated Confluent Cloud clusters in the network.
    • Additionally, once the network is up, you can set up network connectivity. In this case, you’re setting up a peering connection between the Confluent Cloud peering network to an AWS Transit Gateway that you own, rather than to a VPC (or VNet) that you own.
  • Because this is another peering connection, Confluent supports the same bidirectional capabilities available to standard peering connection. This has many benefits as well as some drawbacks. On the plus side this means that your clients access Confluent Cloud over the peering connection, and fully managed connectors can access IP addresses in your network. This allows you to, for example, use a fully managed connector to read data from a database in your network, or sink data to a data sink in your network. We still don’t have access to your DNS, but we have access to your IP address space. As long as the data source or data sink is reachable by IP address, or is resolvable by public DNS (and as long as you set up proper firewall rules to allow the Confluent Cloud network to access your services), fully managed connectors running in Confluent Cloud can be integrated with your data sources and data sinks.
  • Think of a Transit Gateway as an enhancement to peering connections in Confluent Cloud; it has all of the same abilities, it’s just more flexible.
    • For services that need to read or write to Confluent Cloud, you have more flexibility in where you run them—it’s much easier to directly connect on-premises networks, or networks in other clouds, to the Transit Gateway. You also don’t have to set up as many peering connections to Confluent Cloud; rather, you connect everything to a Transit Gateway, which acts as a cloud network router between different networks.
    • For situations where you want to utilize fully managed connectors, you also have more flexibility; with a Transit Gateway, you can grant fully managed connectors running in the Confluent Cloud cluster access to a larger part of your network, including data sources and data sinks running outside of AWS.
  • If you’re running in AWS and using Transit Gateway, it provides all of the benefits of a standard peering connection, with a lot more flexibility; but there are some drawbacks:
    • The Confluent network has more access to your network than a standard peering connection. We’ll never initiate connections to your environment that you don’t set up, but in some situations this is a security concern.

Next we’ll talk about Private Link, and some of the key differences between a peering network and a Private Link network connection.

Use the promo code NETWORKING101 to get $25 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.

AWS Transit Gateway

Hi, I'm Justin Lee with Confluent. In this module, we're going to look at how AWS Transit Gateway can be used to connect with Confluent Cloud. Why use AWS Transit Gateway A special case of peering connections in Confluent Cloud is with AWS Transit Gateway. The Transit Gateway makes network connectivity in AWS significantly more flexible and introduces a number of additional options. Remember that the cloud in general doesn't support transitive peering. Clients can access services directly across a single peering connection. But if they're peered to a VPC, they can't reach other VPCs that are connected to that VPC, specifically, they can't transit multiple peering connections. For example, if I have a peering connection established between VPC B and VPC A, and another one established between VPC B and the Confluent Cloud network, this does not allow Kafka clients in VPC A to reach the Kafka cluster in the Confluent Cloud network. For Kafka clients in VPC A to access Confluent Cloud I have to set up a direct peering connection from VPC A to Confluent Cloud. Basically, whenever I introduce a new VPC that needs access to Confluent Cloud, I need to peer that VPC to the Confluent Cloud VPC. Regular peering connections are one to one. Say for example, we have five VPCs. If we want them all to communicate with each other, we'll have 10 peering connections. One for each of them to talk to all the others. If we want to introduce a sixth VPC, then we need to add five additional peering connections. As we add additional networks to our infrastructure, our peering connection counts grow exponentially. Transit Gateway changes this. Every time you add an additional VPC to your infrastructure, you have to peer that VPC with every other VPC that you want it to be able to communicate with. As the network grows, this can become exponentially more complex. Each peering is often a multi-step process requiring you to initiate a peering connection from one VPC and accept it from the other. In the context of Confluent Cloud, for every VPC you want to communicate with your Confluent Cloud network, if you're using direct peering connections, you have to set up a peering connection between the Confluent Cloud network and your VPC. This requires configuration both on the Confluent side managed by you and on the AWS side managed by your infrastructure team. How to use AWS Transit Gateway AWS specifically has a mechanism to make this whole process easier. Enter the Transit Gateway. Transit Gateways help simplify the peering architecture. They allow you to connect a single Transit Gateway to other AWS VPCs as well as on-premise networks and other cloud providers directly. Rather than peering VPCs directly to each other, you can peer multiple AWS VPCs to a single Transit Gateway. Think of it as a cloud network router, any network connected to the Transit Gateway can access any other network also connected to the Transit Gateway. This eases our overall network architecture. Rather than peering every additional network to every other network in our environment including our Confluent Cloud network, we can peer all of them to a single Transit Gateway. With our earlier example of five cloud networks, we just connect all of them to the same Transit Gateway; five peering connections and they can all talk to each other through the Transit Gateway. To add a sixth network, we just peer that to the Transit Gateway and away we go. Additionally, a Transit Gateway supports other types of connections. It's much easier to connect a Transit Gateway to an on-premise network or a data center. You can also set up VPN connections between an AWS Transit Gateway and other VPN supporting networks. This means that you can connect an AWS Transit Gateway to Azure or Google directly. On the Confluent side, if your Confluent Cloud network is in AWS, in addition to peering that network with one or more VPCs, you can peer that same network with exactly one of your Transit Gateways. Assuming you have a Transit Gateway provisioned in your AWS infrastructure, you can connect it to your Confluent Cloud network. Because this is a special operation, this peering connection currently requires you to open a ticket. This capability is not yet self-service. You can go to the Confluent Support Portal and open a Transit Gateway peering ticket and then provide information about your Transit Gateway. From there, a Confluent network engineer will initiate a peering connection to your Transit Gateway. This has to be accepted by your networking team. Aside from the differences in provisioning, from the Confluent side, it's the same network as a peering network. Everything aside from the provisioning peering process is the same. We still require a non-shared /16 CIDR range to be allocated to the Confluent Cloud network. Again, the CIDR range must be either part of the RFC 1918 private address space, part of the 6598 shared address space, or part of the RFC 2544 benchmark address space. From a routing perspective, we'll set up a default route from the Confluent Cloud network to point at the Transit Gateway. All traffic initiating from the Confluent Cloud network that isn't peered directly with one of your VPCs will go through the Transit Gateway. Given that it's still a Confluent peering network, you follow the same process to provision the peering network in Confluent Cloud. Once that network is up, you can create one or more dedicated Confluent Cloud clusters in the network. Additionally, once the network is up, you can set up network connectivity. In this case, you're setting up a peering connection between the Confluent Cloud peering network to an AWS Transit Gateway that you own rather than to a VPC that you own. Because this is another peering connection, we support the same bidirectional capabilities available to standard peering connections. This has both benefits and drawbacks. On the plus side, this means that your clients can access Confluent Cloud over the peering connection and fully managed connectors can access IP addresses in your network. This allows you to, for example, use a fully managed connector to read data from a database in your network or sink data to a data sink in your network. We still don't have access to your DNS but we have access to your IP address space. As long as the data source or data sink is reachable by IP address, or is resolvable by public DNS, fully managed connectors running in Confluent Cloud can be integrated with your data sources and data sinks. Benefits of AWS Transit Gateway Think of a Transit Gateway as an enhancement to peering connections in Confluent Cloud. It has all of the same abilities, it's just more flexible. For services that need to read or write to Confluent Cloud, you have much more flexibility in where you run them. It's much easier to directly connect to on-premise networks or to networks in other clouds with the Transit Gateway. You also don't have to set up as many peering connections to Confluent Cloud. Rather you connect everything to the Transit Gateway which acts as a cloud network router between different networks. For situations where you want to utilize fully managed connectors, you also have more flexibility. With a Transit Gateway, you can grant fully managed connectors running in the Confluent Cloud network access to a larger part of your network, including data sources and data sinks running outside of AWS. If you're running in AWS and using Transit Gateway that provides all of the benefits of a standard peering connection with a lot more flexibility, but there are some drawbacks. The Confluent network has more access to your network than a standing peering connection. We can access anything that's connected to your Transit Gateway. We'll never initiate connections to your environment that you don't set up but your InfoSec team has to be okay with this. The last drawback with AWS Transit Gateway is that it requires a ticket to actually set up the peering connection. This is not yet self-service. Thank you for joining us, we'll see you in the next module.