AWS Direct Connect (DX) Direct Connect Gateway (DXGW) and AWS-TGW Design and Limitations

Disclaimer

AWS keeps changing the limits and design option from time to time. For most accurate and up to date information please consult with the AWS documentation and links provided in this article

Direct Connect (DX)

• DX is region specific offering
  
  
  
  • It allows On-Prem physical locations to connect to a specific AWS region/location
  
  


• DX supports max of 50 VIFs (including Private and Public) per physical connection


• DX does not support Transit VIF for AWS-TGW connectivity

DXGW

• What is DXGW?
  
  
  
  • According to AWS "Direct Connect gateway is a grouping of virtual private gateways (VGWs)"
  
  


• Only supports Private and Transit VIFs
  
  
  
  • DXGW mainly used to access private resources in VPCs
  
  


• Does not support public VIF
  
  
  
  • DXGW does not provide any benefit of Public Internet Connectivity
  
  


• VGW associated with a DXGW must be “attached” to a VPC


• Does not support transitive routing or transit connectivity
  
  
  
  • VPC in Region-1 cannot directly communicate with VPC in Region-2
  
  
  • DX Location-1 cannot directly communicate with DX Location-2
  
  


• Up to 30 DX physical connections can connect to one single DXGW for physical link redundancy purposes
  
  
  
  • In another words 30 DX locations/regions
  
  


• DX supports max of 50 VIFs (for DXGW only Private and Transit VIFs are applicable)
  
  
  
  • It means one can have Max of 50 DXGW per physical DX link
  
  
  • But one DXGW can connect to max of 10 VPCs
  
  
  • It means Max of 500 VPCs (50 x 10 VPC) per physical DX link across accounts and regions
  
  

DXGW is Must for AWS-TGW

1. Transit VIF is a must when terminating DirecConnect (DX) circuit on AWS-TGW


2. But Transit VIF can only be attached to a DXGW


3. That means AWS-TGW mandates deploying DXGW

Max of 3 AWS-TGW Behind a Direct Connect Circuit

1. Max of 3 AWS-TGW can be attached to one DXGW behind one Transit VIF


2. And only one Transit VIF is possible per DirectConnect circuit


3. Aviatrix Transit does not have this limitation because it uses Private VIF

Transit VIF and Private VIF are not allowed on same DXGW

A single DXGW cannot attach to both Private and Transit VIF

One cannot attach a DX-GW to an AWS-TGW when the DX-GW is already associated with an AWS-VGW or is attached to a Private VIF.

I did a simple test in my lab, and I get an error when I try to connect a Private VIF to DX-GW. This DX-GW had a Transit VIF attached to it.

Also confirmed this from following AWS doc

https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-transit-gateways.html

DXGW with and without AWS-TGW Comparison

DXGW with AWS-TGW Routing Limitations

* Only 20 routes from AWS to On-Prem per AWS-TGW

* Only 100 routes from on-prem to AWS

Reference:

Transit Gateway Reference Architectures for Many VPCs NET406-R1 PDF

Transit Gateway Reference Architectures for Many VPCs NET406-R1 VoD

<br />

Intra-Region AWS-TGW Peering is not Allowed

When multiple AWS Transit Gateways are required in the same region (separation between prod/dev air gap, separate NGFWs or other reasons), inter-region peering cannot be used to route traffic between VPCs attached to the AWS Transit Gateways.

Two AWS Transit Gateways can only be peered when they are in different regions.

Aviatrix Transit Solution

• Works with the Private VIF and does not need Public or Transit VIF


• Does not need any DXGW


• One can deploy as many Aviatrix Transit GW as per the business need
  
  
  
  • AWS-TGW is limited by 5 per region
  
  
  • AWS-TGW Intra Region Peering is not allowed
  
  


• Aviatrix Transit can extent to AWS-TGW if needed as shown in the following diagram

Summary

• Transit VIF can only be attached to a DXGW


• Only one Transit VIF for any AWS Direct Connect 1/2/5/10 Gbps connection


• Less than 1G connections does not support Transit VIF


• Max of 3 AWS-TGW can connect to one DXGW behind one Transit VIF


• A single DXGW cannot attach with both Private and Transit VIF
  
  
  
  • This could be a serious limitation for some customers
  
  
  • I think the underline assumption is that if a customer is already using AWS-TGW then why would he want to use a private VIF attached to the same DXGW?
  
  


• Aviatrix Transit Solution is not bound to these limits

AWS References

https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-limits.html
https://docs.aws.amazon.com/directconnect/latest/UserGuide/limits.html