Disclaimer
AWS keeps changing the limits and design option from time to time. For most accurate and up to date information please consult with the AWS documentation and links provided in this article
Direct Connect (DX)
- DX is region specific offering
- It allows On-Prem physical locations to connect to a specific AWS region/location
- DX supports max of 50 VIFs (including Private and Public) per physical connection
- DX does not support Transit VIF for AWS-TGW connectivity
DXGW
- What is DXGW?
- Only supports Private and Transit VIFs
- DXGW mainly used to access private resources in VPCs
- Does not support public VIF
- DXGW does not provide any benefit of Public Internet Connectivity
- VGW associated with a DXGW must be “attached” to a VPC
- Does not support transitive routing or transit connectivity
- VPC in Region-1 cannot directly communicate with VPC in Region-2
- DX Location-1 cannot directly communicate with DX Location-2
- Up to 30 DX physical connections can connect to one single DXGW for physical link redundancy purposes
- In another words 30 DX locations/regions
- DX supports max of 50 VIFs (for DXGW only Private and Transit VIFs are applicable)
- It means one can have Max of 50 DXGW per physical DX link
- But one DXGW can connect to max of 10 VPCs
- It means Max of 500 VPCs (50 x 10 VPC) per physical DX link across accounts and regions
DXGW is Must for AWS-TGW
- Transit VIF is a must when terminating DirecConnect (DX) circuit on AWS-TGW
- But Transit VIF can only be attached to a DXGW
- That means AWS-TGW mandates deploying DXGW
Max of 3 AWS-TGW Behind a Direct Connect Circuit
- Max of 3 AWS-TGW can be attached to one DXGW behind one Transit VIF
- And only one Transit VIF is possible per DirectConnect circuit
- Aviatrix Transit does not have this limitation because it uses Private VIF
Transit VIF and Private VIF are not allowed on same DXGW
A single DXGW cannot attach to both Private and Transit VIF
One cannot attach a DX-GW to an AWS-TGW when the DX-GW is already associated with an AWS-VGW or is attached to a Private VIF.
I did a simple test in my lab, and I get an error when I try to connect a Private VIF to DX-GW. This DX-GW had a Transit VIF attached to it.
Also confirmed this from following AWS doc
https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-transit-gateways.html
DXGW with and without AWS-TGW Comparison
DXGW without AWS-TGW | DXGW with AWS-TGW |
10 VPCs per DXGW | 3 TGWs per DXGW |
50 DXGW max (b/c of 50 Private VIF) | With Transit VIF only one DXGW is possible |
500 VPCs total | 5,000 VPCs per TGW 15,000 VPC per DX physical link |
Private VIF supported on all Direct Connect connection types | Transit VIF supported only on dedicated or hosted connections of speed 1Gbps and above |
No additional charges | Additional charge for TGW data processing |
DXGW with AWS-TGW Routing Limitations
* Only 20 routes from AWS to On-Prem per AWS-TGW
* Only 100 routes from on-prem to AWS
Reference:
Transit Gateway Reference Architectures for Many VPCs NET406-R1 PDF
Transit Gateway Reference Architectures for Many VPCs NET406-R1 VoD
Intra-Region AWS-TGW Peering is not Allowed
When multiple AWS Transit Gateways are required in the same region (separation between prod/dev air gap, separate NGFWs or other reasons), inter-region peering cannot be used to route traffic between VPCs attached to the AWS Transit Gateways.
Two AWS Transit Gateways can only be peered when they are in different regions.
Aviatrix Transit Solution
- Works with the Private VIF and does not need Public or Transit VIF
- Does not need any DXGW
- One can deploy as many Aviatrix Transit GW as per the business need
- Aviatrix Transit can extent to AWS-TGW if needed as shown in the following diagram
Summary
- Transit VIF can only be attached to a DXGW
- Only one Transit VIF for any AWS Direct Connect 1/2/5/10 Gbps connection
- Less than 1G connections does not support Transit VIF
- Max of 3 AWS-TGW can connect to one DXGW behind one Transit VIF
- A single DXGW cannot attach with both Private and Transit VIF
- This could be a serious limitation for some customers
- I think the underline assumption is that if a customer is already using AWS-TGW then why would he want to use a private VIF attached to the same DXGW?
- Aviatrix Transit Solution is not bound to these limits
AWS References
https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-limits.html
https://docs.aws.amazon.com/directconnect/latest/UserGuide/limits.html