1. SCENARIO
BU1 and BU2 were able to communicate with each other as of Friday EOB (i.e end of LAB2).
However, the network team received a complaint from BU1 Frontend Team that the connectivity with BU2 Mobile App was no longer working.

2. TROUBLESHOOT REQUEST:
-
Verify that the connectivity between BU1 Frontend and BU2 Mobile App is actually broken.
-
SSH to BU1 Frontend and launch ping/ssh towards BU2 Mobile App.
-

-
Check whether the concerned Spokes have the relevant routes or not.
HINT: Go to CoPilot > Cloud Fabric > Gateways > Spoke Gateways > select the relevant gateway > Gateway Routes and search for the subnet 10.1.212.0/24, where BU2 Mobile App resides.

From the outcome above, it is evident that Spoke1 in AWS has the destination route in his RTB.
-
Use Diagnostics tools by clicking on Spoke1 Gateway in AWS (identify the Aviatrix logo in the topology. Do not select a normal EC2 instance!) and try to ping/traceroute the instance behind the other spoke.
HINT: Go to CoPilot > Coud Fabric > Topology then select the relevant GW, click on Tools and then click on Gateway Diagnostics.


-
Try to ping both workloads from the Transit GW in AWS.

-
Check if the relevant Spoke VPCs are inspected by FireNet.
HINT: Go to Controller > FIREWALL NETWORK > Policy

-
Try to ping the LAN interface of the FW from the Transit FireNet Gateway.
HINT: first you need to retrieve the LAN Interface IP address of the FW.
Go to Controller > FIREWALL NETWORK > List > select the Transit Firenet GW, under the Firenet TAB > click on Details > Diagnostics > then click on Run and then on Show.

Scroll through the whole configuration till the ACE-FW section: the IP is beside the string “lan_private_ip”.

-
lWORKAROUND] Try to find a way to exclude traffic from being sent to the FW.
HINT: Go to Controller > FIREWALL NETWORK > List > Firenet tab > select the Transit FireNet GW and click on DETAILS.

Scroll down till the Network List Excluded From East-West Inspection section and add the source subnet 10.1.211.0/24, as depicted below.

Try to ping once again the BU2 Mobile App from the B1 Frontend. This time the ping will be successfull thanks to the workaround!

-
Remove the workaround previously applied and have a look at the dashboard on the main page of the Controller.

You will notice a clear yellow alarm on the Trasit FireNet widget! Click on the widget, then select the Transit Firenet GW and click on DETAILS.

This is a clear outcome that there is a problem on the FW. Ultimately, the firewall is not reachable.
-
Log into the FW (refer to your POD portal for the credentials for logging in) and explore its configuration.


HINT: Go to FW > Network > Interfaces and click the LAN interface (port2) and fix the issue!
The port was disabled to simulate the failure of the GW...

- Relaunch the ping from BU1 Frontend towards BU2 Mobile App.
