Visibility into network traffic is extremely important to cloud operations. Access to evidential data is critical to pinpoint an issue, reduce MTTR and to prove innocence. Everyday we see customers who end up wasting countless cycles to troubleshoot basic issues in cloud. The lack of visibility and ability to gather evidential data from native tools makes it extremely hard for operations teams to do their job.
Aviatrix platform offers Aviatrix CoPilot which is our premier day2 operations tool focusing on visualization, trending, analytics and reporting network and security events in cloud networks. Aviatrix CoPilot collects several metrics along with flow data from Aviatrix data plane and provides and easy to use, historical way of searching, visualizing and analyzing network traffic in cloud. This data allows customers to produce and share historical and current evidential data which reduces MTTR and allows better and faster interaction with CSPs and partner organizations.
Native cloud constructs, such as VPC Flow logs are very expensive to use and we often see that customers only turn them ON on-demand for troubleshooting hence missing any historic data. In addition, they only provide very basic information such as source dest ip:port and security group action. For ex, VPC Flow logs will show you following:
${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
2 673665204127 eni-057059ee1f68fa76d 10.101.103.78 116.0.48.18 33668 80 6 15 868 1614199401 1614199459 ACCEPT OK
Example Scenario:
As an example, you just received an abuse report from AWS. In the report, AWS indicates there is a host from your VPC doing SYN attacks to a webserver on the internet. The onus is on you to prove and provide evidential data tha you are not doing a SYN attack. In order to avoid blocking of services or account suspension, you have to provide this evidential data to AWS within 24 hours.
What’s a SYN attack:
Let’s take a look what is a SYN-Attack, SYN-Attack is a form of DDoS, which an attacker generates a lot of TCP SYN packets (the first packet for establishing TCP session), without completing the TCP handshake (no ACK packet). Here is a diagram shows how TCP handshake works:
A SYN-Attack does 2 things. First, it floods session tables of all stateful devices like FWs, IDS, IPS, Load-balancers along the path of an attack, and server itself. It depletes session tables very quickly and cause performance degradation. In addition, all stateful devices have limit on session setup rate (a very CPU intensive process), if the rate of an attack exceeds session setup rate, legit traffic will start getting dropped.
Reproducing the problem in your lab:
Disclaimer: USE this tool ONLY to PRIVATE IP only.
One of very popular testing tools is hping3. Here is how to install it on an Ubuntu host:
# sudo apt update -y
# sudo apt install hping3 -y
To generate TCP SYN flood on port 80 to my-lab-web-server-private-ip:
#sudo hping3 -S --flood -V -p 80 my-lab-web-server-private-ip
Here is how a SYN Attack looks like (TCP Flag: SYN):
Now let’s take a look how Aviatrix CoPilot software can help providing evidential data with some of the attributes that’s NOT in CSP’s flow logs (one of them is TCP flag, a very important indicator for many types of DDoS attack).
I created a web server (40.83.170.249) on the internet and generated a lot of HTTP traffic from an AWS host (10.11.4.75) with a legitimate traffic pattern that will trigger an abuse report.
Solution – Collecting Evidential Data in minutes:
Here is how you can prove it’s a false-positive in a few minutes with 2 easy steps utilizing extensive searching capability of Aviatrix CoPilot’s FlowIQ feature.
In Aviatrix CoPilot, Click on FlowIQ -> Overview Tab -> Set the Date/Time Range -> Add Source Address, Destination Address and Destination Port -> Click on APPLY CHANGES:
Click on RECORDS -> Click on EDIT COLUMNS -> Check TCP Flag Tags
Now you can see:
The flow traverses 2 Aviatrix GWs (54.177.59.26 & 13.52.58.87). To prove this is a false-positive, we have SYN (to start the session), ACK (to complete TCP 3-way handshake), PSH (getting the data), and FIN (to properly close the session). SYN-Attack contains ONLY SYN, which can’t be found in any of the sessions we have.
In addition, if you want, you can also use Aviatrix to take packet capture of the traffic in a matter of seconds without requiring a SPAN.
and you can open it with Wireshark:
Conclusion:
Evidential data is very important to network operation, able to search/sort quickly makes resolving issues much easier and faster. Aviatrix platform not only simplifies networking in cloud but it also brings unprecedented visibility and operational efficiency in cloud.
If you want to do a quick Aviatrix test drive, you can follow the link below to build your own sandbox environment: