Analyze SYN Flood by Using Netstat
What is SYN flood ?
A SYN flood (half-open attack) is a denyal-of-service (DDoS) attack which exploits the way of TCP 3-Way Handshake by repeteadly sending initial connection request (SYN) packets then overhelm all available ports on the targeted server. It can also cause SYN backlog at Linux to be full which can cause memory to be full too.
Because we will discuss backlog at netstat, we need to discuss SYN and Accept Queue at Linux.
SYN and Accept Queue
SYN Queue is used to store inbound SYN packets. It is also responsible to send SYN+ACK and resend when timeout. After transmitting SYN+ACK to client, SYN Queue waits for an ACK from the client. The received ACK should match against the fully established connection table. After SYN Queue match, the kernel removes the item from SYN Queue then create full fledged connection and adds it to the Accept Queue. [1]
About SYN+ACK resend when timeout, it's number is regulated by net.ipv4.tcp_synack_retries. Based on the description of tcp_syn_ack_retries:
tcp_synack_retries = INTEGER Number of times SYNACKs for a passive TCP connection attempt will be retransmitted. Should not be higher than 255. Default value is 5, which corresponds to 31 seconds till the last retransmission with the current initial RTO of 1second. With this the final timeout for a passive TCP connection will happen after 63 seconds.
net.ipv4.tcp_synack_retries have default 5 retries after when there is timeout after sending SYN+ACK.
Accept Queue is used to store fully established connections. When a process calls accept(), the sockets are de-queued and passed to the application. [1]
The SYN Queue cap can be configured by changing the kernel variable net.ipv4.tcp_max_syn_backlog. Nowadays net.core.somaxconn caps both SYN Queue and Accept Queue.
SYN Flood
At Linux System Level, SYN Flood attack will cause SYN Queue to be full which can cause all TCP packets to be denied. Because each slot in SYN Queue use some memory, which is 256 bytes of memory in kernel 4.14, so it can cause memory to be fully occupied.
There is some mitigation that can be done at kernel variable, which are:
-
net.ipv4.tcp_syncookies
This is used to prevent against the common
SYN flood attack. It is used to send out syncookies when syn backlog queue is full becauseSYN Flood attack. Default value is 1. -
net.ipv4.tcp_max_syn_backlog
Maximal number of remembered connection requests, which have not received an acknowledgment from the client (SYN Queue). We can change this to a greater value if we see our backlog is full based on netstat.
-
net.core.somaxconn
This is a caps for SYN Queue and Accept Queue. The condition for increasing this value is the same as
net.ipv4.tcp_max_syn_backlog. -
net.ipv4.tcp_synack_retries
The number of SYN+ACK retries sended after timeout. We need to decrease this value to reduce too many SYN+ACK retries in the event of SYN Flood. Because there will be no ACK from the client, so there will be too many SYN received and SYN+ACK sended from SYN+ACK retries. If we reduce this value, we can reduce the load at our TCP socket.
Analyze SYN Flood from netstat
If we try to execute netstat -an, we will see this output:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:57621 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:34499 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN We can see Send-Q in column three. So what is Send-Q ? Based on man of netstat, Send-Q is the count of bytes not acknowledged by the remote host. Since kernel 2.6.18, this column contains the maximum size of the SYN backlog. So it is basically the same as SYN backlog.
Then how can we use the value of Send-Q to decide if there is SYN flood?
First you need to know which port is attacked. As the attacked port in my real case is 443, so i need to filter port 443 from netstat.
You can analyze the Send-Q usage by using this command netstat -n | awk '$4 ~ ":443"' | awk '{print $3}' | sort | uniq -c | sort -nr.
This is the sample from my real case attack:
netstat -n | awk '$4 ~ ":443"' | awk '{print $3}' | sort | uniq -c | sort -nr
775 0
194 1
28 40
22 64
5 83689
5 4486
4 83690
4 79165
3 79161
3 69300
2 83688
2 79166
2 544
2 333
2 19131
2 1229
2 1228
1 8480
1 84243
1 84242
1 83692
1 82024
1 80646
1 79716
1 79714
1 79167
1 79163
1 79160
1 79159
1 79158
1 78240
1 77918
1 77917
1 72850
1 71196
1 70478
1 69852
1 69298
1 69297
1 69191
1 69148
1 65803
1 65545
1 6527
1 61890
1 61790
1 60343
1 58864
1 58724
1 568
1 5296
1 5262
1 50935
1 48
1 37130
1 37126
1 35933
1 35
1 34014
1 27171
1 25275
1 24048
1 23740
1 21469
1 21331
1 21174
1 19133
1 17400
1 1566
1 15223
1 126
1 1235
1 1233
1 1230
1 10029As the output above shows, we can see quite many Send-Q with value higher than 1500 so we can regard all the request above is SYN flood.
If we want to block all the suspected IP, we can also use this script:
domain=[ATTACKED_DOMAIN]
checksynflood=$(netstat -n | awk '$4 ~ ":443"' | awk '{print $3,$5}' | sort | uniq -c | sort -k 2 -nr)
if [[ $(echo -e "$checksynflood" | awk '$2 >= 1500' | wc -l) -ge 80 ]]
then
ip_attackers=$(tail -n 5000 /var/log/nginx/access.log | grep -E $(echo -e "$checksynflood" | head -n 100 | awk '{print $3}' | cut -d":" -f1 | grep -oP "^(\d{1,3}\.){3}\d{1,3}" | sed "s#^#^#g" | tr "\n" "|" | sed "s#|\$##g") | grep $domain | awk '{print $1}' | sort -u | tr "\n" "," | sed "s#,\$##g")
[ -n "$ip_attackers" ] && iptables -A INPUT -s $ip_attackers -p tcp -j REJECT
fiAt the script above, i compare netstat result with access.log of nginx because i am using nginx. You can customize the script based on your need too.
That is all what i can share, i hope the knowledge from my real case can help you to understand how to analyze SYN flood attack by using netstat.
[1] Marek, M., "SYN Packet Handling in The Wild", CloudFlare, 2018,[Online]. Available: https://blog.cloudflare.com/syn-packet-handling-in-the-wild/