I’ve been busy tuning Varnish to prevent outages and that makes me take a much closer look at things and how to get through things quickly especially when it comes to working with farms of servers. I noticed in my cacti graphs a huge rise in FIN_WAIT_1 connection states on my FreeBSD servers to a degree of 25x the number of established connections so I decided to do some work to see what’s going. What I came to I think will provide some nice insight for many different scenarios.
First how I made my determination of my ratios, I know what connection states I’m looking so I create an array with those values, I know that it’s the 6th field and I’d like a count of where each stands. This is the oneliner
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done
I added the carriage return to clean up the output
767 FIN_WAIT_1
103 FIN_WAIT_2
866 TIME_WAIT
54 ESTABLISHED
220 SYN_RCVD
7 LAST_ACK
Now that I have that I want to know who’s connected, how many connections they have by connection state this oneliner takes care of that
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do echo -e "\n" $a; netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}' |awk '{freq[$0]++} END {for (x in freq) {print freq[x], x }}' |sort -nr;done
and that produces an output like this
FIN_WAIT_1
627 67.198.71.114
390 64.90.159.102
264 89.241.213.178
221 174.108.76.131
71 186.32.54.114
FIN_WAIT_2
5 66.66.133.117
5 65.0.139.23
5 173.169.40.34
5 166.147.120.16
4 96.246.240.75
TIME_WAIT
98 10.10.10.20
68 10.10.10.22
60 10.10.10.23
60 10.10.10.24
49 10.10.10.25
ESTABLISHED
13 186.32.54.114
9 98.255.88.48
7 64.90.159.102
6 70.208.78.19
5 98.204.42.34
SYN_RCVD
22 108.45.28.123
2 12.160.201.226
1 98.255.88.48
LAST_ACK
6 71.215.198.128
2 66.104.42.194
1 80.58.205.48
1 41.79.120.19
1 121.120.116.205
Some of these gave me a hard time so let’s take a closer look at some of these
I’ve covered the gotchas from using a bash array variable in an earlier blog so I won’t cover that again. What I did discover is awk unkowingly modifies the output. I didn’t understand why I was getting this strange frequency output until I broke down what was happening and that’s when I saw what awk was doing
netstat -n |awk '($6 = /FIN_WAIT_1/)' |head -n 5
tcp4 0 532 10.10.10.169.80 173.175.47.91.55257 1
tcp4 0 532 10.10.10.169.80 173.175.47.91.55256 1
tcp4 0 533 10.10.10.169.80 108.45.28.123.49944 1
tcp4 0 0 10.10.10.169.80 113.197.8.162.41647 1
tcp4 0 0 10.10.10.169.80 113.197.8.162.2879 1
UH? I thought maybe it was how netstat behaved and I just never noticed
netstat -n |grep FIN_WAIT_1 |head -n 5
tcp4 0 24606 10.10.10.169.80 199.224.118.221.62975 FIN_WAIT_1
tcp4 0 3579 10.10.10.169.80 74.12.184.206.1323 FIN_WAIT_1
tcp4 0 533 10.10.10.169.80 173.175.47.91.57498 FIN_WAIT_1
tcp4 0 533 10.10.10.169.80 173.175.47.91.57497 FIN_WAIT_1
tcp4 0 533 10.10.10.169.80 173.175.47.91.57496 FIN_WAIT_1
Nope
OK so now to account for this we’ve passed the bash variable in and accounted for it in the search, in the print section I’d normally have x as its my variable for the for loop but x was returning a blank field
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], x, "\n" }}' |sort -nr;done
2513
167
789
128
213
6
this is no good so I try passing the bash variable like I did in the search
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], '"${a}"', "\n" }}' |sort -nr;done
2495
230
892
149
124
6
Same thing, then it hit me maybe its not being passed as a string so I added double quotes
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done
2563 FIN_WAIT_1
194 FIN_WAIT_2
901 TIME_WAIT
108 ESTABLISHED
158 SYN_RCVD
5 LAST_ACK
Perfect
The next one I hadn’t had to deal with previously on linux was FreeBSD doesn’t use : for the port associated with the IP address, it uses a .
I used gsub to clean up my formatting
bash variable is looking for one of my states, I know I want the 5th field so I print and pipe the output and want to strip the port so I use gsub, now the 1st gsub adds an additional period at the end of the address so I add a 2nd gsub and I look for any period at the end of the line and remove it and then print the output
netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}'
I hope you’ve found this useful and find ways to expand and modify it for your needs