I’ve been busy tuning Varnish to prevent outages and that makes me take a much closer look at things and how to get through things quickly especially when it comes to working with farms of servers. I noticed in my cacti graphs a huge rise in FIN_WAIT_1 connection states on my FreeBSD servers to a degree of 25x the number of established connections so I decided to do some work to see what’s going. What I came to I think will provide some nice insight for many different scenarios.
First how I made my determination of my ratios, I know what connection states I’m looking so I create an array with those values, I know that it’s the 6th field and I’d like a count of where each stands. This is the oneliner
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done
I added the carriage return to clean up the output
767 FIN_WAIT_1 103 FIN_WAIT_2 866 TIME_WAIT 54 ESTABLISHED 220 SYN_RCVD 7 LAST_ACK
Now that I have that I want to know who’s connected, how many connections they have by connection state this oneliner takes care of that
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do echo -e "\n" $a; netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}' |awk '{freq[$0]++} END {for (x in freq) {print freq[x], x }}' |sort -nr;done
and that produces an output like this
FIN_WAIT_1 627 67.198.71.114 390 64.90.159.102 264 89.241.213.178 221 174.108.76.131 71 186.32.54.114 FIN_WAIT_2 5 66.66.133.117 5 65.0.139.23 5 173.169.40.34 5 166.147.120.16 4 96.246.240.75 TIME_WAIT 98 10.10.10.20 68 10.10.10.22 60 10.10.10.23 60 10.10.10.24 49 10.10.10.25 ESTABLISHED 13 186.32.54.114 9 98.255.88.48 7 64.90.159.102 6 70.208.78.19 5 98.204.42.34 SYN_RCVD 22 108.45.28.123 2 12.160.201.226 1 98.255.88.48 LAST_ACK 6 71.215.198.128 2 66.104.42.194 1 80.58.205.48 1 41.79.120.19 1 121.120.116.205
Some of these gave me a hard time so let’s take a closer look at some of these
I’ve covered the gotchas from using a bash array variable in an earlier blog so I won’t cover that again. What I did discover is awk unkowingly modifies the output. I didn’t understand why I was getting this strange frequency output until I broke down what was happening and that’s when I saw what awk was doing
netstat -n |awk '($6 = /FIN_WAIT_1/)' |head -n 5 tcp4 0 532 10.10.10.169.80 173.175.47.91.55257 1 tcp4 0 532 10.10.10.169.80 173.175.47.91.55256 1 tcp4 0 533 10.10.10.169.80 108.45.28.123.49944 1 tcp4 0 0 10.10.10.169.80 113.197.8.162.41647 1 tcp4 0 0 10.10.10.169.80 113.197.8.162.2879 1
UH? I thought maybe it was how netstat behaved and I just never noticed
netstat -n |grep FIN_WAIT_1 |head -n 5 tcp4 0 24606 10.10.10.169.80 199.224.118.221.62975 FIN_WAIT_1 tcp4 0 3579 10.10.10.169.80 74.12.184.206.1323 FIN_WAIT_1 tcp4 0 533 10.10.10.169.80 173.175.47.91.57498 FIN_WAIT_1 tcp4 0 533 10.10.10.169.80 173.175.47.91.57497 FIN_WAIT_1 tcp4 0 533 10.10.10.169.80 173.175.47.91.57496 FIN_WAIT_1
Nope
OK so now to account for this we’ve passed the bash variable in and accounted for it in the search, in the print section I’d normally have x as its my variable for the for loop but x was returning a blank field
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], x, "\n" }}' |sort -nr;done
2513
167
789
128
213
6
this is no good so I try passing the bash variable like I did in the search
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], '"${a}"', "\n" }}' |sort -nr;done
2495
230
892
149
124
6
Same thing, then it hit me maybe its not being passed as a string so I added double quotes
connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done
2563 FIN_WAIT_1
194 FIN_WAIT_2
901 TIME_WAIT
108 ESTABLISHED
158 SYN_RCVD
5 LAST_ACK
Perfect
The next one I hadn’t had to deal with previously on linux was FreeBSD doesn’t use : for the port associated with the IP address, it uses a .
I used gsub to clean up my formatting
bash variable is looking for one of my states, I know I want the 5th field so I print and pipe the output and want to strip the port so I use gsub, now the 1st gsub adds an additional period at the end of the address so I add a 2nd gsub and I look for any period at the end of the line and remove it and then print the output
netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}'
I hope you’ve found this useful and find ways to expand and modify it for your needs
