awk – Who is connected, how many connections do they have and what is their connection state

I’ve been busy tuning Varnish to prevent outages and that makes me take a much closer look at things and how to get through things quickly especially when it comes to working with farms of servers. I noticed in my cacti graphs a huge rise in FIN_WAIT_1 connection states on my FreeBSD servers to a degree of 25x the number of established connections so I decided to do some work to see what’s going. What I came to I think will provide some nice insight for many different scenarios.

First how I made my determination of my ratios, I know what connection states I’m looking so I create an array with those values, I know that it’s the 6th field and I’d like a count of where each stands. This is the oneliner

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done

I added the carriage return to clean up the output

767 FIN_WAIT_1 

103 FIN_WAIT_2

866 TIME_WAIT

54 ESTABLISHED

220 SYN_RCVD

7 LAST_ACK

Now that I have that I want to know who’s connected, how many connections they have by connection state this oneliner takes care of that

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do echo -e "\n" $a; netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}' |awk '{freq[$0]++} END {for (x in freq) {print freq[x], x }}' |sort -nr;done

and that produces an output like this

FIN_WAIT_1
627 67.198.71.114
390 64.90.159.102
264 89.241.213.178
221 174.108.76.131
71 186.32.54.114

FIN_WAIT_2
5 66.66.133.117
5 65.0.139.23
5 173.169.40.34
5 166.147.120.16
4 96.246.240.75

TIME_WAIT
98 10.10.10.20
68 10.10.10.22
60 10.10.10.23
60 10.10.10.24
49 10.10.10.25

ESTABLISHED
13 186.32.54.114
9 98.255.88.48
7 64.90.159.102
6 70.208.78.19
5 98.204.42.34

SYN_RCVD
22 108.45.28.123
2 12.160.201.226
1 98.255.88.48

LAST_ACK
6 71.215.198.128
2 66.104.42.194
1 80.58.205.48
1 41.79.120.19
1 121.120.116.205

Some of these gave me a hard time so let’s take a closer look at some of these
I’ve covered the gotchas from using a bash array variable in an earlier blog so I won’t cover that again. What I did discover is awk unkowingly modifies the output. I didn’t understand why I was getting this strange frequency output until I broke down what was happening and that’s when I saw what awk was doing

netstat -n |awk '($6 = /FIN_WAIT_1/)' |head -n 5
tcp4 0 532 10.10.10.169.80 173.175.47.91.55257 1
tcp4 0 532 10.10.10.169.80 173.175.47.91.55256 1
tcp4 0 533 10.10.10.169.80 108.45.28.123.49944 1
tcp4 0 0 10.10.10.169.80 113.197.8.162.41647 1
tcp4 0 0 10.10.10.169.80 113.197.8.162.2879 1

UH? I thought maybe it was how netstat behaved and I just never noticed

 netstat -n |grep FIN_WAIT_1 |head -n 5
tcp4       0  24606 10.10.10.169.80      199.224.118.221.62975  FIN_WAIT_1
tcp4       0   3579 10.10.10.169.80      74.12.184.206.1323     FIN_WAIT_1
tcp4       0    533 10.10.10.169.80      173.175.47.91.57498    FIN_WAIT_1
tcp4       0    533 10.10.10.169.80      173.175.47.91.57497    FIN_WAIT_1
tcp4       0    533 10.10.10.169.80      173.175.47.91.57496    FIN_WAIT_1

Nope

OK so now to account for this we’ve passed the bash variable in and accounted for it in the search, in the print section I’d normally have x as its my variable for the for loop but x was returning a blank field

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]};  do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], x, "\n" }}' |sort -nr;done
2513  

167  

789  

128  

213  

6

this is no good so I try passing the bash variable like I did in the search

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]};  do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], '"${a}"', "\n" }}' |sort -nr;done
2495  

230  

892  

149  

124  

6

Same thing, then it hit me maybe its not being passed as a string so I added double quotes

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]};  do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done
2563 FIN_WAIT_1 

194 FIN_WAIT_2 

901 TIME_WAIT 

108 ESTABLISHED 

158 SYN_RCVD 

5 LAST_ACK

Perfect

The next one I hadn’t had to deal with previously on linux was FreeBSD doesn’t use : for the port associated with the IP address, it uses a .
I used gsub to clean up my formatting

bash variable is looking for one of my states, I know I want the 5th field so I print and pipe the output and want to strip the port so I use gsub, now the 1st gsub adds an additional period at the end of the address so I add a 2nd gsub and I look for any period at the end of the line and remove it and then print the output

netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}'

I hope you’ve found this useful and find ways to expand and modify it for your needs