awk – Who is connected, how many connections do they have and what is their connection state

I’ve been busy tuning Varnish to prevent outages and that makes me take a much closer look at things and how to get through things quickly especially when it comes to working with farms of servers. I noticed in my cacti graphs a huge rise in FIN_WAIT_1 connection states on my FreeBSD servers to a degree of 25x the number of established connections so I decided to do some work to see what’s going. What I came to I think will provide some nice insight for many different scenarios.

First how I made my determination of my ratios, I know what connection states I’m looking so I create an array with those values, I know that it’s the 6th field and I’d like a count of where each stands. This is the oneliner

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done

I added the carriage return to clean up the output

767 FIN_WAIT_1 

103 FIN_WAIT_2





Now that I have that I want to know who’s connected, how many connections they have by connection state this oneliner takes care of that

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]}; do echo -e "\n" $a; netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}' |awk '{freq[$0]++} END {for (x in freq) {print freq[x], x }}' |sort -nr;done

and that produces an output like this







Some of these gave me a hard time so let’s take a closer look at some of these
I’ve covered the gotchas from using a bash array variable in an earlier blog so I won’t cover that again. What I did discover is awk unkowingly modifies the output. I didn’t understand why I was getting this strange frequency output until I broke down what was happening and that’s when I saw what awk was doing

netstat -n |awk '($6 = /FIN_WAIT_1/)' |head -n 5
tcp4 0 532 1
tcp4 0 532 1
tcp4 0 533 1
tcp4 0 0 1
tcp4 0 0 1

UH? I thought maybe it was how netstat behaved and I just never noticed

 netstat -n |grep FIN_WAIT_1 |head -n 5
tcp4       0  24606  FIN_WAIT_1
tcp4       0   3579     FIN_WAIT_1
tcp4       0    533    FIN_WAIT_1
tcp4       0    533    FIN_WAIT_1
tcp4       0    533    FIN_WAIT_1


OK so now to account for this we’ve passed the bash variable in and accounted for it in the search, in the print section I’d normally have x as its my variable for the for loop but x was returning a blank field

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]};  do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], x, "\n" }}' |sort -nr;done






this is no good so I try passing the bash variable like I did in the search

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]};  do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], '"${a}"', "\n" }}' |sort -nr;done






Same thing, then it hit me maybe its not being passed as a string so I added double quotes

connstates=( FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT ESTABLISHED SYN_RCVD LAST_ACK); for a in ${connstates[@]};  do netstat -n |awk '($6 = /'"${a}"'/){freq['"${a}"']++} END {for (x in freq) {print freq[x], "'"${a}"'", "\n" }}' |sort -nr;done
2563 FIN_WAIT_1 

194 FIN_WAIT_2 






The next one I hadn’t had to deal with previously on linux was FreeBSD doesn’t use : for the port associated with the IP address, it uses a .
I used gsub to clean up my formatting

bash variable is looking for one of my states, I know I want the 5th field so I print and pipe the output and want to strip the port so I use gsub, now the 1st gsub adds an additional period at the end of the address so I add a 2nd gsub and I look for any period at the end of the line and remove it and then print the output

netstat -n |awk '($6 = /'"${a}"'/){print $5}'|awk -F \. '{$5 = ""; gsub(/\ /, "."); gsub(/.$/,""); print}'

I hope you’ve found this useful and find ways to expand and modify it for your needs

Adding static routes in FreeBSD

Adding routes is easy but doesn’t survive reboots. In order to make a route static or persistent we add an entry to the /etc/rc.conf file

route add

becomes this in /etc/rc.conf
net1=”-net 192.1681.0/24

or if you have multiple routes they are space delimited like this
static_routes=”net1 net2″
route_net1=”-net  192.1681.0/24″
route_net2=”-net  192.1682.0/24″

Upgrading from FreeBSD 9.0 to 9.1

This weekend I was having an issue updating chromium (chrome for FreeBSD) on my FreeBSD laptop and I finally succumbed and went on the boards and was told that my version 9.0 was EOL. Crap!! Since I have 3 9.0 systems at the house I taught myself how to upgrade to 9.1 and vetted the upgrade against my other two machines. The biggest gotcha I found was to make sure the system is as up to date as possible (ie I couldn’t update chromium one one), I left another system with 35 patches on it and let’s just say things didn’t go well on that system. I was lucky enough it was VM and smart enough to clone the system that gave me a problem so I was able to test this a few times.

Here are the steps I took to perform the upgrade

#Update your ports tree
sudo portsnap fetch update

#Determine what packages require updating
sudo portversion |grep –v \=

#To update all packages out of date or pass the packages to upgrade individually to portupgrade
sudo portupgrade –rRv ` portversion |grep –v \=`

#Prepare the upgrade
sudo freebsd-update fetch install

#If patches were installed reboot
sudo shutdown –r now

#Upgrade to 9.1 release
sudo freebsd-update upgrade –r 9.1-RELEASE
sudo freebsd-update install

#Reboot to the new kernel
sudo shutdown –r now

#Update needs to run again against userland components
sudo freebsd-update install

#I hear with 8.X you have to run the last command again to rebuild 3rd party packages but it said no packages when I did it on 9.1 Kernel from 9.0, I’ll be testing upgrading from 8.3 later this week.

#Reboot to the 9.1 release
sudo shutdown –r now

Adding FreeBSD 9 servers to Active Directory with Samba36 and Kerberos

#Take care of time sync issues before anything else
#Configure ntpdate against domain controller and enable or make sure all servers are #pointed to the same time source it’s best to use the domain controller

vi /etc/rc.conf


#Min number of files required for the kernel to deal with AD is 16384, modify systctl.conf
vi /etc/sysctl.conf


#Reboot so the changes take effect
#Make sure /etc/hosts is configured properly
#Adjust per system localhost kgfree

#Create krb5.conf
vi /etc/krb5.conf

default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

dns_lookup_realm = true
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
kdc =
admin_server =
default_domain = MYDOMAIN
kdc =
[domain_realm] = MYDOMAIN.COM = MYDOMAIN.COM

#Test kerberos
kinit Administrator@MYDOMAIN.COM
#Confirm ticket

#Install Samba
cd /usr/ports/net/samba36
make install clean
#configure options

[X] LDAP With LDAP support
[X] ADS With Active Directory support
[X] WINBIND With WinBIND support
[X] ACL_SUPPORT With ACL support
[X] SYSLOG With Syslog support
[X] PAM_SMBPASS With PAM authentication vs passdb backends
[X] DNSUPDATE With dynamic DNS update(require ADS)
[X] POPT With system-wide POPT library

Choose python bindings
No TLS/SSL for cups

#Configure Samba to start
vi /etc/rc.conf


#Configure Samba
vi /usr/local/etc/smb.conf

workgroup = MYDOMAIN
server string = FreeBSD Server
security = ads
idmap config * : range = 10000-20000
idmap config * : backend = idmap_rid:MYDOMAIN=10000-20000
template shell = /bin/sh
template homedir = /home/%D/%U
winbind use default domain = Yes
load printers = yes
log file = /var/log/samba/log.%m
max log size = 50
wins server =
dns proxy = no

comment = Home Directories
browseable = no
writable = yes

#Test to make sure the conf file is correct and in the expected location (sometimes you #may have to move the smb.conf to /usr/local/etc/samba )

#Join the domain
net ads join -U Administrator

#Home directory creation, we can use pam_mkhomedir
cd /usr/ports/security/pam_mkhomedir
make install clean

#Enable homedir creation with sshd
vi /etc/pam.d/sshd

auth            sufficient             no_warn try_first_pass
auth sufficient /usr/local/lib/ try_first_pass
session required /usr/local/lib/

#Start samba
/usr/local/etc/rc.d/samba start

#Test users and groups (if everything is configure properly you will see a list of domain #users and groups
wbinfo -u
wbinfo -g

Dell D630 FreeBSD 9 Build

Here’s the build of my latest system I won’t go into installation or network configuration as that’s very well documented in the handbook

# I chose the 64bit install for my D630
Freebsd 9 installation DVD x64 iso (I pxeboot the install from a linux server)
# I select source (I need it to compile certain ports I use)
Choose SRC as an additional installation option

# Create your local user(s)
Set user preferences up

# Get the ports tree current
portsnap fetch extract

# Later I just run this to maintain ports as my regular user
sudo portsnap fetch update


#login as root

# If you want your user to be able to reboot replace user_name with your username
pw groupmod operator -m user_name

# I use two port management tools portmaster and portupgrade

cd /usr/ports/ports-mgmt/portmaster && make install clean
cd /usr/ports/ports-mgmt/portupgrade && make install clean

# I’m partial to sudo
portinstall sudo
add your user to the sudoers file

# logout as root
# I log back in as my regular user because I reference my home directory for some configuration files

# I use portmaster to configure xorg-server
sudo portmaster x11-servers/xorg-server

# I configure HAL in case well just in case
select HAL option
I remove support from vid drivers that aren’t relevant to my laptop
Enabled libsigsegv for diagnostics
the rest of the options for the port and dependencies I leave the defaults

# Move and configure defaults for portmaster ( I do this after xorg so I can configure some options rather than running configure)
sudo cp /usr/local/etc/portmaster.rc.sample /usr/local/etc/portmaster.rc

# I use vi  to edit files so use your preferred editor and edit the above file make sure the options between start and end exist and are uncommented
## Start
# Always delete stale distfiles without prompting (-d)
# Be verbose (-v)
# Install packages for build-only dependencies (–packages-build)
# Delete build-only dependencies when finished (–delete-build-only)
#Suppress the build confirmation message (–no-confirm)
## End

# For better desktop performance edit /boot/loader.conf
Edit loader.conf to have kern.maxfiles=”25000″

# I always get errors with portmaster with one or more of these so I’ve switched my install at this step to use portinstall and install multiple ports at once
sudo portinstall -v xauth xinit xorg-fonts webfonts

# configure  Xorg
sudo Xorg -configure
sudo cp /etc/X11/xorg.conf
# After several attempted installs on two different laptops I have come to just add this to the ServerLayout section of the xorg.conf as the final option
Option “AllowEmptyInput” “Off”
# Add the path for webfonts in the Files section
FontPath “/usr/local/lib/X11/fonts/webfonts/”

# If you have an nvidia card
sudo portinstall nvidia-driver
sudo portinstall nvidia-xconfig
sudo nvidia-xconfig

# If the nvidia card fails to load due to error 6 add this to your loader.conf
sudo echo ‘debug.acpi.disabled=”sysres”‘ >> /boot/loader.conf

# I like the configurability of fluxbox and how lightweight it is so
sudo portinstall fluxbox
Choose imlib2 and PDF options

# I use tint2 as a toolbar and wbar as a launcher but I’ve been working more with the menu on this build so I’m not starting it at startup for this build
sudo portinstall tint2 wbar

# I like these two terminals
sudo portinstall xterm rxvt-unicode

# I’m using devd automounting even though I have hal enabled
sudo portinstall automount

# My preferred browsers
sudo portmaster WWW/Firefox
sudo portinstall www/chromium
Add gstreamer
Add ffmpeg
Add lame alsa

# install pidgin for IM
sudo portinstall pidgin

#setup directory for tint2
mkdir -p ~/.config/tint2
cp /usr/local/share/examples/tint2/tint2rc ~/.config/tint2/tint2rc

# I find it easier to create the fluxbox dir and then launch fluxbox and shut it down so it creates on the default files for me

#First create the dir
mkdir ~/.fluxbox

# create the startup file I’ve found it to either be .fluxbox/startup or ~/.xinitrc for this build I used ~/.xinitrc

# Enter this as the last line in ~/.xinitrc
exec fluxbox

# launch X and as soon as it starts right click and exit fluxbox

# Edit ~/.fluxbox/init and find session.screen0.toolbar.visible: true if it does not exist create the entry and set it to false

# To receive notifications I install
sudo portinstall notification-daemon

# To monitor cpu, battery and network these are nice lightweight apps and uses the slit
sudo portinstall wmnd wmcpuload wmbsdbatt

# I like to use atop for monitoring system performance
sudo portinstall atop

# To maintain your desktop image you change the screen number to change the wallpaper on that desktop default is 0 – 3 edit .fluxbox/init
session.screen0.rootCommand:  fbsetbg -f ~/daemonsunset.jpg

# I edit ~/.xinitrc and add the programs I want to start at launch these all must be before the command exec fluxbox I start tint, wmnd for wlan0 (you launch one per interface), cpuload, terminal daemon and notification daemon, my current config is between the start and end

( sleep 2 && tint2 ) &
wmnd -i wlan0 &
wmcpuload &
wmbsdbatt -b &
urxvtd -q -f -o &
/usr/local/libexec/notification-daemon &
exec fluxbox

# I use the sample config for automount
sudo cp /usr/local/etc/automount.conf.sample /usr/local/etc/automount.conf

# To take screenshots I use scrot because it offers some nice options
sudo portinstall scrot

# Set up a screensaver
sudo portinstall xscreensaver

# Reboot and at this point log back in as your user and edit.fluxbox/menu. There should already be a default from fluxbox there are many options you can do or use for now I’ll just set some basic favorites at the top and change the title of my menu I’ll eventually add IRC so i’m adding it now

[begin] (TheMadIndian’s Fluxbox Menu)
[exec] (Firefox) {firefox} </usr/local/lib/firefox/chrome/icons/default/default48.png>
[exec] (Chrome) { /usr/local/bin/chrome} </usr/local/share/chromium/product_logo_48.png>
[exec] (Terminal) {$TERM}
[exec] (Irc) {xchat}
[exec] (Lock screen) {xscreensaver-command -lock}
[separator ]

# save the file and startx

I come up with a simply configured fluxbox, I’ll add more apps, I may update the build again but with the rdesktop port I have been working effectively all day on this configuration administering linux, freebsd and windows servers.