So you have a OpenBSD firewall. Actually you have at least two because you are doing carp for high availability (not load balancing but HA), right?
Awesome! It’s fun isn’t it? I suppose I have to admit it’s more fun testing it in a lab environment than in production. :)
One thing I noticed when doing a bit of non-scientific load testing on a pair of small carped firewalls is that in OpenBSD the size of the state table is limited to 10000 entries by default. I would imagine that most people won’t run into the limit, but I was surprised at how easy it was to hit 10000 sessions using something like siege.
Using a client laptop–an older core 2 duo with 4 gigs of ram and a 60 gig SSD drive (a Lenovo T61 specifically) on one side of the firewall and a virtualized web server with 512MB of RAM and one CPU on the other–I was able to hit the state limit in a couple of seconds with a command such as:
laptop$ siege -b -c 40 -r 100 http://testserver/testpage
where the resulting test page looks like:
hi there
When the siege command is run I watch the state tables on the OpenBSD firewalls with a command such as:
openbsd_fw{1,2}# while true; do pfctl -s info; sleep 1; done
With nothing happening the result of that command looks about like this:
openbsd_fw1$ pfctl -s info
Status: Enabled for 0 days 00:22:05 Debug: err
State Table Total Rate
current entries 38
searches 215881 162.9/s
inserts 30364 22.9/s
removals 46342 35.0/s
Counters
match 30804 23.2/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 362 0.3/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
We can see that the current limit is 10000:
openbsd_fw1$ pfctl -sm
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000
So lets fire up that siege command and see what happens by watching the current entries on the firewall that has the master carp IP. (Note that with pfsync all the states will be transferred to the backup firewall as well, but for simplicity let’s focus on the master.)
openbsd_fw1$ ifconfig | grep -i master
carp: MASTER carpdev fxp2 vhid 1 advbase 1 advskew 0
status: master
carp: MASTER carpdev fxp0 vhid 2 advbase 1 advskew 0
status: master
openbsd_fw1$ while true; do pfctl -s info | grep "current entries"; sleep 1; done
current entries 17
current entries 15
current entries 13
current entries 12
current entries 12
# siege starts up here
current entries 4820
current entries 10000
current entries 10000
current entries 10000
current entries 10000
current entries 10000
^C
So you can see it only takes a couple seconds to hit that limit.
Let’s up it to 200000 and see what happens.
openbsd_fw1$ grep "set limit" /etc/pf.conf
set limit states 200000
openbsd_fw1$ pfctl -nf /etc/pf.conf
openbsd_fw1$ pfctl -f /etc/pf.conf
openbsd_fw1$ pfctl -sm
states hard limit 200000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000
Run the siege command on the client:
openbsd_fw1$ while true; do pfctl -s info | grep "current entries"; sleep 1; done
current entries 40
current entries 40
current entries 40
current entries 40
# siege starts up here
current entries 560
current entries 6686
current entries 12480
current entries 17728
current entries 23060
current entries 27116
current entries 28332
current entries 29498
current entries 29884
current entries 29884
current entries 29884
current entries 29884
current entries 29884
current entries 29884
current entries 29884
current entries 29884
^C
And looks like we max out the siege command now at about 30k sessions. Nice!
To conclude, this was just a quick look at session limits on OpenBSD. If you’re running a pf firewall it may be something to consider looking at to make sure you’re not hitting the limit which would reduce the effectiveness of your firewall.
Note that I haven’t shown any memory usage from the firewall, but the small boxes have 512MB of RAM and even at 200K sessions the memory usage only went up very slightly so I don’t think it’s constrained for memory reasons.