Tornado at work

6, Aug, 2017

FreeBSD Redis Cluster with CARP failover

11, May, 2017

Selecting FreeBSD for Redis Cluster

FreeBSD 11.0 has strong networking and memory management performance. It also does clustering in-kernel via CARP. This makes a great choice for high performance, high availability applications.

Some may argue that Linux is better documented and supported than FreeBSD. Stand-alone Redis installs and master/slave replication (often erroneously called clustering) using Sentinel to manage failovers are well documented. Clustering Redis docs generally refer to building from scratch or adding unofficial repos then manually building directories/configs. Information needs to be pieced together to make a clean, stable setup. FreeBSD makes a nice maintainable setup with little fuss.

Cluster the Hosts with Virtual IP

We are using CARP to give us a virtual IP to connect to the cluster. If the primary host is unavailable, the next CARP host will take over as master and the virtual IP will be active on that machine. If the clients are connecting to the virtual IP instead of one of the physical IPs, they will automatically connect to the next host if the primary fails. We are using PFSync to keep the state table in sync between the hosts. This will make failover transparent to the clients.

VMWare VM Config

For security concerns, we don’t enable promiscuous mode on the VMWare vSwitches unless needed. We’ve created dedicated vSwitches with promiscuous enabled for CARP hosts. Choose Load Balancer Promiscuous (VLAN4) for the Load Balancer interface since this is where the clients will connect. We also need a dedicated interface to sync the traffic. We are using an interface on the Quorum VLAN with a different subnet assigned.

Enable PF, PFSync, and CARP in the Kernel

Add the modules to enable in the /boot/loader.conf.


Configure Interfaces

We are using em0 on VLAN1, em1 is our load balancer interface, and em2 is our dedicated pfsync interface. The em1 interface is where the virtual IP ( will be aliased.

Contents of our /etc/rc.conf.d/network:

ifconfig_em0="inet netmask"
ifconfig_em1="inet netmask"
ifconfig_em1_alias0="inet vhid 10 advskew 0 pass FMxjubyqfNYTVAdS6aZ alias"
ifconfig_em2="inet netmask"

The em1 alias breakdown:

  • vhid: The same number for the other hosts your are clustering. (This needs to be unique on the subnet to avoid collisions).
  • advskew: Set between 0-255. The lower the number, the higher the priority. 0 is the master and higher numbers are backups.
  • pass: The same password for all the members of the vhid. It’s highly recommended to generate a different password for each vhid.
  • alias: This is the virtual IP that will be active on the master. When a new master is activated, the IP will move to that host.

We are using the firewall in the load balancer VLAN for our default gateway.

Contents of our /etc/rc.conf.d/routing:


CARP Preempt

Add preempt to /etc/sysctl.conf to allow a backup to take over.


Installing Redis

Install via pkg

The quickest way to install Redis is via the package manager. You can install from ports and do a make config to automatically install the redis-trib.rb script (TRIB=ON). The downside is that you have to keep ports tree updated and run portmaster to rebuild the updated ports.

sudo pkg install redis

Install Management Script

Check for Recent Ruby

The script requires a recent version of ruby installed

ruby -v

You can install Ruby 2.3 if you don’t have ruby installed or have an old version.

sudo pkg install ruby23

Pull the management script from the Redis website

curl -O

Make Script Executable

chmod +x redis-trib.rb

Move Script to /usr/local/bin/

sudo mv redis-trib.rb /usr/local/bin/

Configure Redis

We’re configuring 3 master and 3 slaves since Redis clusters require a minimum of 3 masters to create a cluster and maintain a quorum. We’re doing this on 3 hosts for additional performance and redundancy. Any host could run on its own if we lost the other 2 hosts.

Create a Configuration File for Each Instance

Each instance on a host requires a config file named redis-{$NAME}.conf. The general convention is to use the port number for the name of each instance e.g. redis-7000.conf.

Delete the default config since we won’t be using it for clustering. It could cause issues if loaded during startup and keeps the directory cleaner.

sudo rm /usr/local/etc/redis.conf

Create /usr/local/etc/redis-7000.conf:

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /var/db/redis/7000
pidfile /var/run/redis/
daemonize yes

Create redis-7001.conf through redis-7005.conf files. Substitute the port number in the port, dir, and pidfile entries. Adjust the bind IP addresses for each host as well.

Create the Working Dir for Each Instance

Each instance needs a working directory to store the db file to disk and auto-created files. These are stored in /var/db/redis/*. We need to create directories owned by the redis user named for each instance (port number).

cd /var/db/redis/

sudo mkdir 7000 7001 7002 7003 7004 7005

sudo chown -R redis:redis *

Add Startup And Profiles To /etc/rc.conf.d/redis

FreeBSD doesn’t start installed services automatically after install. We will tell it to start and tell it which profiles (instances) to start.

sudo vim /etc/rc.conf.d/redis

redis_profiles="7000 7001 7002 7003 7004 7005"

Start The Instances

sudo service redis start

We now have 6 unconfigured instances ready to be clustered.

Cluster the Redis Instances

We are ready to use the redis-trib.rb management script to configure all of the instances on all 3 of our hosts into a unified cluster.

redis-trib.rb create --replicas 1

redis-trib.rb breakdown:

  • create: Tells the script to create a cluster.
  • replicas: Tells how many slaves per master to create.
  • list of instances: These are all the unconfigured instances across all 3 hosts.

Keep in mind that redis is single threaded and needs several masters to get the performance you are looking for. Extra slaves are good but have plenty of masters as well. You can add more slaves afterwards if you want.

The script will list how it’s going to configure each instance then ask you if this looks good. Type “yes” to let it configure the instances into a cluster. It will distribute the slaves across hosts to minimize losing both master/slave in the event of a lost host.

Virtual IP Binding Cluster Issue

I reset all the instances to rebuild the cluster fresh for production. It would timeout (well, sit for a couple hours waiting) on the cluster creation when the virtual IP ( was bound in the instances. I originally removed the virtual IP, created the cluster, and re-added the virtual IP. While this works, it’s annoying as the number of instances goes up.

I realized that it would be much better to just redirect traffic from the virtual IP to the physical IP at the firewall level. This way the local instances don’t need to know or care about the virtual IP. It will look like all traffic is coming directly to the physical IP.

What to do if the Cluster Create Fails

Once in a while, the cluster script hangs up while creating the cluster. You can kill the script (control-c), stop the instances on all hosts, delete the contents of the data directories, restart the services, and run the create again.

sudo service redis stop

sudo rm /var/db/redis/700?/*

sudo service redis start

Firewall Setup to Limit VLAN1 Access

Redis security is very minimal in standalone and replication modes. It’s even more limited in cluster mode. Primarily, the webservers will be accessing the cluster directly from the load balancer network. We do want some limited access to the senior Web Developers and the System Admins to monitor and troubleshoot from their machines. We created a basic blocking of the VLAN1 IPs on the redis ports (7000-7100) then allow only specific machines to access them from VLAN1.

Edit pf.conf

FreeBSD uses the Packet Filter (PF) firewall and /etc/pf.conf is the main configuration file. The /etc/redis_allow file is a list of authorized IP addresses. The redirect line takes traffic targeted for the virtual IP and redirects to the physical IP on the interface. This makes the failover process transparent to the instances and removes the requirement to bind to the virtual IP on each instance.

Contents of our pf.conf:

### Basic Redis Firewall Setup ###
### Interfaces ###
## Tables ###
table <redis_allow> persist file "/etc/redis_allow"
# Redirect virtual IP to primary
rdr on $dmz_if proto tcp from any to $virtual -> $dmz_if:0
### Block/Pass ###
block in on $int_if proto tcp from any to any port 7000:7100
pass in on $int_if proto tcp from <redis_allow> to any port 7000:7100 keep state
pass out keep state

Installing x84 BBS on Raspberry Pi Zero running FreeBSD 11

19, Feb, 2017

Back in the 80s, I had an Apple ][gs with a 1200baud modem. I lived in farm country and everything was long distance. The next town over was 9 miles away. Because there was a county line there, it was an intraLATA call and long distance. I did have a friend in the next town and we connected via modems to chat and transfer files. Everything was all manual modem AT commands but it was fun. The problem was that my parents didn’t like me tying up the phone line for too long. Plus it got expensive for teenager with little extra cash to pay for the phone bill. Kids and parents worrying about overages is not a new thing. ;-)

I had a subscription to The Computer Shopper and it was a massive catalog packed with articles and ads from every computer company of the time. It took a lot of time to go through every month (usually the next issue was here before you were done). It was a great wealth of computer information in the pre-Internet days. I learned about dialup Bulletin Board Systems (BBS) and tried dialing up a few free boards but rarely staying on for very long at a time because I knew it was going to rack up the bill quickly.

Bulltin Boards started falling out of popularity as the Internet came on the scene. A lot of the functionality provided by a BBS was now available on websites. Many shut down and a few added telnet so you could reach them over the Internet. There’s still many BBS that are still running today. I decided that maybe it’s time to finally go play with some BBS since I didn’t get much of a chance the 1st time around.

I decided to run this on my Raspberry Pi Zero because the single core ARM runs the ANSI screens at a reasonable speed. The original Raspberry Pi A or B model would be good candidates too. I tried it on a quad core system and the screens went by way too fast to read/see properly. Logins are a little slow but that’s keeping with the feel of how things used to be. I bypassed a lot of proprietary BBS software because it wasn’t going to be able to run on my hardware/OS and didn’t give you much room to tweak. I wanted to keep to software that could run on many platforms (python, ruby, etc.). I did find a node.js BBS that looked nice but there’s not a lot of node packages for FreeBSD/ARM. Then I ran across x84. It’s written in Python and thought it looked very promising. I tried it out and decided it’s a keeper (as long as I don’t find something better).

Down to the actual install: (finally)

Let me help you setup all the pre-requisites and tweaks so you don’t have to spend hours of googling and trial & error. I went through 3 or 4 installs to figure this all out. Here’s some packages to get everything prepped for the install.

sudo pkg install gcc py27-pip py27-sqlite3 py27-gmpy 
py27-wcwidth py27-requests py27-six py27-more-itertools  
py27-jaraco.timing py27-jaraco.util py27-irc py27-dateutil 
py27-feedparser py27-html2text py27-pycparser py27-cffi 
py27-bcrypt py27-idna py27-pyasn1 py27-enum34 py27-ipaddress 
py27-cryptography py27-ecdsa py27-paramiko py27-openssl 
webpy py27-cherrypy curl bash

FreeBSD uses some cross-compile tools on some embedded platforms (mips, arm, aarch64, etc.) which aren’t used in this setup and will cause build errors. Change all references of /nxb-bin/usr/bin/cc to /usr/local/bin/gcc in /usr/local/lib/python2.7/ to remove the cross-compile dependencies and allow gcc to build the remaining modules.

sudo sed -i 's//nxb-bin/usr/bin/cc//usr/local/bin/gcc/g' 

Install x84 via pip. The option with_crypto enables ssh and ssl if you want to use the embedded webserver. You can omit this if you want telnet only though ssh would be recommended.

pip install --user x84[with_crypto]

Start it up (~/.local/bin/x84) to generate the default config. Let it finish starting up then control-c to kill the BBS.

Edit ~/.x84/default.ini and change to the IP address of your network interface. You can also set bbsname while you’re editing.

Fire it back up in tmux (or screen if you prefer). Login with telnet on port 6023 or ssh -p 6022 new@bbsaddress (or anonymous@) to register a new user. The first user setup is automatically a sysop and can change system settings from within the BBS.

Screenshot of default setup. I used cool-retro-term with DOS profile set to at least 80 column wide. Nothing like adding scanlines, phosphor refresh, static, etc. to make it feel more old school. Hyper Term has a similar retro theme.

Now you have a default install of x84 BBS running on your Raspberry Pi. Time to read the docs to learn how to use and customize your setup. Read the doors section to figure out how to run any external ascii/curses program from the menu (returns to the menu after exit). I’ve played with silly apps like nyancat, cmatrix, etc. but have been messing with frotz to play old Infocom text adventure games too. I’ve looked through every command line, curses game for FreeBSD on Freshports to see what might be a good fit on my BBS.

The ansi screen files are in the following location:

You can view and edit ANSI/ASCII files with PabloDraw. It’s not a bad idea to have figlet, cowsay, etc. to generate some ascii art to get you off the ground. You can view and download some ANSI files from the Sixteen Colors ANSI Art site.


26, Aug, 2016

The last few years, I’ve taken a motorcycle trip to Arkansas on my birthday. My wife and kids are in school so I take the day riding and back in the evening to celebrate with the family. This year, we drove to Nebraska for my Aunt’s funeral. I cancelled the ride and figured that I’ll do it again next year.

The following week, my son has a cross-country meet in Muskogee on Friday morning. I decided to take the day off to go watch him run and then go on to the fun roads afterwards. The end of the week was looking rainy. I’ve done plenty of rides on the motorcycle while chasing and being chased by rain. It can be a lot of fun. I got a Subaru WRX a couple months ago and this would be a good chance to take it out on the twisty AR roads. Who wouldn’t want to take an all-wheel drive, turbo charged sports car out for some fun?

I woke up late this morning, saw I barely had time to get to the meet, jumped into my clothes and ran out the door. I buzzed down the Muskogee Turnpike and missed my exit when I got to town. It was pouring rain 1/2 mile south of the college that was holding the cross-country meet. I drove a little farther and the rain suddenly stops. It’s dry when I get to the school.

I pull into a spot and head for the course. My son’s race had just started so I go find a spot to cheer him on. I see him come around the corner and cheer him to catch the guy in front of him. Then I head up to the finish to see how he did. He said he was going to grab his stuff so he could go with me. I know he just wants to take a shower and recover after a race. After explaining my plans to be out all day, he decided to take the bus back so he could get home quicker. I would have loved to brought him along but knew he wouldn’t have enjoyed being thrown into corners all day in that condition.

I headed east to Talequah and then took highway 62 east toward AR. It was raining off and on and the car was sure footed in the wet. The traffic was light on 62 but enough to slow me down in some of the corners. Press on to Fayetteville, AR and on out on Highway 16. The rain stopped and traffic was light. Some fast sweepers and turn south on the Pig Trail (Hwy 23). Into the forest and through the mountains.

Plenty of tight twisty turns with downshifting and spooling up the turbos. It’s pure magic when it hooks up and shoots you out of a corner.

Stop at the scenic overlook halfway through and I can smell my tires are definitely warmed and up to temperature.

Down Hwy 215 along the Mulberry River to the Oark Cafe in Oark, AR for lunch. Still a little baffled about the session computer saying I got 28MPG through the most aggressive driving section.

Bacon cheeseburger and curly fries.

Buttermilk pie. I usually don’t have room for pie when I stop here but I managed today.

Highway 103 south of Oark. This is a really twisty road with several switchbacks. One section drops 600ft of altitude in 1/2mile or so. This is a fun road.

  • Posted by dale in in Car

Let’s Encrypt SSL certificates on FreeBSD

18, Nov, 2015

To quote the Let’s Encrypt about page:

“Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. Let’s Encrypt is a service provided by the Internet Security Research Group (ISRG).”

The nutshell of what they are doing is creating free SSL certificates that are automatically recognized in all major browsers and automatically renewed. They are currently using 90 day expiration (with the dicussion of possibly dropping to 60 day) with renewals at 60 days to give you 30 day window to get it renewed. One of the bigger issue with long renewals is that most clients never check if a certificate is revoked or not. They’ll just assume it’s good if it’s not expired. A revoked certificate would not get renewed and would throw errors on the client much quicker with short renewals.

One place that automatic renewing certificates could be a potential issue is on external SSL services like SSL accelerators, Web Application Firewalls (WAF), etc. because they have the certificates loaded directly on them to inspect traffic before sending it on the webserver or other SSL endpoint. You’ll often see EV certificates on these appliances and Let’s Encrypt isn’t doing Extended Validation certificates. So it’s not huge issue but it would be nice to see the vendors adopt Let’s Encrypt support for the people not using EV certs on their appliances/services.

Getting started:

It’s currently in a limited beta so the first thing you need to do is to fill in the Sign up form to request your domain(s) and any subdomains since they need to be specified in the request and an email address to be used for your account. Then wait for the confirmation email. I submitted on a Thursday afternoon and got the acceptance email on Monday afternoon so be patient. It goes into open beta December 3rd so this will change some.

There’s a few different clients to get your certificates and I checked into which ones might work best on my FreeBSD webserver. In the end, I used the default client from Github and it works great on FreeBSD. Props to everyone that got the kinks worked out already.

The client listens on the port so you have to temporarily stop your webserver for it to answer. The first -d is the domain that will be signed to the certificate. More -d are for SAN names. You can make one certificate with all of your domain/subdomains or make individual certificates for each domain (plus at least I prefer to make one cert plus all subdomains for better compatibility (some clients have issues if your domain is only a SAN and not directly signed).

sudo ./letsencrypt-auto certonly -a standalone -d -d -d –server –agree-dev-preview –debug (the –debug is required for FreeBSD, not other OSes)

This creates /etc/letsencrypt/live/ directory with cert.pem, chain.pem, fullchain.pem, privkey.pem. Fullchain.pem is the cert and chain file in one file for services that don’t have a seperate chain command (some imap, pop3, smtp, etc.) or if you want to keep it simple on your webserver. This directory also contains your account info, CSR, private key for generating the certs, etc. It’s root only access but you should make a backup of it so you don’t have to start over later.

Start the webserver back up once you’ve created all the certificates that you want.

Point your vhost(s) to the /etc/letsencrypt/live/ (and chain.pem, fullchain.pem, privkey.pem) so when you renew the certificate(s) in 60 days, it will automatically be pointing at the updated files. I’ve written a script to stop the webserver, re-run the scripts, restart the webserver, and email me (to make sure it went fine). I’ve added it to run every other month in my crontab. (I’m not running the job at the top of the hour like a good sysadmin should do when polling a service likely to be hit by many clients.)

Here you can see I’m running TLS 1.2 with High ciphers

My domain as the CN and you can see it’s signed by Let’s Encrypt

SSL Labs scanner gives my current setup an A. I’ve looked at what it take to get an A+ and it can break some clients so I’ll check it out before I decide what changes to do next.

I set my http vhost to automatically redirect to https. If you are running Flatpress like I am here, you’ll need to edit the defaults.php in the root of your install. Change the BLOG_BASEURL from http:// to https:// and you won’t have to worry about it sending mixed content and not loading half the page. ;-)