Tornado at work

6, Aug, 2017

FreeBSD Redis Cluster with CARP failover

11, May, 2017

Selecting FreeBSD for Redis Cluster

FreeBSD 11.0 has strong networking and memory management performance. It also does clustering in-kernel via CARP. This makes a great choice for high performance, high availability applications.

Some may argue that Linux is better documented and supported than FreeBSD. Stand-alone Redis installs and master/slave replication (often erroneously called clustering) using Sentinel to manage failovers are well documented. Clustering Redis docs generally refer to building from scratch or adding unofficial repos then manually building directories/configs. Information needs to be pieced together to make a clean, stable setup. FreeBSD makes a nice maintainable setup with little fuss.

Cluster the Hosts with Virtual IP

We are using CARP to give us a virtual IP to connect to the cluster. If the primary host is unavailable, the next CARP host will take over as master and the virtual IP will be active on that machine. If the clients are connecting to the virtual IP instead of one of the physical IPs, they will automatically connect to the next host if the primary fails. We are using PFSync to keep the state table in sync between the hosts. This will make failover transparent to the clients.

VMWare VM Config

For security concerns, we don’t enable promiscuous mode on the VMWare vSwitches unless needed. We’ve created dedicated vSwitches with promiscuous enabled for CARP hosts. Choose Load Balancer Promiscuous (VLAN4) for the Load Balancer interface since this is where the clients will connect. We also need a dedicated interface to sync the traffic. We are using an interface on the Quorum VLAN with a different subnet assigned.

Enable PF, PFSync, and CARP in the Kernel

Add the modules to enable in the /boot/loader.conf.

carp_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

Configure Interfaces

We are using em0 on VLAN1, em1 is our load balancer interface, and em2 is our dedicated pfsync interface. The em1 interface is where the virtual IP (10.0.0.10) will be aliased.

Contents of our /etc/rc.conf.d/network:

ifconfig_em0="inet 192.168.10.11 netmask 255.255.255.0"
ifconfig_em1="inet 10.0.0.11 netmask 255.255.255.0"
ifconfig_em1_alias0="inet vhid 10 advskew 0 pass FMxjubyqfNYTVAdS6aZ alias 10.0.0.10/32"
ifconfig_em2="inet 10.0.1.11 netmask 255.255.0.0"
pfsync_enable="YES"
pfsync_syncdev="em2"

The em1 alias breakdown:

  • vhid: The same number for the other hosts your are clustering. (This needs to be unique on the subnet to avoid collisions).
  • advskew: Set between 0-255. The lower the number, the higher the priority. 0 is the master and higher numbers are backups.
  • pass: The same password for all the members of the vhid. It’s highly recommended to generate a different password for each vhid.
  • alias: This is the virtual IP that will be active on the master. When a new master is activated, the IP will move to that host.

We are using the firewall in the load balancer VLAN for our default gateway.

Contents of our /etc/rc.conf.d/routing:

defaultrouter="10.0.0.1"

CARP Preempt

Add preempt to /etc/sysctl.conf to allow a backup to take over.

net.inet.carp.preempt=1

Installing Redis

Install via pkg

The quickest way to install Redis is via the package manager. You can install from ports and do a make config to automatically install the redis-trib.rb script (TRIB=ON). The downside is that you have to keep ports tree updated and run portmaster to rebuild the updated ports.

sudo pkg install redis

Install Management Script

Check for Recent Ruby

The script requires a recent version of ruby installed

ruby -v

You can install Ruby 2.3 if you don’t have ruby installed or have an old version.

sudo pkg install ruby23

Pull the management script from the Redis website

curl -O http://download.redis.io/redis-stable/src/redis-trib.rb

Make Script Executable

chmod +x redis-trib.rb

Move Script to /usr/local/bin/

sudo mv redis-trib.rb /usr/local/bin/

Configure Redis

We’re configuring 3 master and 3 slaves since Redis clusters require a minimum of 3 masters to create a cluster and maintain a quorum. We’re doing this on 3 hosts for additional performance and redundancy. Any host could run on its own if we lost the other 2 hosts.

Create a Configuration File for Each Instance

Each instance on a host requires a config file named redis-{$NAME}.conf. The general convention is to use the port number for the name of each instance e.g. redis-7000.conf.

Delete the default config since we won’t be using it for clustering. It could cause issues if loaded during startup and keeps the directory cleaner.

sudo rm /usr/local/etc/redis.conf

Create /usr/local/etc/redis-7000.conf:

bind 10.0.0.11 192.168.10.11 127.0.0.1
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /var/db/redis/7000
pidfile /var/run/redis/7000.pid
daemonize yes

Create redis-7001.conf through redis-7005.conf files. Substitute the port number in the port, dir, and pidfile entries. Adjust the bind IP addresses for each host as well.

Create the Working Dir for Each Instance

Each instance needs a working directory to store the db file to disk and auto-created files. These are stored in /var/db/redis/*. We need to create directories owned by the redis user named for each instance (port number).

cd /var/db/redis/

sudo mkdir 7000 7001 7002 7003 7004 7005

sudo chown -R redis:redis *

Add Startup And Profiles To /etc/rc.conf.d/redis

FreeBSD doesn’t start installed services automatically after install. We will tell it to start and tell it which profiles (instances) to start.

sudo vim /etc/rc.conf.d/redis

redis_enable="YES"
redis_profiles="7000 7001 7002 7003 7004 7005"

Start The Instances

sudo service redis start

We now have 6 unconfigured instances ready to be clustered.

Cluster the Redis Instances

We are ready to use the redis-trib.rb management script to configure all of the instances on all 3 of our hosts into a unified cluster.

redis-trib.rb create --replicas 1 10.0.0.11:7000 10.0.0.11:7001
10.0.0.11:7002 10.0.0.11:7003 10.0.0.11:7004 10.0.0.11:7005
10.0.0.12:7000 10.0.0.12:7001 10.0.0.12:7002 10.0.0.12:7003
10.0.0.12:7004 10.0.0.12:7005 10.0.0.13:7000 10.0.0.13:7001
10.0.0.13:7002 10.0.0.13:7003 10.0.0.13:7004 10.0.0.13:7005

redis-trib.rb breakdown:

  • create: Tells the script to create a cluster.
  • replicas: Tells how many slaves per master to create.
  • list of instances: These are all the unconfigured instances across all 3 hosts.

Keep in mind that redis is single threaded and needs several masters to get the performance you are looking for. Extra slaves are good but have plenty of masters as well. You can add more slaves afterwards if you want.

The script will list how it’s going to configure each instance then ask you if this looks good. Type “yes” to let it configure the instances into a cluster. It will distribute the slaves across hosts to minimize losing both master/slave in the event of a lost host.

Virtual IP Binding Cluster Issue

I reset all the instances to rebuild the cluster fresh for production. It would timeout (well, sit for a couple hours waiting) on the cluster creation when the virtual IP (10.0.0.10) was bound in the instances. I originally removed the virtual IP, created the cluster, and re-added the virtual IP. While this works, it’s annoying as the number of instances goes up.

I realized that it would be much better to just redirect traffic from the virtual IP to the physical IP at the firewall level. This way the local instances don’t need to know or care about the virtual IP. It will look like all traffic is coming directly to the physical IP.

What to do if the Cluster Create Fails

Once in a while, the cluster script hangs up while creating the cluster. You can kill the script (control-c), stop the instances on all hosts, delete the contents of the data directories, restart the services, and run the create again.

sudo service redis stop

sudo rm /var/db/redis/700?/*

sudo service redis start

Firewall Setup to Limit VLAN1 Access

Redis security is very minimal in standalone and replication modes. It’s even more limited in cluster mode. Primarily, the webservers will be accessing the cluster directly from the load balancer network. We do want some limited access to the senior Web Developers and the System Admins to monitor and troubleshoot from their machines. We created a basic blocking of the VLAN1 IPs on the redis ports (7000-7100) then allow only specific machines to access them from VLAN1.

Edit pf.conf

FreeBSD uses the Packet Filter (PF) firewall and /etc/pf.conf is the main configuration file. The /etc/redis_allow file is a list of authorized IP addresses. The redirect line takes traffic targeted for the virtual IP and redirects to the physical IP on the interface. This makes the failover process transparent to the instances and removes the requirement to bind to the virtual IP on each instance.

Contents of our pf.conf:

### Basic Redis Firewall Setup ###
#
### Interfaces ###
int_if="em0"
dmz_if="em1"
virtual="10.0.0.10"
#
## Tables ###
table <redis_allow> persist file "/etc/redis_allow"
#
# Redirect virtual IP to primary
rdr on $dmz_if proto tcp from any to $virtual -> $dmz_if:0
#
### Block/Pass ###
block in on $int_if proto tcp from any to any port 7000:7100
pass in on $int_if proto tcp from <redis_allow> to any port 7000:7100 keep state
pass out keep state

Installing x84 BBS on Raspberry Pi Zero running FreeBSD 11

19, Feb, 2017

Back in the 80s, I had an Apple ][gs with a 1200baud modem. I lived in farm country and everything was long distance. The next town over was 9 miles away. Because there was a county line there, it was an intraLATA call and long distance. I did have a friend in the next town and we connected via modems to chat and transfer files. Everything was all manual modem AT commands but it was fun. The problem was that my parents didn’t like me tying up the phone line for too long. Plus it got expensive for teenager with little extra cash to pay for the phone bill. Kids and parents worrying about overages is not a new thing. ;-)

I had a subscription to The Computer Shopper and it was a massive catalog packed with articles and ads from every computer company of the time. It took a lot of time to go through every month (usually the next issue was here before you were done). It was a great wealth of computer information in the pre-Internet days. I learned about dialup Bulletin Board Systems (BBS) and tried dialing up a few free boards but rarely staying on for very long at a time because I knew it was going to rack up the bill quickly.

Bulltin Boards started falling out of popularity as the Internet came on the scene. A lot of the functionality provided by a BBS was now available on websites. Many shut down and a few added telnet so you could reach them over the Internet. There’s still many BBS that are still running today. I decided that maybe it’s time to finally go play with some BBS since I didn’t get much of a chance the 1st time around.

I decided to run this on my Raspberry Pi Zero because the single core ARM runs the ANSI screens at a reasonable speed. The original Raspberry Pi A or B model would be good candidates too. I tried it on a quad core system and the screens went by way too fast to read/see properly. Logins are a little slow but that’s keeping with the feel of how things used to be. I bypassed a lot of proprietary BBS software because it wasn’t going to be able to run on my hardware/OS and didn’t give you much room to tweak. I wanted to keep to software that could run on many platforms (python, ruby, etc.). I did find a node.js BBS that looked nice but there’s not a lot of node packages for FreeBSD/ARM. Then I ran across x84. It’s written in Python and thought it looked very promising. I tried it out and decided it’s a keeper (as long as I don’t find something better).

Down to the actual install: (finally)

Let me help you setup all the pre-requisites and tweaks so you don’t have to spend hours of googling and trial & error. I went through 3 or 4 installs to figure this all out. Here’s some packages to get everything prepped for the install.

sudo pkg install gcc py27-pip py27-sqlite3 py27-gmpy 
py27-wcwidth py27-requests py27-six py27-more-itertools  
py27-jaraco.timing py27-jaraco.util py27-irc py27-dateutil 
py27-feedparser py27-html2text py27-pycparser py27-cffi 
py27-bcrypt py27-idna py27-pyasn1 py27-enum34 py27-ipaddress 
py27-cryptography py27-ecdsa py27-paramiko py27-openssl 
webpy py27-cherrypy curl bash

FreeBSD uses some cross-compile tools on some embedded platforms (mips, arm, aarch64, etc.) which aren’t used in this setup and will cause build errors. Change all references of /nxb-bin/usr/bin/cc to /usr/local/bin/gcc in /usr/local/lib/python2.7/_sysconfigdata.py to remove the cross-compile dependencies and allow gcc to build the remaining modules.

sudo sed -i 's//nxb-bin/usr/bin/cc//usr/local/bin/gcc/g' 
 /usr/local/lib/python2.7/_sysconfigdata.py

Install x84 via pip. The option with_crypto enables ssh and ssl if you want to use the embedded webserver. You can omit this if you want telnet only though ssh would be recommended.

pip install --user x84[with_crypto]

Start it up (~/.local/bin/x84) to generate the default config. Let it finish starting up then control-c to kill the BBS.

Edit ~/.x84/default.ini and change 127.0.0.1 to the IP address of your network interface. You can also set bbsname while you’re editing.

Fire it back up in tmux (or screen if you prefer). Login with telnet on port 6023 or ssh -p 6022 new@bbsaddress (or anonymous@) to register a new user. The first user setup is automatically a sysop and can change system settings from within the BBS.

Screenshot of default setup. I used cool-retro-term with DOS profile set to at least 80 column wide. Nothing like adding scanlines, phosphor refresh, static, etc. to make it feel more old school. Hyper Term has a similar retro theme.

Now you have a default install of x84 BBS running on your Raspberry Pi. Time to read the docs to learn how to use and customize your setup. Read the doors section to figure out how to run any external ascii/curses program from the menu (returns to the menu after exit). I’ve played with silly apps like nyancat, cmatrix, etc. but have been messing with frotz to play old Infocom text adventure games too. I’ve looked through every command line, curses game for FreeBSD on Freshports to see what might be a good fit on my BBS.

The ansi screen files are in the following location:
~/.local/lib/python2.7/site-packages/x84/default/art/*.ans

You can view and edit ANSI/ASCII files with PabloDraw. It’s not a bad idea to have figlet, cowsay, etc. to generate some ascii art to get you off the ground. You can view and download some ANSI files from the Sixteen Colors ANSI Art site.