FreeBSD Redis Cluster with CARP failover

Selecting FreeBSD for Redis Cluster

FreeBSD 11.0 has strong networking and memory management performance. It also does clustering in-kernel via CARP. This makes a great choice for high performance, high availability applications.

Some may argue that Linux is better documented and supported than FreeBSD. Stand-alone Redis installs and master/slave replication (often erroneously called clustering) using Sentinel to manage failovers are well documented. Clustering Redis docs generally refer to building from scratch or adding unofficial repos then manually building directories/configs. Information needs to be pieced together to make a clean, stable setup. FreeBSD makes a nice maintainable setup with little fuss.

Cluster the Hosts with Virtual IP

We are using CARP to give us a virtual IP to connect to the cluster. If the primary host is unavailable, the next CARP host will take over as master and the virtual IP will be active on that machine. If the clients are connecting to the virtual IP instead of one of the physical IPs, they will automatically connect to the next host if the primary fails. We are using PFSync to keep the state table in sync between the hosts. This will make failover transparent to the clients.

VMWare VM Config

For security concerns, we don't enable promiscuous mode on the VMWare vSwitches unless needed. We've created dedicated vSwitches with promiscuous enabled for CARP hosts. Choose Load Balancer Promiscuous (VLAN4) for the Load Balancer interface since this is where the clients will connect. We also need a dedicated interface to sync the traffic. We are using an interface on the Quorum VLAN with a different subnet assigned.

Enable PF, PFSync, and CARP in the Kernel

Add the modules to enable in the /boot/loader.conf.

carp_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

Configure Interfaces

We are using em0 on VLAN1, em1 is our load balancer interface, and em2 is our dedicated pfsync interface. The em1 interface is where the virtual IP (10.0.0.10) will be aliased.

Contents of our /etc/rc.conf.d/network:

ifconfig_em0="inet 192.168.10.11 netmask 255.255.255.0"
ifconfig_em1="inet 10.0.0.11 netmask 255.255.255.0"
ifconfig_em1_alias0="inet vhid 10 advskew 0 pass FMxjubyqfNYTVAdS6aZ alias 10.0.0.10/32"
ifconfig_em2="inet 10.0.1.11 netmask 255.255.0.0"
pfsync_enable="YES"
pfsync_syncdev="em2"

The em1 alias breakdown:

• vhid: The same number for the other hosts your are clustering. (This needs to be unique on the subnet to avoid collisions).
• advskew: Set between 0-255. The lower the number, the higher the priority. 0 is the master and higher numbers are backups.
• pass: The same password for all the members of the vhid. It's highly recommended to generate a different password for each vhid.
• alias: This is the virtual IP that will be active on the master. When a new master is activated, the IP will move to that host.

We are using the firewall in the load balancer VLAN for our default gateway.

Contents of our /etc/rc.conf.d/routing:

defaultrouter="10.0.0.1"

CARP Preempt

Add preempt to /etc/sysctl.conf to allow a backup to take over.

net.inet.carp.preempt=1

Installing Redis

Install via pkg

The quickest way to install Redis is via the package manager. You can install from ports and do a make config to automatically install the redis-trib.rb script (TRIB=ON). The downside is that you have to keep ports tree updated and run portmaster to rebuild the updated ports.

sudo pkg install redis

Install Management Script

Check for Recent Ruby

The script requires a recent version of ruby installed

ruby -v

You can install Ruby 2.3 if you don't have ruby installed or have an old version.

sudo pkg install ruby23

Pull the management script from the Redis website

curl -O http://download.redis.io/redis-stable/src/redis-trib.rb

Make Script Executable

chmod +x redis-trib.rb

Move Script to /usr/local/bin/

sudo mv redis-trib.rb /usr/local/bin/

Configure Redis

We're configuring 3 master and 3 slaves since Redis clusters require a minimum of 3 masters to create a cluster and maintain a quorum. We're doing this on 3 hosts for additional performance and redundancy. Any host could run on its own if we lost the other 2 hosts.

Create a Configuration File for Each Instance

Each instance on a host requires a config file named redis-{$NAME}.conf. The general convention is to use the port number for the name of each instance e.g. redis-7000.conf.

Delete the default config since we won't be using it for clustering. It could cause issues if loaded during startup and keeps the directory cleaner.

sudo rm /usr/local/etc/redis.conf

Create /usr/local/etc/redis-7000.conf:

bind 10.0.0.11 192.168.10.11 127.0.0.1
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /var/db/redis/7000
pidfile /var/run/redis/7000.pid
daemonize yes

Create redis-7001.conf through redis-7005.conf files. Substitute the port number in the port, dir, and pidfile entries. Adjust the bind IP addresses for each host as well.

Create the Working Dir for Each Instance

Each instance needs a working directory to store the db file to disk and auto-created files. These are stored in /var/db/redis/*. We need to create directories owned by the redis user named for each instance (port number).

cd /var/db/redis/

sudo mkdir 7000 7001 7002 7003 7004 7005

sudo chown -R redis:redis *

Add Startup And Profiles To /etc/rc.conf.d/redis

FreeBSD doesn't start installed services automatically after install. We will tell it to start and tell it which profiles (instances) to start.

sudo vim /etc/rc.conf.d/redis

redis_enable="YES"
redis_profiles="7000 7001 7002 7003 7004 7005"

Start The Instances

sudo service redis start

We now have 6 unconfigured instances ready to be clustered.

Cluster the Redis Instances

We are ready to use the redis-trib.rb management script to configure all of the instances on all 3 of our hosts into a unified cluster.

redis-trib.rb create --replicas 1 10.0.0.11:7000 10.0.0.11:7001  
10.0.0.11:7002 10.0.0.11:7003 10.0.0.11:7004 10.0.0.11:7005 
10.0.0.12:7000 10.0.0.12:7001 10.0.0.12:7002 10.0.0.12:7003 
10.0.0.12:7004 10.0.0.12:7005 10.0.0.13:7000 10.0.0.13:7001 
10.0.0.13:7002 10.0.0.13:7003 10.0.0.13:7004 10.0.0.13:7005

redis-trib.rb breakdown:

• create: Tells the script to create a cluster.
• replicas: Tells how many slaves per master to create.
• list of instances: These are all the unconfigured instances across all 3 hosts.

Keep in mind that redis is single threaded and needs several masters to get the performance you are looking for. Extra slaves are good but have plenty of masters as well. You can add more slaves afterwards if you want.

The script will list how it's going to configure each instance then ask you if this looks good. Type "yes" to let it configure the instances into a cluster. It will distribute the slaves across hosts to minimize losing both master/slave in the event of a lost host.

Virtual IP Binding Cluster Issue

I reset all the instances to rebuild the cluster fresh for production. It would timeout (well, sit for a couple hours waiting) on the cluster creation when the virtual IP (10.0.0.10) was bound in the instances. I originally removed the virtual IP, created the cluster, and re-added the virtual IP. While this works, it's annoying as the number of instances goes up.

I realized that it would be much better to just redirect traffic from the virtual IP to the physical IP at the firewall level. This way the local instances don't need to know or care about the virtual IP. It will look like all traffic is coming directly to the physical IP.

What to do if the Cluster Create Fails

Once in a while, the cluster script hangs up while creating the cluster. You can kill the script (control-c), stop the instances on all hosts, delete the contents of the data directories, restart the services, and run the create again.

sudo service redis stop

sudo rm /var/db/redis/700?/*

sudo service redis start

Firewall Setup to Limit VLAN1 Access

Redis security is very minimal in standalone and replication modes. It's even more limited in cluster mode. Primarily, the webservers will be accessing the cluster directly from the load balancer network. We do want some limited access to the senior Web Developers and the System Admins to monitor and troubleshoot from their machines. We created a basic blocking of the VLAN1 IPs on the redis ports (7000-7100) then allow only specific machines to access them from VLAN1.

Edit pf.conf

FreeBSD uses the Packet Filter (PF) firewall and /etc/pf.conf is the main configuration file. The /etc/redis_allow file is a list of authorized IP addresses. The redirect line takes traffic targeted for the virtual IP and redirects to the physical IP on the interface. This makes the failover process transparent to the instances and removes the requirement to bind to the virtual IP on each instance.

Contents of our pf.conf:

### Basic Redis Firewall Setup ###
#
### Interfaces ###
int_if="em0"
dmz_if="em1"
virtual="10.0.0.10"
#
## Tables ###
table <redis_allow> persist file "/etc/redis_allow"
#
# Redirect virtual IP to primary
rdr on $dmz_if proto tcp from any to $virtual -> $dmz_if:0
#
### Block/Pass ###
block in on $int_if proto tcp from any to any port 7000:7100
pass in on $int_if proto tcp from <redis_allow> to any port 7000:7100 keep state
pass out keep state