High Availability on Vultr with Floating IP and BGP

Vultr allows you to combine two of our features (Floating IPs and BGP) in order to achieve high availability.

Setup

You'll need two instances in the same location, and a floating IP. You'll also need to open a ticket requesting BGP be setup on a private ASN for floating IPs. (You can also use this feature if you're running BGP with us on a public ASN)

You'll need a BGP daemon as well, we recommend BIRD. BIRD is usually available via your operating system's package manager.

We'll use 192.0.2.10/32 as our example floating IP and 198.51.100.99 as the IP address of one of our instances.

Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.

IP Configuration

We're going to use a Linux "dummy" interface to bind the IP address to. You can create this with the following commands:

ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.0.2.10/32

Confirm that this was configured properly:

# ip addr show dev dummy1
5: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether ba:23:57:2c:ad:bc brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.10/32 scope global dummy1

BIRD Configuration

Next, we'll configure BIRD. These instructions vary slightly between host operating systems, see the footnotes at the bottom of this section.

Create an /etc/bird.conf file:

log "/var/log/bird" all;

router id 198.51.100.99;

protocol device
{
    scan time 60;
}

protocol direct
{
    interface "dummy1";
}

protocol bgp vultr
{
    local as <<YOURAS>>;
    source address 198.51.100.99;
    import none;
    export all;
    graceful restart on;
    next hop self;
    multihop 2;
    neighbor 169.254.169.254 as 64515;
    password "<<YOURPASSWORD>>";
}

You'll need to update YOURAS and YOURPASSWORD with the AS number and BGP password assigned to your account. This information can be found on the BGP tab of an instance page in the Vultr control panel.

This config file will tell BIRD to look for the dummy1 interface, and advertise any IPs it finds to our infrastructure via BGP. This means that as soon as your instance is running, you'll start receiving traffic, and if it ever crashes traffic will stop.

FreeBSD configuration

If you're using FreeBSD as a host instead of Linux, there are a few differences.

1. The kernel needs to be recompiled for TCP MD5 signature support. These instructions are outside of the scope of this article. If your BSD kernel does not support TCP MD5 signatures, you will see the following output in the BIRD log.

   $ cat /var/log/bird
   2017-12-15 01:35:00 <INFO> Started
   2017-12-15 01:35:00 <ERR> vultr: Socket error: Kernel does not support TCP MD5 signatures
   

2. The BIRD configuration file is located at /usr/local/etc/bird.conf on BSD.

Verify BGP Connectivity

Start the BIRD service service bird start, and wait a few seconds. Check that the BGP session has been established:

# birdc show proto all vultr
BIRD 1.5.0 ready.
name     proto    table    state  since       info
vultr    BGP      master   up     2016-01-15  Established
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 1 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:      255919581          0  255919581          0          0
    Import withdraws:      1905513          0        ---  257825094          0
    Export updates:              1          0          0        ---          1
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Established
    Neighbor address: 169.254.169.254
    Neighbor AS:      YOURAS
    Neighbor ID:      x.x.x.x (Host IP)
    Neighbor caps:    refresh enhanced-refresh restart-able AS4
    Session:          external multihop AS4
    Source address:   198.51.100.99
    Hold timer:       184/240
    Keepalive timer:  30/80

If everything is working properly, you should see "Established" next to BGP state. A common problem here is having a firewall blocking the BGP port (TCP 179). Also, if this instance was deployed before Vultr set up your BGP session, it will need to be restarted via the control panel before BGP is available. If you're still having problems, look at /var/log/bird for further details.

Testing

You can make sure that BIRD is advertising the route to your floating IP with the following:

# birdc show route
BIRD 1.5.0 ready.
192.0.2.10/32    dev dummy1 [direct1 2015-12-29] * (240)

To confirm that this is working properly, you can disable the dummy1 interface (with ip link set dummy1 down), then repeat the show route command. BIRD will have noticed that the interface has disappeared and will withdraw the route.

Production Usage

In order to ensure that your site remains up, you'd want more than one server running the same BGP configuration. If any one of the instances goes down, traffic would get dynamically redirected to one of the other instances. There is no limit to the number of instances you can run with this configuration in a particular location, however only one of them will be active at any given time.

In some of our locations, traffic will be distributed randomly between any instances that you have configured this way. Eventually, all the locations will be configured in this way. If you want to have one instance get all the traffic unless it's offline, you'd want to use prepends to steer traffic.

For example, if you have two instances:

• Instance A - main instance, should receive all traffic normally
• Instance B - backup instance, should receive traffic only if Instance A is down

To accomplish this, add the following section to your BIRD configuration on instance B as follows:

export filter {
    bgp_path.prepend(YOURAS);
    accept;
};

This would ensure that traffic will always go to Instance A, unless it's down.

If you have an Instance C, that should only receive traffic when A and B are down, you can simply add another 'bgp_path.prepend' line to accomplish this.

IPv6 Setup

This process will also work with IPv6 reserved subnets, although you'll use "bird6" instead of "bird" and "birdc6" instead of "birdc".