High Availability on Vultr with Floating IP and BGP
Vultr allows you to combine two of our features (Floating IPs and BGP) in order to achieve high availability.
Setup
You'll need two instances in the same location, and a floating IP. You'll also need to open a ticket requesting BGP be setup on a private ASN for floating IPs. (You can also use this feature if you're running BGP with us on a public ASN)
You'll need a BGP daemon as well, we recommend BIRD. BIRD is usually available via your operating system's package manager.
We'll use 192.0.2.10/32 as our example floating IP and 198.51.100.99 as the IP address of one of our instances.
Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.
IP Configuration
We're going to use a Linux "dummy" interface to bind the IP address to. You can create this with the following commands:
ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.0.2.10/32
Confirm that this was configured properly:
# ip addr show dev dummy1
5: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether ba:23:57:2c:ad:bc brd ff:ff:ff:ff:ff:ff
inet 192.0.2.10/32 scope global dummy1
BIRD Configuration
Next, we'll configure BIRD. These instructions vary slightly between host operating systems, see the footnotes at the bottom of this section.
Create an /etc/bird.conf file:
log "/var/log/bird" all;
router id 198.51.100.99;
protocol device
{
scan time 60;
}
protocol direct
{
interface "dummy1";
}
protocol bgp vultr
{
local as <<YOURAS>>;
source address 198.51.100.99;
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.254 as 64515;
password "<<YOURPASSWORD>>";
}
You'll need to update YOURAS and YOURPASSWORD with the AS number and BGP password assigned to your account. This information can be found on the BGP tab of an instance page in the Vultr control panel.
This config file will tell BIRD to look for the dummy1 interface, and advertise any IPs it finds to our infrastructure via BGP. This means that as soon as your instance is running, you'll start receiving traffic, and if it ever crashes traffic will stop.
FreeBSD configuration
If you're using FreeBSD as a host instead of Linux, there are a few differences.
The kernel needs to be recompiled for TCP MD5 signature support. These instructions are outside of the scope of this article. If your BSD kernel does not support TCP MD5 signatures, you will see the following output in the BIRD log.
$ cat /var/log/bird 2017-12-15 01:35:00 <INFO> Started 2017-12-15 01:35:00 <ERR> vultr: Socket error: Kernel does not support TCP MD5 signaturesThe BIRD configuration file is located at
/usr/local/etc/bird.confon BSD.
Verify BGP Connectivity
Start the BIRD service service bird start, and wait a few seconds. Check that the BGP session has been established:
# birdc show proto all vultr
BIRD 1.5.0 ready.
name proto table state since info
vultr BGP master up 2016-01-15 Established
Preference: 100
Input filter: REJECT
Output filter: ACCEPT
Routes: 0 imported, 1 exported, 0 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 255919581 0 255919581 0 0
Import withdraws: 1905513 0 --- 257825094 0
Export updates: 1 0 0 --- 1
Export withdraws: 0 --- --- --- 0
BGP state: Established
Neighbor address: 169.254.169.254
Neighbor AS: YOURAS
Neighbor ID: 169.254.169.254
Neighbor caps: refresh enhanced-refresh restart-able AS4
Session: external multihop AS4
Source address: 198.51.100.99
Hold timer: 184/240
Keepalive timer: 30/80
If everything is working properly, you should see "Established" next to BGP state. A common problem here is having a firewall blocking the BGP port (TCP 179). Also, if this instance was deployed before Vultr set up your BGP session, it will need to be restarted via the control panel before BGP is available. If you're still having problems, look at /var/log/bird for further details.
Testing
You can make sure that BIRD is advertising the route to your floating IP with the following:
# birdc show route
BIRD 1.5.0 ready.
192.0.2.10/32 dev dummy1 [direct1 2015-12-29] * (240)
To confirm that this is working properly, you can disable the dummy1 interface (with ip link set dummy1 down), then repeat the show route command. BIRD will have noticed that the interface has disappeared and will withdraw the route.
Production Usage
In order to ensure that your site remains up, you'd want more than one server running the same BGP configuration. If any one of the instances goes down, traffic would get dynamically redirected to one of the other instances. There is no limit to the number of instances you can run with this configuration in a particular location, however only one of them will be active at any given time.
In some of our locations, traffic will be distributed randomly between any instances that you have configured this way. Eventually, all the locations will be configured in this way. If you want to have one instance get all the traffic unless it's offline, you'd want to use prepends to steer traffic.
For example, if you have two instances:
- Instance A - main instance, should receive all traffic normally
- Instance B - backup instance, should receive traffic only if Instance A is down
To accomplish this, add the following section to your BIRD configuration on instance B as follows:
export filter {
bgp_path.prepend(YOURAS);
accept;
};
This would ensure that traffic will always go to Instance A, unless it's down.
If you have an Instance C, that should only receive traffic when A and B are down, you can simply add another 'bgp_path.prepend' line to accomplish this.
IPv6 Setup
This process will also work with IPv6 reserved subnets, although you'll use "bird6" instead of "bird" and "birdc6" instead of "birdc".