High Availability on Vultr with Floating IP and BGP
Use this guide to combine two of Vultr's features (Floating IPs and BGP) in order to achieve high availability. We also have BGP guides available if you plan to bring your own IP space to any of our locations.
Setup
You'll need two instances in the same location, and a floating IP. See our instructions to reserve a floating IP. The reserved IP, and the instances, must be in the same location. You'll also need to open a ticket requesting BGP be setup on a private ASN for floating IPs. (You can also use this feature if you're running BGP with us on a public ASN)
You'll need a BGP daemon as well, we recommend BIRD. BIRD is usually available via your operating system's package manager.
We'll use 192.0.2.10/32 as our example floating IP and 198.51.100.99 as the IP address of one of our instances.
Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.
IP Configuration
We're going to use a Linux "dummy" interface to bind the IP address to. You can create this with the following commands:
ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.0.2.10/32
Confirm that this was configured properly:
# ip addr show dev dummy1
5: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether ba:23:57:2c:ad:bc brd ff:ff:ff:ff:ff:ff
inet 192.0.2.10/32 scope global dummy1
BIRD Configuration
Next, we'll configure BIRD. These instructions vary slightly between host operating systems, see the footnotes at the bottom of this section.
Create an /etc/bird/bird.conf file:
log "/var/log/bird.log" all;
router id 198.51.100.99;
protocol device
{
scan time 60;
}
protocol direct
{
interface "dummy1";
}
protocol bgp vultr
{
local as <<YOURAS>>;
source address 198.51.100.99;
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.254 as 64515;
password "<<YOURPASSWORD>>";
}
You'll need to update YOURAS and YOURPASSWORD with the AS number and BGP password assigned to your account. This information can be found on the BGP tab of an instance page in the Vultr control panel.
This config file will tell BIRD to look for the dummy1 interface, and advertise any IPs it finds to our infrastructure via BGP. This means that as soon as your instance is running, you'll start receiving traffic, and if it ever crashes traffic will stop.
BIRD on Bare Metal
On Bare Metal servers, the neighbor address and Vultr AS value are different - replace that line in the above /etc/bird/bird.conf configuration with the following:
neighbor 169.254.1.1 as 20473;
FreeBSD configuration
If you're using FreeBSD as a host instead of Linux, there are a few differences.
The kernel needs to be recompiled for TCP MD5 signature support. These instructions are outside of the scope of this article. If your BSD kernel does not support TCP MD5 signatures, you will see the following output in the BIRD log.
$ cat /var/log/bird.log 2017-12-15 01:35:00 <INFO> Started 2017-12-15 01:35:00 <ERR> vultr: Socket error: Kernel does not support TCP MD5 signaturesThe BIRD configuration file is located at
/usr/local/etc/bird.confon BSD.
Verify BGP Connectivity
Start the BIRD service service bird start, and wait a few seconds. Check that the BGP session has been established:
# birdc show proto all vultr
BIRD 1.5.0 ready.
name proto table state since info
vultr BGP master up 2016-01-15 Established
Preference: 100
Input filter: REJECT
Output filter: ACCEPT
Routes: 0 imported, 1 exported, 0 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 255919581 0 255919581 0 0
Import withdraws: 1905513 0 --- 257825094 0
Export updates: 1 0 0 --- 1
Export withdraws: 0 --- --- --- 0
BGP state: Established
Neighbor address: 169.254.169.254
Neighbor AS: YOURAS
Neighbor ID: x.x.x.x (Host Node IP)
Neighbor caps: refresh enhanced-refresh restart-able AS4
Session: external multihop AS4
Source address: 198.51.100.99
Hold timer: 184/240
Keepalive timer: 30/80
If everything is working properly, you should see "Established" next to BGP state. A common problem here is having a firewall blocking the BGP port (TCP 179). Also, if this instance was deployed before Vultr set up your BGP session, it will need to be restarted via the control panel before BGP is available. If you're still having problems, look at /var/log/bird for further details.
Testing
You can make sure that BIRD is advertising the route to your floating IP with the following:
# birdc show route
BIRD 1.5.0 ready.
192.0.2.10/32 dev dummy1 [direct1 2015-12-29] * (240)
To confirm that this is working properly, you can disable the dummy1 interface (with ip link set dummy1 down), then repeat the show route command. BIRD will have noticed that the interface has disappeared and will withdraw the route.
Production Usage
In order to ensure that your site remains up, you'd want more than one server running the same BGP configuration. If any one of the instances goes down, traffic would get dynamically redirected to one of the other instances. There is no limit to the number of instances you can run with this configuration in a particular location, however only one of them will be active at any given time.
In some of our locations, traffic will be distributed randomly between any instances that you have configured this way. Eventually, all the locations will be configured in this way. If you want to have one instance get all the traffic unless it's offline, you'd want to use prepends to steer traffic.
For example, if you have two instances:
- Instance A - main instance, should receive all traffic normally
- Instance B - backup instance, should receive traffic only if Instance A is down
To accomplish this, add the following section to your BIRD configuration on instance B as follows:
export filter {
bgp_path.prepend(YOURAS);
accept;
};
This would ensure that traffic will always go to Instance A, unless it's down.
If you have an Instance C, that should only receive traffic when A and B are down, you can simply add another 'bgp_path.prepend' line to accomplish this.
IPv6 Setup
This process will also work with IPv6 reserved subnets, although you'll use "bird6" instead of "bird" and "birdc6" instead of "birdc".
Dummy Interface IPv6 Configuration
As with IPv4, we’ll assume the instance’s Main IPv4 IP address is 198.51.100.99 for the sake of this example. Additionally, we’ll assume the instance’s Main IPv6 address is 2001:19f0:5:50e7:5400:3ff:fe06:4cf2 - please note, that the instance must have a public IPv6 subnet and address assigned to it for Floating IPv6 BGP to work.
Here, we’ll assume our Reserved IPv6 /64 subnet is 2001:19f0:5:6aaf::/64. Use our IPv6 Calculator to see the range of usable IP addresses (https://www.vultr.com/resources/subnet-calculator-ipv6/). You can assign as many IPv6 addresses within the Reserved /64 Subnet subnet as you need. Here we will do 2 of them:
ip addr add dev dummy1 2001:19f0:5:6aaf::1/64
ip addr add dev dummy1 2001:19f0:5:6aaf::2/64
Now to verify the interface:
# ip addr show dev dummy1
5: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether ba:23:57:2c:ad:bc brd ff:ff:ff:ff:ff:ff
inet 192.0.2.10/32 scope global dummy1
valid_lft forever preferred_lft forever
inet6 2001:19f0:5:6aaf::2/64 scope global
valid_lft forever preferred_lft forever
inet6 2001:19f0:5:6aaf::1/64 scope global
valid_lft forever preferred_lft forever
BIRD6 Configuration
Create an /etc/bird/bird.conf file:
log "/var/log/bird.log" all;
router id 198.51.100.99;
protocol device
{
scan time 60;
}
protocol direct
{
interface "dummy1";
}
protocol bgp vultr
{
local as <<YOURAS>>;
source address 2001:19f0:5:50e7:5400:3ff:fe06:4cf2;
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.254 as 64515;
password "<<YOURPASSWORD>>";
}
Verify IPv6 BGP Connectivity
Start the Bird6 process
systemctl start bird6
And then check the Bird6 console:
# birdc6 show route
BIRD 1.6.8 ready.
2001:19f0:5:6aaf::/64 dev dummy1 [direct1 18:17:21] * (240)
# birdc6 show proto all vultr
BIRD 1.6.8 ready.
name proto table state since info
vultr BGP master up 18:17:24 Established
Preference: 100
Input filter: REJECT
Output filter: ACCEPT
Routes: 0 imported, 1 exported, 0 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 279366 0 279366 0 0
Import withdraws: 2692 0 --- 282058 0
Export updates: 1 0 0 --- 1
Export withdraws: 0 --- --- --- 0
BGP state: Established
Neighbor address: 2001:19f0:ffff::1
Neighbor AS: 64515
Neighbor ID: X.X.X.X (Host Node IP)
Neighbor caps: refresh restart-aware AS4 add-path-rx
Session: external multihop AS4
Source address: 2001:19f0:5:50e7:5400:3ff:fe06:4cf2
Hold timer: 170/180
Keepalive timer: 53/60
From outside our network, we now have connectivity on the 2 addresses we added to the dummy.
# ping6 -c2 2001:19f0:5:6aaf::1
PING 2001:19f0:5:6aaf::1(2001:19f0:5:6aaf::1) 56 data bytes
64 bytes from 2001:19f0:5:6aaf::1: icmp_seq=1 ttl=44 time=154 ms
64 bytes from 2001:19f0:5:6aaf::1: icmp_seq=2 ttl=44 time=55.5 ms
--- 2001:19f0:5:6aaf::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 55.530/104.768/154.006/49.238 ms
# ping6 -c2 2001:19f0:5:6aaf::2
PING 2001:19f0:5:6aaf::2(2001:19f0:5:6aaf::2) 56 data bytes
64 bytes from 2001:19f0:5:6aaf::2: icmp_seq=1 ttl=44 time=55.4 ms
64 bytes from 2001:19f0:5:6aaf::2: icmp_seq=2 ttl=44 time=51.9 ms
--- 2001:19f0:5:6aaf::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 51.939/53.672/55.405/1.733 ms
Note: Sometimes it can take a few minutes 15 minutes for our network to pick up the IPv6 routes.
BIRD6 on Bare Metal
As with IPv4, the configuration varies slightly on Bare Metal - the neighbor is the same, but the as value is different.
neighbor 2001:19f0:ffff::1 as 20473;
Manage Reserved IPs via API
The Vultr API offers several endpoints to manage reserved IPs.
- Create a new reserved IP.
- List all reserved IPs in your account.
- Get information about a reserved IP.
- Attach a reserved IP to an instance.
- Detach a reserved IP.
- Convert the IP address on an instance into a reserved IP.
- Delete a reserved IP.
Want to contribute?
You could earn up to $300 by adding new articles
