ok, recently i am trying to setup a low cost HA on the cloud. And i decide to go with Vultr since they provide $2.5 machine for me to test and play around (although i can only spam 2 of such machine) Here i will drop down how i did it so i won't forget and you can copy it if you want.
Initial Cloud VPS Setup
ok here, you will need at least 3 machines, i'll call them the following with the spec i've used to create this setup
- Ubuntu 17.10 x64 1024 MB Server - 192.168.23.10 [Worker1]
- Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker2]
- Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker3]
- Floating IP - 192.168.10.10
i name them worker1, worker2 and worker3 as show above. Now, the reason why i choose Ubuntu 17 instead of CoreOS is purely because CoreOS is stuck at Docker 1.12 the time i am writing and you need to install its Alpha version to get Docker 17.09-ce
Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.
Installing lastest
Docker on Ubuntu 17.10
Fire the following comment on each machine,
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common -y curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt-get update sudo apt-get install docker-ce -y
and if it doesn't work, most likely docker doesn't support your newest Ubuntu and you have to switch it to a lower one as show below,
vi /etc/apt/sources.list
change artful to zesty
deb [arch=amd64] https://download.docker.com/linux/ubuntu zesty stable
then go ahead and install again.
Once its done, check the version. You should get something like this below,
root@worker1:/etc/bird# docker -v Docker version 17.09.0-ce, build afdb6d4
once this is done, we are going to install blue for BGP
Installing and Configuring BGP using Bird in Ubuntu 17.10
First, do yourself a favor and add its repo
sudo add-apt-repository ppa:cz.nic-labs/bird apt-get update -y
once this is done, you'll get an error because it doesn't support Ubuntu 17.10 yet. Hence, gonna go downwards again.
vi /etc/apt/sources.list.d/cz_nic-labs-ubuntu-bird-artful.list
change it to the below,
deb http://ppa.launchpad.net/cz.nic-labs/bird/ubuntu xenial main
save it and try to install bird below,
sudo apt-get install bird
And we have bird installed. Now to configure it, we will paste the following into /etc/bird/bird.conf
log "/var/log/bird" all;
router id 192.168.23.10; # instance ip
protocol device
{
scan time 60;
}
protocol direct
{
interface "dummy1";
}
protocol bgp vultr
{
local as 11512; #instances ASN number
source address 192.168.23.10; #instance ip
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.252 as 64019; #vultr IP and ASN number
password "xC4c2EGaXuP7a74!"; #vulr password
}
then proceed to restart
service bird restart
service bird status
● bird.service - BIRD Internet Routing Daemon (IPv4)
Loaded: loaded (/lib/systemd/system/bird.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2017-10-30 10:30:51 UTC; 19min ago
Process: 5140 ExecStartPre=/usr/sbin/bird -p (code=exited, status=0/SUCCESS)
Process: 5130 ExecStartPre=/usr/lib/bird/prepare-environment (code=exited, status=0/SUCCESS)
Main PID: 5145 (bird)
Tasks: 1 (limit: 4915)
Memory: 640.0K
CPU: 53ms
CGroup: /system.slice/bird.service
└─5145 /usr/sbin/bird -f -u bird -g bird
Oct 30 10:30:51 worker_a systemd[1]: Starting BIRD Internet Routing Daemon (IPv4)...
Oct 30 10:30:51 worker_a systemd[1]: Started BIRD Internet Routing Daemon (IPv4).
now make sure you have the above, if not, its most likely because your log permission is set as root, if that's the case, try the below,
chown bird:bird /var/log/bird
and restart bird again. Ensure your firewall isn't blocking port 179 for BGP to communicate with your machine. Once this is done, you can test it out externally via telnet
telnet 192.168.23.10 179 Trying 192.168.23.10... Connected to 192.168.23.10.vultr.com. Escape character is '^]'. Connection closed by foreign host.
you should get something like this. Now, the most important part if not you will get the following
birdc show proto all vultr
BIRD 1.6.3 ready.
name proto table state since info
vultr BGP master start 13:49:30 Idle Received: Bad peer AS
Preference: 100
Input filter: REJECT
Output filter: ACCEPT
Routes: 0 imported, 0 exported, 0 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 0 0 0 0 0
Import withdraws: 0 0 --- 0 0
Export updates: 0 0 0 --- 0
Export withdraws: 0 --- --- --- 0
BGP state: Idle
Neighbor address: 169.254.169.254
Neighbor AS: 64515
Error wait: 10/300
Last error: Received: Bad peer AS
Setup Dummy IP Configuration
run the following to create a dummy IP configuration for your Bird to forward to
ip link add dev dummy1 type dummy ip link set dummy1 up ip addr add dev dummy1 192.168.10.10
where 192.168.10.10 is the floating IP. Now you can test whether the above setup is correct with the following
ip addr show dev dummy1
6: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 4e:79:76:36:6d:b0 brd ff:ff:ff:ff:ff:ff
inet 45.77.110.78/32 scope global dummy1
valid_lft forever preferred_lft forever
inet6 fe80::4c79:76ff:fe36:6db0/64 scope link
valid_lft forever preferred_lft forever
Now, restart your Bird so that it gets the above dummy. To set it persistently, do the following
vi /etc/rc.local ip link add dev dummy1 type dummy ip link set dummy1 up ip addr add dev dummy1 192.168.10.10/32
then restart or reboot your machine
service bird restart
Once this is done, you should get the following,
birdc show proto all vultr
BIRD 1.6.3 ready.
name proto table state since info
vultr BGP master up 14:44:15 Established
Preference: 100
Input filter: REJECT
Output filter: ACCEPT
Routes: 0 imported, 1 exported, 0 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 0 0 0 0 0
Import withdraws: 0 0 --- 0 0
Export updates: 1 0 0 --- 1
Export withdraws: 0 --- --- --- 0
BGP state: Established
Neighbor address: 169.254.169.254
Neighbor AS: 64515
Neighbor ID: 208.167.241.119
Neighbor caps: refresh restart-aware AS4 add-path-rx
Session: external multihop AS4
Source address: 45.77.78.111
Hold timer: 132/180
Keepalive timer: 29/60
Once you get this, all should be good. Try it out.
