Clay, Author at Hungred Dot Com

Setup HA + Floating IP + BGP on Vultr on Ubuntu 17.10

ok, recently i am trying to setup a low cost HA on the cloud. And i decide to go with Vultr since they provide $2.5 machine for me to test and play around (although i can only spam 2 of such machine) Here i will drop down how i did it so i won't forget and you can copy it if you want.

Initial Cloud VPS Setup

ok here, you will need at least 3 machines, i'll call them the following with the spec i've used to create this setup

Ubuntu 17.10 x64 1024 MB Server - 192.168.23.10 [Worker1]
Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker2]
Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker3]
Floating IP - 192.168.10.10

i name them worker1, worker2 and worker3 as show above. Now, the reason why i choose Ubuntu 17 instead of CoreOS is purely because CoreOS is stuck at Docker 1.12 the time i am writing and you need to install its Alpha version to get Docker 17.09-ce

Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.

Installing lastest
Docker on Ubuntu 17.10

Fire the following comment on each machine,

sudo apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce -y

and if it doesn't work, most likely docker doesn't support your newest Ubuntu and you have to switch it to a lower one as show below,

vi /etc/apt/sources.list

change artful to zesty

deb [arch=amd64] https://download.docker.com/linux/ubuntu zesty stable

then go ahead and install again.
Once its done, check the version. You should get something like this below,

root@worker1:/etc/bird# docker -v
Docker version 17.09.0-ce, build afdb6d4

once this is done, we are going to install blue for BGP

Installing and Configuring BGP using Bird in Ubuntu 17.10

First, do yourself a favor and add its repo

sudo add-apt-repository ppa:cz.nic-labs/bird
apt-get update -y

once this is done, you'll get an error because it doesn't support Ubuntu 17.10 yet. Hence, gonna go downwards again.

vi /etc/apt/sources.list.d/cz_nic-labs-ubuntu-bird-artful.list

change it to the below,

deb http://ppa.launchpad.net/cz.nic-labs/bird/ubuntu xenial main

save it and try to install bird below,

sudo apt-get install bird

And we have bird installed. Now to configure it, we will paste the following into /etc/bird/bird.conf

log "/var/log/bird" all;

router id 192.168.23.10; # instance ip

protocol device
{
scan time 60;
}

protocol direct
{
interface "dummy1";
}

protocol bgp vultr
{
local as 11512; #instances ASN number
source address 192.168.23.10; #instance ip
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.252 as 64019; #vultr IP and ASN number
password "xC4c2EGaXuP7a74!"; #vulr password
}

then proceed to restart

service bird restart
service bird status
● bird.service - BIRD Internet Routing Daemon (IPv4)
   Loaded: loaded (/lib/systemd/system/bird.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-10-30 10:30:51 UTC; 19min ago
  Process: 5140 ExecStartPre=/usr/sbin/bird -p (code=exited, status=0/SUCCESS)
  Process: 5130 ExecStartPre=/usr/lib/bird/prepare-environment (code=exited, status=0/SUCCESS)
 Main PID: 5145 (bird)
    Tasks: 1 (limit: 4915)
   Memory: 640.0K
      CPU: 53ms
   CGroup: /system.slice/bird.service
           └─5145 /usr/sbin/bird -f -u bird -g bird

Oct 30 10:30:51 worker_a systemd[1]: Starting BIRD Internet Routing Daemon (IPv4)...
Oct 30 10:30:51 worker_a systemd[1]: Started BIRD Internet Routing Daemon (IPv4).

now make sure you have the above, if not, its most likely because your log permission is set as root, if that's the case, try the below,

chown bird:bird /var/log/bird

and restart bird again. Ensure your firewall isn't blocking port 179 for BGP to communicate with your machine. Once this is done, you can test it out externally via telnet

telnet 192.168.23.10 179
Trying 192.168.23.10...
Connected to 192.168.23.10.vultr.com.
Escape character is '^]'.
Connection closed by foreign host.

you should get something like this. Now, the most important part if not you will get the following

birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   start  13:49:30    Idle          Received: Bad peer AS
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 0 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              0          0          0        ---          0
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Idle
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Error wait:       10/300
    Last error:       Received: Bad peer AS

Setup Dummy IP Configuration

run the following to create a dummy IP configuration for your Bird to forward to

ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.168.10.10

where 192.168.10.10 is the floating IP. Now you can test whether the above setup is correct with the following

ip addr show dev dummy1
6: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 4e:79:76:36:6d:b0 brd ff:ff:ff:ff:ff:ff
    inet 45.77.110.78/32 scope global dummy1
       valid_lft forever preferred_lft forever
    inet6 fe80::4c79:76ff:fe36:6db0/64 scope link
       valid_lft forever preferred_lft forever

Now, restart your Bird so that it gets the above dummy. To set it persistently, do the following

vi /etc/rc.local
ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.168.10.10/32

then restart or reboot your machine

service bird restart

Once this is done, you should get the following,

birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   up     14:44:15    Established
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 1 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              1          0          0        ---          1
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Established
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Neighbor ID:      208.167.241.119
    Neighbor caps:    refresh restart-aware AS4 add-path-rx
    Session:          external multihop AS4
    Source address:   45.77.78.111
    Hold timer:       132/180
    Keepalive timer:  29/60

Once you get this, all should be good. Try it out.

Yii2 Restful API UploadedFile Hack

ok, here a little hack that i did because i couldn't get the tutorial one working on the following link

http://www.yiiframework.com/doc-2.0/guide-input-file-upload.html

with the following code

namespace app\controllers;

use Yii;
use yii\web\Controller;
use app\models\UploadForm;
use yii\web\UploadedFile;

class SiteController extends Controller
{
    public function actionUpload()
    {
        $model = new UploadForm();

        if (Yii::$app->request->isPost) {
            $model->imageFile = UploadedFile::getInstance($model, 'imageFile');
            if ($model->upload()) {
                // file is uploaded successfully
                return;
            }
        }

        return $this->render('upload', ['model' => $model]);
    }
}

Sadly, no matter how hard you try, $model->imageFile will always equal to null. If you are working on an Upload API, and you are trying to get your image or whatever upload. You might soon figure that this is an impossible task. But if you try to return $_FILES['imageFile'], it does show you some hot stuff which you have uploaded but the given code just doesn't work! Hence, I have a little hack to FORCE it to work for me at least.

By digging the code and logic within yii\web\UploadedFile i came down with the solution below,

namespace app\controllers;

use Yii;
use yii\web\Controller;
use app\models\UploadForm;
use yii\web\UploadedFile;

class SiteController extends Controller
{
    public function actionUpload()
    {
        $model = new UploadForm();

        if (Yii::$app->request->isPost) {
                        $file = $_FILES['imageFile'];
                        $model->imageFile = new UploadedFile( [
                                'name' => $file['name'],
                                'tempName' => $file['tmp_name'],
                                'type' => $file['type'],
                                'size' => $file['size'],
                                'error' => $file['error'],
                                ]);
            if ($model->upload()) {
                // file is uploaded successfully
                return;
            }
        }

        return $this->render('upload', ['model' => $model]);
    }
}

Its exactly the same as what is given just that i manually created the instances instead of using the method getInstance since getInstanceByName also doesn't work. Well, hopefully this help some poor folk out there. Good Luck!

ping: icmp open socket: Operation not permitted Centos 6 LXC

If you are facing an issue with Centos 6.8 template in LXC. Showing you the error

[root@server ~]# ping google.com
ping: icmp open socket: Operation not permitted
[root@server ~]# ls -l $( which ping );
-rwsr-xr-x 1 100000 100000 38264 May 10  2016 /bin/ping

everything looks ok but you still can't ping. btw, I'm on unprivileged container. Firing the below works for me.

[root@server ~]# setcap cap_net_raw+ep /bin/ping

and you should be able to ping after that.

Manual Restore Bacula Without Database

OK, another problem i have. I though my data was gone for good although i do remember my Bacula was doing all the backup! And i finally found a way to get those 1TB files back! Well, as much as you don't know anything about Bacula, you do know where those files are stored right? These files are called 'Volume'. And we will be using these volumes to restore our backup! We will be using bacula volume utility tools to assist us in extracting these precious data!

What's in the Bacula Volume?

Before you can do anything at all, the first thing you need to do is to scan your volume to see whether your stuff is located in the volume!

bls -j -V volume-0177 devicenamehere

and the above will show you something like the one below,


Begin Job Session Record: File:blk=0:8814 SessId=161 SessTime=1480534092 JobId=481
   Job=job.name.com.2017-01-20_01.00.00_33 Date=25-Jan-2017 21:26:12 Level=I Type=B
End Job Session Record: File:blk=0:8814 SessId=161 SessTime=1480534092 JobId=481
   Date=25-Jan-2017 22:53:20 Level=I Type=B Files=2 Bytes=942 Errors=0 Status=T

And what's important on the above are SessId and SessTime. So that we can create a Bootstrap file! Create a file call bootstrap.bsr as show below,

Volume = volume-0177
VolSessionId = 161
VolSessionTime = 1480534092

Now, with this information, we will be able to extract the information out of Bacula Volume!

Extracting Bacula Volume?

In order to extract from Bacula volume, there are a few ways to do it. You can either use your bootstrap file as created above and fire the below command

bextract -p -b ./bootstrap.bsr devicename /home

or you can specific which volume you want to extract without using a bootstrap file as show below,

bextract -p -V volume-0177 devicename/home

and file will starts extracting to /home directory where volume-0177 is the file name and devicename is the actual device name you found on /etc/bacula/bacula-sd file that you wish to restore.

The following shows you some options you can add to your command,

Usage: bextract [-d debug_level] <device-name> <directory-to-store-files>
       -b <file>       specify a bootstrap file
       -dnn            set debug level to nn
       -e <file>       exclude list
       -i <file>       include list
       -p              proceed inspite of I/O errors
       -V              specify Volume names (separated by |)
       -?              print this message

-p is useful if your backup is like 1TB and it throws off an i/o error after 50 hours of extracting. -p basically prevent that.
-i takes in a file path to include only these files or folder to your restoration plan
-e takes in a file path to exclude these files or folder out of your restoration plan
-V specific a volume as shown on my example
-b takes in a file path which is a bootstrap file to tell bextract what you want to do

Now, go save your own ass from getting whoop! Peace out!

Schedule Rsync Backup From Windows to Linux Server

Windows, WHY ARE YOU ALWAYS SO DIFFICULT! Gosh. Damn you are. This time. i wanted to do schedule a backup from my windows server 2012r2 to my linux backup drive. Its as simple as that (while i though it was at least). Google doesn't help with so many rubbish online. Hence, here is a guide that will help us out (me included)

Environment

Enterprise server (Windows 2012 R2)

This is a windows environment server 2012 R2 where our data is

Backup server (Debian Linux)

This is my backup server where i would like to rsync over.

Installation

On Windows server 2012 r2

Download cwRsync
Unzup cwRsync and copy to "C:\cwRsync".
Add "C:\cwRsync\bin" to PATH.
Create the directory "C:\cwRsync\home" and "C:\cwRsync\home\USER" (USER should be the name of the user who will run the Rsync in my case its "admin").
Create public/private keys with the following command:
```
ssh-keygen -t rsa
```
- Paths with "/home/USER/" correspond to the directories that we created in "C:\cwRsync\".
- Leave the password blank.

On Linux

Install openssh-server and rsync.
Provide data to a partition (eg.: /backup/).
Place the public key in /home/USER/.ssh/ and rename the file to authorized_keys. (assuming its root)

On Windows

Test the connection without a password with the following command:

ssh USER@BackupServerIP

Test Rsync:

rsync -v -rlt -z --delete "/myfiles/" "USER@BackupServerIP:/backups/"

where cygdrive is the directory on C:\cygdrive so the above C:\cygdrive\myfiles
To Test Other port

rsync  -e "ssh -p 14000" -arv "--exclude=.svn/" /myfiles USER@BackupServerIP:/backups/

Create a bat file with the rsync command and place it in C:\cwRsync\bin.
Schedule execution every day at 0:30 (half past midnight).

Helpful Resources

http://stackoverflow.com/questions/34147565/rsync-uid-gid-impossible-to-set-cases-cause-future-hard-link-failure-how-to
http://www.smellems.com/tiki-read_article.php?articleId=14