linux

Manual Restore Bacula Without Database

OK, another problem i have. I though my data was gone for good although i do remember my Bacula was doing all the backup! And i finally found a way to get those 1TB files back! Well, as much as you don't know anything about Bacula, you do know where those files are stored right? These files are called 'Volume'. And we will be using these volumes to restore our backup! We will be using bacula volume utility tools to assist us in extracting these precious data!

What's in the Bacula Volume?

Before you can do anything at all, the first thing you need to do is to scan your volume to see whether your stuff is located in the volume!

bls -j -V volume-0177 devicenamehere

and the above will show you something like the one below,


Begin Job Session Record: File:blk=0:8814 SessId=161 SessTime=1480534092 JobId=481
   Job=job.name.com.2017-01-20_01.00.00_33 Date=25-Jan-2017 21:26:12 Level=I Type=B
End Job Session Record: File:blk=0:8814 SessId=161 SessTime=1480534092 JobId=481
   Date=25-Jan-2017 22:53:20 Level=I Type=B Files=2 Bytes=942 Errors=0 Status=T

And what's important on the above are SessId and SessTime. So that we can create a Bootstrap file! Create a file call bootstrap.bsr as show below,

Volume = volume-0177
VolSessionId = 161
VolSessionTime = 1480534092

Now, with this information, we will be able to extract the information out of Bacula Volume!

Extracting Bacula Volume?

In order to extract from Bacula volume, there are a few ways to do it. You can either use your bootstrap file as created above and fire the below command

bextract -p -b ./bootstrap.bsr devicename /home

or you can specific which volume you want to extract without using a bootstrap file as show below,

bextract -p -V volume-0177 devicename/home

and file will starts extracting to /home directory where volume-0177 is the file name and devicename is the actual device name you found on /etc/bacula/bacula-sd file that you wish to restore.

The following shows you some options you can add to your command,

Usage: bextract [-d debug_level] <device-name> <directory-to-store-files>
       -b <file>       specify a bootstrap file
       -dnn            set debug level to nn
       -e <file>       exclude list
       -i <file>       include list
       -p              proceed inspite of I/O errors
       -V              specify Volume names (separated by |)
       -?              print this message
  • -p is useful if your backup is like 1TB and it throws off an i/o error after 50 hours of extracting. -p basically prevent that.
  • -i takes in a file path to include only these files or folder to your restoration plan
  • -e takes in a file path to exclude these files or folder out of your restoration plan
  • -V specific a volume as shown on my example
  • -b takes in a file path which is a bootstrap file to tell bextract what you want to do

Now, go save your own ass from getting whoop! Peace out!

Converting Webm to MP4 using latest avconv or ffmpeg

Ok, there is no ffmpeg here since its so old and we have all been using the latest avconv. However, there are so many legacy documentation about ffmpeg conversion and webm that leaves me searching up and downs for a solutions for my problems. Although in the end, it wasn't my problem at all but i manage to resolve it somehow so writing it out might helps someone out there too.

Problems converting webm to mp4

if you are facing problems converting webm (whammy) from webrtc of Chrome using older version of ffmpeg or avconv. You are not alone. You may face the following problems

[matroska,webm @ 0x18399cb8] Unknown/unsupported CodecID V_VP9.
[matroska,webm @ 0x18399cb8] Unknown/unsupported CodecID A_OPUS.

which means your avconv doesn't really support the new A_OPUS or V_VP9. And when you try to convert your whammy webm to mp4, it will 99% throws you one of the following errors

Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

or

Decoder (codec id 0) not found for input stream #0:0

or

webm decoding for stream 0 failed

or

23_1460406164.wav: could not find codec parameters

and etc. but it will just get stucked at #0:0 and you will be staring at your screen for a couple of hours (if you are lucky) and wondering what the hell goes wrong when you did nothing to it.

Solution to convert webm to mp4

If you manage to debug your avconv conversion instruction, and saw the word Unknown/unsupported CodecID A_OPUS, you most probably fall into the category of having a older Ubuntu or avconv which is what happen to me. However, if you are using a older version of Ubuntu, you will need to get the latest version from Source, since installing from apt-get will only gives you what your Ubuntu version can support. Once installed has completed, you will get the latest version of avconv 11. And you will need to fire the following command

avconv -i 23_1460420555.webm -c:v libx264  -strict experimental out.mp4

and you will get the following results

root@test:# avconv -i 23_1460420555.webm -c:v libx264  -strict experimental out.mp4
avconv version 11.6, Copyright (c) 2000-2014 the Libav developers
  built on Apr 12 2016 01:42:20 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
[matroska,webm @ 0x2082100] Estimating duration from bitrate, this may be inaccurate
Input #0, matroska,webm, from '23_1460420555.webm':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0.0(eng): Video: vp8, yuv420p, 640x480, PAR 1:1 DAR 4:3, 29.97 fps, 1k tbn (default)
    Stream #0.1(eng): Audio: opus, 48000 Hz, mono, fltp (default)
File 'out.mp4' already exists. Overwrite ? [y/N] y
[libx264 @ 0x2097d80] using SAR=1/1
[libx264 @ 0x2097d80] using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
[libx264 @ 0x2097d80] profile High, level 3.0
[libx264 @ 0x2097d80] 264 - core 120 r2151 a3f4407 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf56.1.0
    Stream #0.0(eng): Video: libx264, yuv420p, 640x480 [PAR 1:1 DAR 4:3], q=-1--1, 29.97 fps, 2997 tbn, 29.97 tbc (default)
    Metadata:
      encoder         : Lavc56.1.0 libx264
    Stream #0.1(eng): Audio: libvo_aacenc, 48000 Hz, mono, s16, 200 kb/s (default)
    Metadata:
      encoder         : Lavc56.1.0 libvo_aacenc
Stream mapping:
  Stream #0:0 -> #0:0 (vp8 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (opus (native) -> aac (libvo_aacenc))
Press ctrl-c to stop encoding
Non-monotonous DTS in output stream 0:1; previous: 10576, current: 8864; changing to 10577. This may result in incorrect timestamps in the output file.
Non-monotonous DTS in output stream 0:1; previous: 10577, current: 9888; changing to 10578. This may result in incorrect timestamps in the output file.
Non-monotonous DTS in output stream 0:1; previous: 38912, current: 37584; changing to 38913. This may result in incorrect timestamps in the output file.
Non-monotonous DTS in output stream 0:1; previous: 38913, current: 38608; changing to 38914. This may result in incorrect timestamps in the output file.
Non-monotonous DTS in output stream 0:1; previous: 74064, current: 72400; changing to 74065. This may result in incorrect timestamps in the output file.
Non-monotonous DTS in output stream 0:1; previous: 74065, current: 73424; changing to 74066. This may result in incorrect timestamps in the output file.
frame=   58 fps=  0 q=-1.0 Lsize=     133kB time=1.84 bitrate= 593.4kbits/s
video:115kB audio:15kB other streams:0kB global headers:0kB muxing overhead: 2.336649%
[libx264 @ 0x2097d80] frame I:1     Avg QP:20.91  size: 26829
[libx264 @ 0x2097d80] frame P:51    Avg QP:22.32  size:  1685
[libx264 @ 0x2097d80] frame B:6     Avg QP:25.44  size:   638
[libx264 @ 0x2097d80] consecutive B-frames: 86.2%  0.0%  0.0% 13.8%
[libx264 @ 0x2097d80] mb I  I16..4: 19.3% 67.0% 13.7%
[libx264 @ 0x2097d80] mb P  I16..4:  0.9%  1.0%  0.1%  P16..4: 22.0%  4.0%  3.6%  0.0%  0.0%    skip:68.5%
[libx264 @ 0x2097d80] mb B  I16..4:  0.1%  0.1%  0.0%  B16..8: 30.9%  0.3%  0.0%  direct: 0.2%  skip:68.4%  L0:42.6% L1:56.6% BI: 0.8%
[libx264 @ 0x2097d80] 8x8 transform intra:58.6% inter:73.5%
[libx264 @ 0x2097d80] coded y,uvDC,uvAC intra: 66.9% 77.4% 53.2% inter: 10.0% 11.2% 1.3%
[libx264 @ 0x2097d80] i16 v,h,dc,p: 21% 11% 44% 24%
[libx264 @ 0x2097d80] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 13% 34%  5%  5%  6%  7%  7%  6%
[libx264 @ 0x2097d80] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 21% 31%  5%  6%  5%  6%  5%  4%
[libx264 @ 0x2097d80] i8c dc,h,v,p: 64% 17% 16%  3%
[libx264 @ 0x2097d80] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x2097d80] ref P L0: 70.3% 20.2%  4.6%  4.9%
[libx264 @ 0x2097d80] ref B L0: 96.0%  2.2%  1.8%
[libx264 @ 0x2097d80] ref B L1: 89.9% 10.1%
[libx264 @ 0x2097d80] kb/s:482.04

And you will get a nice video with good quality right to your face! But do take note that the "Non-monotonous DTS in output" warning. Although its a warning, if you have compile avconv wrongly, your version conversion will not work. And if you are using the wrong encoder, pretty much nothing works for you and that's how pussy avconv can be about its encoding and stuff.

Of course, there are a good links to avconv that you could read up! especially what each avconv parameter means!

Installing avconv from source in Ubuntu

This is pretty interesting as avconv is a pretty sensitive fellow. If you do this wrongly, your avconv is pretty much not workable no matter how you are going to spin this. Hence, i do myself a favour and write this up instead. Now, before i began, let me explain what version of Ubuntu i am using. I am on Ubuntu 12.04.5 LTS and if i do the following from repo

sudo apt-get install ffmpeg libav-tools

I will be getting myself avcon version 8.x.x since i am on a older version of Ubuntu. Hence, the installed of avcon from source.

Removing older avconv or ffmpeg

Before we do anything at all, we need to remove the older version of avcon.

apt-get remove --purge libav-tools

and this will remove the built in avconv that you might have installed

Installing prerequisite files

Now, fire the below command to install everything you may need for your avconv compilation 

apt-get install build-essential
apt-get install yasm
apt-get install libfdk-aac-dev
apt-get install libx264-dev

Once you have done that, we are good to go!

Installing avconv from source

I am going to install avconv version 11 from libav.org. 

wget https://libav.org/releases/libav-11.6.tar.gz  --no-check-certificate
tar zxvf libav-11.6.tar.gz
cd libav-11.6/
./configure --enable-libvo-aacenc --enable-gpl --enable-nonfree --enable-version3 --enable-libx264 --enable-static --disable-shared
make && make install

After you have finished compilation (which takes some time), you are pretty much installed with the newer version of avconv! Not that tough right?