ping: icmp open socket: Operation not permitted Centos 6 LXC

If you are facing an issue with Centos 6.8 template in LXC. Showing you the error

[root@server ~]# ping google.com
ping: icmp open socket: Operation not permitted
[root@server ~]# ls -l $( which ping );
-rwsr-xr-x 1 100000 100000 38264 May 10  2016 /bin/ping

everything looks ok but you still can't ping. btw, I'm on unprivileged container. Firing the below works for me.

[root@server ~]# setcap cap_net_raw+ep /bin/ping

and you should be able to ping after that.

Backlog limit exceeded error and freeze in CentOS 6

"Backlog limit exceeded error", basically what happen is that your OS audit folder is getting flooded with audit events and is unable to write to /var/log/audit directory as the write are too damn fast. It cause the whole system to freeze and you won't be able to login either. Here are a few solutions,

Disable Audit Log

The easiest way is to disable audit log altogether. This will prevent the problem altogether but leaving you with empty audit log to figure out what is going on in your system

/etc/init.d/auditd stop
chkconfig auditd off

Usually we want the log hence this might not work for everyone.

Increase the buffer size

By increasing the buffer size, we can prevent the system from crashing

 auditctl -b 8192

of you can head over to /etc/audit/audit.rules and change the value permanently

# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 8192

# Feel free to add below this line. See auditctl man page

or change the priority at /etc/audit/auditd.conf

#
# This file controls the configuration of the audit daemon
#

log_file = /var/log/audit/audit.log
log_format = RAW
log_group = root
priority_boost = 6

But then, there are times when it grows so big, you want to look into the source of the problem. Hence, might not work for everyone too.

Debugging it

Now, the last solution i have is to look for the issue.

aureport --start today --event --summary -i

which will show us what happen today that crashed my server

-bash-4.1# aureport --start today --event --summary -i

Event Summary Report
======================
total  type
======================
9474  AVC
1196  USER_ACCT
1196  CRED_ACQ
1195  USER_START
1194  LOGIN
1194  CRED_DISP
1194  USER_END
83  NETFILTER_CFG
5  CRYPTO_KEY_USER
3  USER_AUTH
2  CRYPTO_SESSION
1  USER_LOGIN
1  DAEMON_START
1  CONFIG_CHANGE

Now notice this AVC. It has a total of 9474! Gosh, what the heck is that.

-bash-4.1# aureport --start today

Summary Report
======================
Range of time in logs: 11/07/2016 00:00:01.323 - 11/07/2016 10:49:26.950
Selected time for report: 11/07/2016 00:00:00 - 11/07/2016 10:49:26.950
Number of changes in configuration: 86
Number of changes to accounts, groups, or roles: 0
Number of logins: 1
Number of failed logins: 0
Number of authentications: 3
Number of failed authentications: 0
Number of users: 5
Number of terminals: 7
Number of host names: 2
Number of executables: 27
Number of commands: 26
Number of files: 1705
Number of AVC's: 9746
Number of MAC events: 0
Number of failed syscalls: 1263
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 7
Number of integrity events: 0
Number of virt events: 0
Number of keys: 0
Number of process IDs: 4475
Number of events: 17107

Google a bit gives me this

For SELinux there are two main types of audit event:

AVC Audit Events - These are generated by the AVC subsystem as a result of access denials, or where specific events have requested an audit message (i.e. where an auditallow rule has been used in the policy).

SELinux-aware Application Events - These are generated by the SELinux kernel services and SELinux-aware applications for events such as system errors, initialisation, policy load, changing boolean states, setting of enforcing / permissive mode, relabeling etc.

Ah! My SELinux is enable! More Info on AVC (https://selinuxproject.org/page/NB_AL) Looking into /etc/selinux/config


# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Now disabling it won't flood my audit log! Since i don't need SELINUX on my server.

How to move all cPanel accounts to new server via command line

This is a short how-to tutorial to migrate or transfer all cPanel accounts from my old 1.5TB server to another new SSD server through command line.

Backup all cPanel accounts

i am assuming, you know what you want, so we have to first backup all the cPanel accounts in our old system using the following command

ls /var/cpanel/users | while read a; do
/scripts/pkgacct $a
done

remember to screen first before doing the above, as this might take a while if there is a lot of accounts in your machine.

Transfer all cPanel accounts

now, we need to transfer all the cPanel accounts from our old server to the new ones. Notice the pkgacct script generate all the cpmove file on /home directory, use the command below,

bash-4.1# rsync -av --progress /home/*.tar.gz [email protected]:/home

where 192.168.0.2 is your new server. Now, all the files are transferring to our new server!

Restore all cPanel accounts

Finally, in our new server, fire the following command,

ls /home/ | awk -F'[-.]' '{print $2}' | while read a; do
/scripts/killacct --user=$a
/scripts/restorepkg $a
done

similarly, remmeber to screen first before doing the above.

Change new server ip address

You might want to change the ip address of your new server to the old ones. Do the following,

To change the server's main IP address, perform the following steps:
Open the /etc/sysconfig/network-scripts/ifcfg-eth0 file with a text editor.
Edit the IPADDR and GATEWAY lines to use the IP address and gateway of your old server.
Open the /etc/ips file with a text editor.
Add your old server's primary IP address, net mask, and gateway to the file.
Note:
Remove the new server’s primary IP address from this file if it is present.
Restart the network service with the following commands:
For CentOS, CloudLinux™, and Red Hat® Enterprise Linux (RHEL) 6 and earlier, and Amazon Linux, run the service network restart command.
Note:
Amazon Linux always runs in a NAT configuration. 
 
For CentOS, CloudLinux, and RHEL 7 and later, run the systemctl restart network command.
Run the /scripts/mainipcheck command to add the IP address to the /var/cpanel/mainip file.
Run the /scripts/fixetchosts command to add the IP address and hostname of your server to the /etc/hosts file.

and you should be good to go. Test it out and enjoy your new environment!

Installing Gitlab with centos 6 using Apache server on cPanel DNS ONLY server

Please take note these instruction is installed on top of cPanel DNS only server, therefore all services for both gitlab and cpanel DNS Only will still works side by side.

Instroduction to GitLab

Before get start with the installation process , let me give a short description about gitlab. Gitlab is open source software to collaborate on code. It can be considered as an open source self hosted. Even gitlab is very similar to github adoption will be straightforward, but it also allows us to completely control the environment.So,Who directly benefits from / will use this improvement?(target audiences) and the answer will be -> Developers, Doc team. There are few advantage why Gitlab

  • hosted on our own servers
  • better git viewer
  • integrated dreditor (comment on line level)
  • inline editing of all files
  • linking issues
  • pull requests
  • protected branches
  • private repos possible

Read this for more information regrading Gitlab

For the Installation process we believe you know some of the centos Command
If not familiar Read Some tutorial orRefer this link

The GitLab installation consists of setting up the following components:

  1. Install the base operating system (CentOS 6.5 Minimal) and Packages / Dependencies
  2. Ruby
  3. System Users
  4. GitLab shell(GitLab Shell is an application that allows you to execute git commands and provide ssh access to git repositories. It is not a unix shell nor a replacement for Bash or Zsh.)
  5. Database(Mysql)
  6. Gitlab
  7. Web Server(apache)

For fresh installation of the linux centos refer to the below link..
Refer this

1.Installing the operating system

The process start with the installation of the clean CentOS 6.5 "minimal" and it
can be accomplished by downloading the appropriate installation iso file. Just boot the system of the iso file and install the system.

Important !
Note that during the installation you use the "Configure Network" option (it's a button in the same screen where you specify the hostname) to enable the "Connect automatically" option for the network interface and hand (usually eth0).

If you forget this option the network will NOT start at boot.

Updating and installing basic software and services

Installing EPEL repository

EPEL is a volunteer-based community effort from the Fedora project to create a repository of high-quality add-on packages

Download the GPG key for EPEL repository from fedoraproject and install it on your system:

wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 https://www.fedoraproject.org/static/0608B895.txt

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

Verify that the key got installed successfully:

rpm -qa gpg*
gpg-pubkey-0608b895-4bd22942

install the epel-release-6-8.noarch package,to be enable EPEL repository on your system:

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm


PUIAS Computational repository

The PUIAS Computational repository is a part of PUIAS/Springdale Linux, a custom Red Hat:registered: distribution maintained by Princeton University and the Institute for Advanced Study. We take advantage of the PUIAS Computational repository to obtain a git v1.8.x package since the base CentOS repositories only provide v1.7.1 which is not compatible with GitLab. Although the PUIAS offers an RPM to install the repo, it requires the other PUIAS repos as a dependency, so you'll have to add it manually. Otherwise you can install git from source (instructions below).

Download PUIAS repo:

wget -O /etc/yum.repos.d/PUIAS_6_computational.repo https://gitlab.com/gitlab-org/gitlab-recipes/raw/master/install/centos/PUIAS_6_computational.repo

Next download and install the gpg key:

wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-puias http://springdale.math.ias.edu/data/puias/6/x86_64/os/RPM-GPG-KEY-puias
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-puias

Verify that the key got installed successfully:

rpm -qa gpg*
gpg-pubkey-41a40948-4ce19266

Verify that the EPEL and PUIAS Computational repositories are enabled as shown below:

yum repolist

repo id                 repo name                                                status
PUIAS_6_computational   PUIAS computational Base 6 - x86_64                      2,018
base                    CentOS-6 - Base                                          4,802
epel                    Extra Packages for Enterprise Linux 6 - x86_64           7,879
extras                  CentOS-6 - Extras                                           12
updates                 CentOS-6 - Updates                                         814
repolist: 15,525

If you can't see them listed, use the folowing command (from yum-utils package) to enable them:

yum-config-manager --enable epel --enable PUIAS_6_computational

Now install tools that required for gitlab

yum -y update
yum -y groupinstall 'Development Tools'
yum -y install readline readline-devel ncurses-devel gdbm-devel glibc-devel tcl-devel openssl-devel curl-devel expat-devel db4-devel byacc sqlite-devel libyaml libyaml-devel libffi libffi-devel libxml2 libxml2-devel libxslt libxslt-devel libicu libicu-devel system-config-firewall-tui redis sudo wget crontabs logwatch logrotate perl-Time-HiRes


RHEL Notes->If some packages (eg. gdbm-devel, libffi-devel and libicu-devel) are NOT installed, add the rhel6 optional packages repo to your server to get those packages:

yum-config-manager --enable rhel-6-server-optional-rpms

Note: During this installation some files will need to be edited manually. If you are familiar with vim set it as default editor with the commands below. If you are not familiar with vim please skip this and keep using the default editor.
optional

# Install vim and set as default editor

yum -y install vim-enhanced
update-alternatives --set editor /usr/bin/vim.basic

# For reStructuredText markup language support, install required package:
yum -y install python-docutils

You have to config redis start on boot :

chkconfig redis on
service redis start

Next, need to install Mail server

# The recommended one is postfix
yum -y install postfix

if you are using exim as mail agent, try to add this command to exim configuration at /etc/exim.conf

extract_addresses_remove_arguments = false

which will allow exim to sent email.

Install Git from Source

#  ---> Install the pre-requisite files for Git compilation:
yum install zlib-devel perl-CPAN gettext curl-devel expat-devel gettext-devel openssl-devel

## ---> Download and extract it:
mkdir /tmp/git && cd /tmp/git
curl --progress https://www.kernel.org/pub/software/scm/git/git-2.0.0.tar.gz | tar xz
cd git-2.0.0/
./configure
make
make prefix=/usr/local install

#Make sure Git is in your $PATH:
which git

Reboot your system to make sure the installation take effect.Note: When editing config/gitlab.yml, change the git bin_path to /usr/local/bin/git

2.Install Ruby

Remove completely the old Ruby 1.8 package if present. GitLab only supports the Ruby 2.0+ release series:

#remove ruby
yum remove ruby

#Remove any other Ruby build if it is <b>still present</b>:
which ruby
cd <your-ruby-source-path>
make uninstall

Time to install ruby and Gem


mkdir /tmp/ruby && cd /tmp/ruby
curl --progress ftp://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz | tar xz
cd ruby-2.1.2
./configure --disable-install-rdoc
make
make prefix=/usr/local install

#Install the Bundler Gem
gem install bundler --no-doc

After the installation is complete is recommended to reboot the system to make sure the $path/installation take effect

After reboot :

which ruby #-># /usr/local/bin/ruby
ruby -v # -> ruby 2.0.0p481 (2014-02-24 revision 45167) [x86_64-linux]
#make sure the ruby version is 2+ ..the gitlab only support ruby version above 2.0+

3.System Users

Create a git user for Gitlab:

adduser --system --shell /bin/bash --comment 'GitLab' --create-home --home-dir /home/git/ git

Important : Important: In order to include /usr/local/bin to git user's PATH, one way is to edit the sudoers file. As root run:

# type an enter :
visudo

#search for the line and and append <b>/usr/local/bin</b> like so:
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

#like this ----->
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

save and exit

4.Database

Install mysql and enable the mysqld service to start on boot:
Make sure you install Mysql version 5.5.14 (optional)

mysql --version
#install the mysql
yum install -y mysql-server mysql-devel
#make sure mysql service start on boot
chkconfig mysqld on
#start it
service mysqld start

#secure you mysql installation
mysql_secure_installation

Login to MySQL (type the database root password) , like below

mysql -u root -p

#Create a user for GitLab or any name you like(REMEMBER THE NAME) (change $YOUR_PASSWORD_HERE in the command below to a real password you #pick):
CREATE USER 'git'@'localhost' IDENTIFIED BY '$YOUR_PASSWORD_HERE';

Ensure you can use the InnoDB engine which is necessary to support long indexes..

# If this fails, check your MySQL config files (e.g. /etc/mysql/*.cnf, /etc/mysql/conf.d/*) #for the setting "innodb = off"
SET storage_engine=INNODB;

Create the GitLab production database:

CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;

Grant the GitLab user necessary permissions on the table:


# VERY IMPORTANT :Make sure the db name and the user name is correct !!
GRANT SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON `gitlabhq_production`.* TO 'git'@'localhost';

#Quit the database session:
\q

Now test the connection with the newly created database with new user

sudo -u git -H mysql -u git -p -D gitlabhq_production
#if all ok
#exit
\q

5.Gitlab

# We'll install GitLab into home directory of the user "git"
#git user u was created before this in(step 6)
#if u missed means go back and done the thing before you continue here
cd /home/git

Clone the Source

# Clone GitLab repository
sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-1-stable gitlab
#Note: You can change 7-1-stable to master if you want the bleeding edge version, but do so with caution!

!Important
Now time to configure Gitlab(Important steps & make sure u done it correctly)


cd /home/git/gitlab

# Copy the example GitLab config####
sudo -u git -H cp config/gitlab.yml.example config/gitlab.yml

#<strong>Important</strong>
# Make sure to change "localhost" to the fully-qualified domain name of your
# host serving GitLab where necessary
#
# If you want to use https make sure that you set `https` to `true`. See #using-https for all necessary details.
#
# If you installed Git from source, change the git bin_path to /usr/local/bin/git
sudo -u git -H editor config/gitlab.yml

# Make sure GitLab can write to the log/ and tmp/ directories
chown -R git {log,tmp}
chmod -R u+rwX  {log,tmp}

# Create directory for satellites
sudo -u git -H mkdir /home/git/gitlab-satellites
chmod u+rwx,g+rx,o-rwx /home/git/gitlab-satellites

# Make sure GitLab can write to the tmp/pids/ and tmp/sockets/ directories
chmod -R u+rwX  tmp/{pids,sockets}

# Make sure GitLab can write to the public/uploads/ directory
chmod -R u+rwX  public/uploads

# Copy the example Unicorn config
sudo -u git -H cp config/unicorn.rb.example config/unicorn.rb

# Enable cluster mode if you expect to have a high load instance
# Ex. change amount of workers to 3 for 2GB RAM server
sudo -u git -H editor config/unicorn.rb

# Copy the example Rack attack config
sudo -u git -H cp config/initializers/rack_attack.rb.example config/initializers/rack_attack.rb

# Configure Git global settings for git user, useful when editing via web
# Edit user.email according to what is set in config/gitlab.yml
sudo -u git -H git config --global user.name "GitLab"
sudo -u git -H git config --global user.email "gitlab@localhost"
sudo -u git -H git config --global core.autocrlf input

Important Note: Make sure to edit both gitlab.yml and unicorn.rb to match your setup.

Configure GitLab DB settings

# MySQL only:
sudo -u git cp config/database.yml.mysql config/database.yml

# MySQL remote  only:
# Update username/password in config/database.yml.
# You only need to adapt the production settings (first part).
# If you followed the database guide then please do as follows:
# Change 'secure password' with the value you have given to $password
# You can keep the double quotes around the password
sudo -u git -H editor config/database.yml

#  MySQL:
# Make config/database.yml readable to git only
sudo -u git -H chmod o-rwx config/database.yml

Install Gems
Note: As of bundler 1.5.2, you can invoke bundle install -jN (where N the number of your processor cores) and enjoy the parallel gems installation with measurable difference in completion time (~60% faster). Check the number of your cores with nproc. For more information check this post. First make sure you have bundler >= 1.5.2 (run bundle -v).

The installation Process


cd /home/git/gitlab

# For MySQL (note, the option says "without ... postgres")
sudo -u git -H bundle install --deployment --without development test postgres aws

Install GitLab shell

GitLab Shell is an ssh access and repository management software developed specially for GitLab.

# Go to the Gitlab installation folder:
cd /home/git/gitlab

# Run the installation task for gitlab-shell (replace `REDIS_URL` if needed):
sudo -u git -H bundle exec rake gitlab:shell:install[v1.9.6] REDIS_URL=redis://localhost:6379 RAILS_ENV=production

# By default, the gitlab-shell config is generated from your main gitlab config.
#
# Note: When using GitLab with HTTPS please change the following:
# - Provide paths to the certificates under `ca_file` and `ca_path options.
# - The `gitlab_url` option must point to the https endpoint of GitLab.
# - In case you are using self signed certificate set `self_signed_cert` to `true`.
# See #using-https for all necessary details.
#
# You can review (and modify) it as follows:
sudo -u git -H editor /home/git/gitlab-shell/config.yml

# Ensure the correct SELinux contexts are set
# Read http://wiki.centos.org/HowTos/Network/SecuringSSH
restorecon -Rv /home/git/.ssh

Initialize Database and Activate Advanced Features

sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production
#Type yes to create the database. When done you see Administrator account created

Install Init Script

Download the init script (will be /etc/init.d/gitlab):

#download the file
wget -O /etc/init.d/gitlab https://gitlab.com/gitlab-org/gitlab-recipes/raw/master/init/sysvinit/centos/gitlab-unicorn
#give execute permission
chmod +x /etc/init.d/gitlab

chkconfig --add gitlab

#Make GitLab start on boot:
chkconfig gitlab on

#Set up logrotate
cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab

Check Application Status

#Check if GitLab and its environment are configured correctly:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
#Compile assets
sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production
#Start your GitLab instance
service gitlab start

6. Configure the web server(Apache)

We will configure apache with module mod_proxy which is loaded by default when installing apache and mod_ssl which will provide ssl support:

yum -y install httpd mod_ssl
chkconfig httpd on
wget -O /etc/httpd/conf.d/gitlab.conf https://gitlab.com/gitlab-org/gitlab-recipes/raw/master/web-server/apache/gitlab-ssl.conf
mv /etc/httpd/conf.d/ssl.conf{,.bak}
mkdir /var/log/httpd/logs/

Open /etc/httpd/conf.d/gitlab.conf with your editor and replace git.example.org with your FQDN. Also make sure the path to your certificates is valid.

Add LoadModule ssl_module /etc/httpd/modules/mod_ssl.so in /etc/httpd/conf/httpd.conf.

In /etc/httpd/conf/httpd.conf   Change the DocumentRoot to /home/git/gitlab/public/

AND

in Directory --> /home/git/gitlab/public

 

Finally, start apache:

service httpd start

Please take note that, the above configuration is a http setup rather than a ssl setup. the configuration file for ssl is a little buggy but can be found on https://github.com/gitlabhq/gitlab-recipes/tree/master/web-server/apache

Done!

Double-check Application Status
To make sure you didn't miss anything run a more thorough check with:

cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production

Initial Login

Visit YOUR_SERVER:8080 in your web browser for your first GitLab login. The setup has created an admin account for you. You can use it to log in:

root
5iveL!fe

Important Note: Please go over to your profile page and immediately change the password, so nobody can access your GitLab by using this login information later on.

Potential Errors
In case the when start the httpd(apache service),it may throw exception (SSL ERROR) or anything regarding Self-Signed Certificates(ssl) - > it because you may miss the or not generate the Self-Signed Certificates. Here the solution for that

#make sure you installed the mod_ssl 

yum install mod_ssl

#Next, we need to create a new directory where we will store the server key and certificate

mkdir /etc/httpd/ssl

#When we request a new certificate, we can specify how long the certificate should remain #valid by changing the 365 to the number of days we prefer. As it stands this certificate #will expire after one year.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/httpd/ssl/apache.key -out /etc/httpd/ssl/apache.crt

With this command, we will be both creating the self-signed SSL certificate and the server key that protects it, and placing both of them into the new directory.

This command will prompt terminal to display a lists of fields that need to be filled in.

The most important line is "Common Name". Enter your official domain name here or, if you don't have one yet, your site's IP address.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:NYC
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Awesome Inc
Organizational Unit Name (eg, section) []:Dept of Merriment
Common Name (e.g. server FQDN or YOUR name) []:example.com
Email Address []:[email protected]

Now open and edit the file /etc/httpd/conf.d/gitlab.conf

Under section

Uncomment the DocumentRoot and ServerName line and replace example.com with your DNS approved domain name or server IP address (it should be the same as the common name on the certificate):

ServerName example.com:443

Find the following three lines, and make sure that they match the extensions below:

SSLEngine on
SSLCertificateFile /etc/httpd/ssl/apache.crt
SSLCertificateKeyFile /etc/httpd/ssl/apache.key

Now just Restart Apache --> service httpd restart and run

Very Important**CHECK LIST**
      in /home/git/gitlab/config/unicorn.rb change the listen to "YOUR_SERVER_IP_ADDR:8080"
      change to ur ipaddr and server name in /etc/httpd/conf.d/gitlab.conf

Varnish 403 Error – error on page

You have installed varnish into your web server and being happy with it for a few days and suddenly one day you found out that your page is showing an error page with the text "error" on it. And that kept you wonder why is varnish showing error page and after restarting your varnish it is still showing error.

A quick investigation on the http header sent over from the server shows that varnish is serving 403 error page to our visitors! Why?!

Why Varnish serving 403 error page

There are plenty of reasons why varnish is serving a 403 error page but most likely that your backend is only returning 403 once, but then Varnish is caching it for future requests. Some times it happens so we have to explicitly ask Varnish 3.0 to not cache any error pages. On vcl_fetch section, you might want to add the following to prevent it from caching all error page on varnish 3.0.


if (beresp.status >= 400) {
return (hit_for_pass);
}

But if this is not the case, there is also possbility that 403 points at your backend telling varnish it is forbidden. If varnish is giving you that error then it is working and the backend is giving it 403. Most likely this is caused by the backend apps use some sort of rate limiting per ip as by default when you add varnish to an existing setup the ip that gets passed to the backend is the varnish ip not the source ip. In that case you should update the X forwarded by adding the below code to vlc_recv section,

  remove req.http.X-Forwarded-For;
  set req.http.X-Forwarded-For = client.ip;

The above code ensure the correct client ip is being passed into varnish and prevent itself from throwing 403 error!