Gitlab 500/502 Error After Upgrade

Here is what happen, i got either a 500 or 502 error after an upgrade using Omnibus method. And got the error message "Gitlab is not responding"

40TPT

The first thing i did was to look for what could have happen. And the first place to see that is on the log file located at

/var/log/gitlab/gitlab-rails/production.log

And the log basically just give me an error as shown below,

Started GET "/users/sign_in" for 175.144.6.68 at 2015-12-12 23:48:54 +0800
Processing by SessionsController#new as HTML
Completed 500 Internal Server Error in 98ms (ActiveRecord: 10.8ms)

NoMethodError (undefined method `push_events=' for #<GitlabCiService:0x0000000463dba8>):
  app/models/project.rb:809:in `builds_enabled='
  app/controllers/application_controller.rb:194:in `add_gon_variables'

But when i do a status check, it gives me this

[root@git gitlab-rails]# gitlab-ctl status
run: gitlab-workhorse: (pid 4934) 1009s; run: log: (pid 4147) 1227s
run: logrotate: (pid 4942) 1008s; run: log: (pid 296) 3434s
run: nginx: (pid 4948) 1008s; run: log: (pid 299) 3434s
run: postgresql: (pid 4957) 1007s; run: log: (pid 301) 3434s
run: redis: (pid 4965) 1007s; run: log: (pid 294) 3434s
run: sidekiq: (pid 4972) 1005s; run: log: (pid 302) 3434s
run: unicorn: (pid 4990) 1004s; run: log: (pid 305) 3434s

I have pretty much no idea what is going on. But after trying out different ways, it seems to boil down to the following,

1. Check what is going on

Firing the following command should give you an idea what is going on with your configure.

sudo gitlab-rake gitlab:check

After that you could try see what is causing it.

2. Forget to turn on postgres before upgrade

Well, because gitlab said to shutdown gitlab before upgrading, hence i did this,

gitlab-ctl stop

which stops everything including postgres. Hence, database migration wasn't possible. Therefore, i fire the following command and see whether that helps

gitlab-rake db:migrate

Now, after this i still got a 502 error but at least i'm not stuck with 500 error!

3. Forget to reconfigure after an upgrade

Well, if its not database migration, then every time you did a migration, remember to do a reconfigure!

gitlab-ctl reconfigure

Once i did this. Wait a while, and puff! The screen is back up!

Screen Shot 2015-12-13 at 12.00.47 AM

I'm just grateful everything is ok! Just remember to back up your VM image before doing all these upgrades!

GitLab email sending issues

Credit doesn't goes to me, google helps. if you have setup your gitlab on your cpanel setup, you will most likely be using apache and sendmail on gitlab. And one of the issue you will be seeing is that your gitlab email won't be sending to your receiver.

Solution

fire the following code

cp /home/git/gitlab/config/initializers/smtp_settings.rb.sample /home/git/gitlab/config/initializers/smtp_settings.rb

the above code copy the sample file to the actual smtp configure file. next we will need to open up the smtp file

vi /home/git/gitlab/config/initializers/smtp_settings.rb

and starts editing

  ActionMailer::Base.smtp_settings = {
    address: "email.server.com",
    port: 456,
    user_name: "smtp",
    password: "123456",
    domain: "gitlab.company.com",
    authentication: :login,
    enable_starttls_auto: true
  }

take note, all you have to change are the following parameters

    address: "email.server.com",
    port: 456,
    user_name: "smtp",
    password: "123456",
    domain: "gitlab.company.com",

and if you are not using https, change enable_starttls_auto to false. now restart your gitlab

root@monitor [/home/git/gitlab/config/initializers]# service gitlab restart
Stopping unicorn:                                          [  OK  ]
Stopping sidekiq:                                          [  OK  ]
Starting unicorn:                                          [  OK  ]
Starting sidekiq:                                          [  OK  ]

now go test your email, should work now!

Installing Gitlab with centos 6 using Apache server on cPanel DNS ONLY server

Please take note these instruction is installed on top of cPanel DNS only server, therefore all services for both gitlab and cpanel DNS Only will still works side by side.

Instroduction to GitLab

Before get start with the installation process , let me give a short description about gitlab. Gitlab is open source software to collaborate on code. It can be considered as an open source self hosted. Even gitlab is very similar to github adoption will be straightforward, but it also allows us to completely control the environment.So,Who directly benefits from / will use this improvement?(target audiences) and the answer will be -> Developers, Doc team. There are few advantage why Gitlab

  • hosted on our own servers
  • better git viewer
  • integrated dreditor (comment on line level)
  • inline editing of all files
  • linking issues
  • pull requests
  • protected branches
  • private repos possible

Read this for more information regrading Gitlab

For the Installation process we believe you know some of the centos Command
If not familiar Read Some tutorial orRefer this link

The GitLab installation consists of setting up the following components:

  1. Install the base operating system (CentOS 6.5 Minimal) and Packages / Dependencies
  2. Ruby
  3. System Users
  4. GitLab shell(GitLab Shell is an application that allows you to execute git commands and provide ssh access to git repositories. It is not a unix shell nor a replacement for Bash or Zsh.)
  5. Database(Mysql)
  6. Gitlab
  7. Web Server(apache)

For fresh installation of the linux centos refer to the below link..
Refer this

1.Installing the operating system

The process start with the installation of the clean CentOS 6.5 "minimal" and it
can be accomplished by downloading the appropriate installation iso file. Just boot the system of the iso file and install the system.

Important !
Note that during the installation you use the "Configure Network" option (it's a button in the same screen where you specify the hostname) to enable the "Connect automatically" option for the network interface and hand (usually eth0).

If you forget this option the network will NOT start at boot.

Updating and installing basic software and services

Installing EPEL repository

EPEL is a volunteer-based community effort from the Fedora project to create a repository of high-quality add-on packages

Download the GPG key for EPEL repository from fedoraproject and install it on your system:

wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 https://www.fedoraproject.org/static/0608B895.txt

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

Verify that the key got installed successfully:

rpm -qa gpg*
gpg-pubkey-0608b895-4bd22942

install the epel-release-6-8.noarch package,to be enable EPEL repository on your system:

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm


PUIAS Computational repository

The PUIAS Computational repository is a part of PUIAS/Springdale Linux, a custom Red Hat:registered: distribution maintained by Princeton University and the Institute for Advanced Study. We take advantage of the PUIAS Computational repository to obtain a git v1.8.x package since the base CentOS repositories only provide v1.7.1 which is not compatible with GitLab. Although the PUIAS offers an RPM to install the repo, it requires the other PUIAS repos as a dependency, so you'll have to add it manually. Otherwise you can install git from source (instructions below).

Download PUIAS repo:

wget -O /etc/yum.repos.d/PUIAS_6_computational.repo https://gitlab.com/gitlab-org/gitlab-recipes/raw/master/install/centos/PUIAS_6_computational.repo

Next download and install the gpg key:

wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-puias http://springdale.math.ias.edu/data/puias/6/x86_64/os/RPM-GPG-KEY-puias
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-puias

Verify that the key got installed successfully:

rpm -qa gpg*
gpg-pubkey-41a40948-4ce19266

Verify that the EPEL and PUIAS Computational repositories are enabled as shown below:

yum repolist

repo id                 repo name                                                status
PUIAS_6_computational   PUIAS computational Base 6 - x86_64                      2,018
base                    CentOS-6 - Base                                          4,802
epel                    Extra Packages for Enterprise Linux 6 - x86_64           7,879
extras                  CentOS-6 - Extras                                           12
updates                 CentOS-6 - Updates                                         814
repolist: 15,525

If you can't see them listed, use the folowing command (from yum-utils package) to enable them:

yum-config-manager --enable epel --enable PUIAS_6_computational

Now install tools that required for gitlab

yum -y update
yum -y groupinstall 'Development Tools'
yum -y install readline readline-devel ncurses-devel gdbm-devel glibc-devel tcl-devel openssl-devel curl-devel expat-devel db4-devel byacc sqlite-devel libyaml libyaml-devel libffi libffi-devel libxml2 libxml2-devel libxslt libxslt-devel libicu libicu-devel system-config-firewall-tui redis sudo wget crontabs logwatch logrotate perl-Time-HiRes


RHEL Notes->If some packages (eg. gdbm-devel, libffi-devel and libicu-devel) are NOT installed, add the rhel6 optional packages repo to your server to get those packages:

yum-config-manager --enable rhel-6-server-optional-rpms

Note: During this installation some files will need to be edited manually. If you are familiar with vim set it as default editor with the commands below. If you are not familiar with vim please skip this and keep using the default editor.
optional

# Install vim and set as default editor

yum -y install vim-enhanced
update-alternatives --set editor /usr/bin/vim.basic

# For reStructuredText markup language support, install required package:
yum -y install python-docutils

You have to config redis start on boot :

chkconfig redis on
service redis start

Next, need to install Mail server

# The recommended one is postfix
yum -y install postfix

if you are using exim as mail agent, try to add this command to exim configuration at /etc/exim.conf

extract_addresses_remove_arguments = false

which will allow exim to sent email.

Install Git from Source

#  ---> Install the pre-requisite files for Git compilation:
yum install zlib-devel perl-CPAN gettext curl-devel expat-devel gettext-devel openssl-devel

## ---> Download and extract it:
mkdir /tmp/git && cd /tmp/git
curl --progress https://www.kernel.org/pub/software/scm/git/git-2.0.0.tar.gz | tar xz
cd git-2.0.0/
./configure
make
make prefix=/usr/local install

#Make sure Git is in your $PATH:
which git

Reboot your system to make sure the installation take effect.Note: When editing config/gitlab.yml, change the git bin_path to /usr/local/bin/git

2.Install Ruby

Remove completely the old Ruby 1.8 package if present. GitLab only supports the Ruby 2.0+ release series:

#remove ruby
yum remove ruby

#Remove any other Ruby build if it is <b>still present</b>:
which ruby
cd <your-ruby-source-path>
make uninstall

Time to install ruby and Gem


mkdir /tmp/ruby && cd /tmp/ruby
curl --progress ftp://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz | tar xz
cd ruby-2.1.2
./configure --disable-install-rdoc
make
make prefix=/usr/local install

#Install the Bundler Gem
gem install bundler --no-doc

After the installation is complete is recommended to reboot the system to make sure the $path/installation take effect

After reboot :

which ruby #-># /usr/local/bin/ruby
ruby -v # -> ruby 2.0.0p481 (2014-02-24 revision 45167) [x86_64-linux]
#make sure the ruby version is 2+ ..the gitlab only support ruby version above 2.0+

3.System Users

Create a git user for Gitlab:

adduser --system --shell /bin/bash --comment 'GitLab' --create-home --home-dir /home/git/ git

Important : Important: In order to include /usr/local/bin to git user's PATH, one way is to edit the sudoers file. As root run:

# type an enter :
visudo

#search for the line and and append <b>/usr/local/bin</b> like so:
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

#like this ----->
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

save and exit

4.Database

Install mysql and enable the mysqld service to start on boot:
Make sure you install Mysql version 5.5.14 (optional)

mysql --version
#install the mysql
yum install -y mysql-server mysql-devel
#make sure mysql service start on boot
chkconfig mysqld on
#start it
service mysqld start

#secure you mysql installation
mysql_secure_installation

Login to MySQL (type the database root password) , like below

mysql -u root -p

#Create a user for GitLab or any name you like(REMEMBER THE NAME) (change $YOUR_PASSWORD_HERE in the command below to a real password you #pick):
CREATE USER 'git'@'localhost' IDENTIFIED BY '$YOUR_PASSWORD_HERE';

Ensure you can use the InnoDB engine which is necessary to support long indexes..

# If this fails, check your MySQL config files (e.g. /etc/mysql/*.cnf, /etc/mysql/conf.d/*) #for the setting "innodb = off"
SET storage_engine=INNODB;

Create the GitLab production database:

CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;

Grant the GitLab user necessary permissions on the table:


# VERY IMPORTANT :Make sure the db name and the user name is correct !!
GRANT SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON `gitlabhq_production`.* TO 'git'@'localhost';

#Quit the database session:
\q

Now test the connection with the newly created database with new user

sudo -u git -H mysql -u git -p -D gitlabhq_production
#if all ok
#exit
\q

5.Gitlab

# We'll install GitLab into home directory of the user "git"
#git user u was created before this in(step 6)
#if u missed means go back and done the thing before you continue here
cd /home/git

Clone the Source

# Clone GitLab repository
sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-1-stable gitlab
#Note: You can change 7-1-stable to master if you want the bleeding edge version, but do so with caution!

!Important
Now time to configure Gitlab(Important steps & make sure u done it correctly)


cd /home/git/gitlab

# Copy the example GitLab config####
sudo -u git -H cp config/gitlab.yml.example config/gitlab.yml

#<strong>Important</strong>
# Make sure to change "localhost" to the fully-qualified domain name of your
# host serving GitLab where necessary
#
# If you want to use https make sure that you set `https` to `true`. See #using-https for all necessary details.
#
# If you installed Git from source, change the git bin_path to /usr/local/bin/git
sudo -u git -H editor config/gitlab.yml

# Make sure GitLab can write to the log/ and tmp/ directories
chown -R git {log,tmp}
chmod -R u+rwX  {log,tmp}

# Create directory for satellites
sudo -u git -H mkdir /home/git/gitlab-satellites
chmod u+rwx,g+rx,o-rwx /home/git/gitlab-satellites

# Make sure GitLab can write to the tmp/pids/ and tmp/sockets/ directories
chmod -R u+rwX  tmp/{pids,sockets}

# Make sure GitLab can write to the public/uploads/ directory
chmod -R u+rwX  public/uploads

# Copy the example Unicorn config
sudo -u git -H cp config/unicorn.rb.example config/unicorn.rb

# Enable cluster mode if you expect to have a high load instance
# Ex. change amount of workers to 3 for 2GB RAM server
sudo -u git -H editor config/unicorn.rb

# Copy the example Rack attack config
sudo -u git -H cp config/initializers/rack_attack.rb.example config/initializers/rack_attack.rb

# Configure Git global settings for git user, useful when editing via web
# Edit user.email according to what is set in config/gitlab.yml
sudo -u git -H git config --global user.name "GitLab"
sudo -u git -H git config --global user.email "gitlab@localhost"
sudo -u git -H git config --global core.autocrlf input

Important Note: Make sure to edit both gitlab.yml and unicorn.rb to match your setup.

Configure GitLab DB settings

# MySQL only:
sudo -u git cp config/database.yml.mysql config/database.yml

# MySQL remote  only:
# Update username/password in config/database.yml.
# You only need to adapt the production settings (first part).
# If you followed the database guide then please do as follows:
# Change 'secure password' with the value you have given to $password
# You can keep the double quotes around the password
sudo -u git -H editor config/database.yml

#  MySQL:
# Make config/database.yml readable to git only
sudo -u git -H chmod o-rwx config/database.yml

Install Gems
Note: As of bundler 1.5.2, you can invoke bundle install -jN (where N the number of your processor cores) and enjoy the parallel gems installation with measurable difference in completion time (~60% faster). Check the number of your cores with nproc. For more information check this post. First make sure you have bundler >= 1.5.2 (run bundle -v).

The installation Process


cd /home/git/gitlab

# For MySQL (note, the option says "without ... postgres")
sudo -u git -H bundle install --deployment --without development test postgres aws

Install GitLab shell

GitLab Shell is an ssh access and repository management software developed specially for GitLab.

# Go to the Gitlab installation folder:
cd /home/git/gitlab

# Run the installation task for gitlab-shell (replace `REDIS_URL` if needed):
sudo -u git -H bundle exec rake gitlab:shell:install[v1.9.6] REDIS_URL=redis://localhost:6379 RAILS_ENV=production

# By default, the gitlab-shell config is generated from your main gitlab config.
#
# Note: When using GitLab with HTTPS please change the following:
# - Provide paths to the certificates under `ca_file` and `ca_path options.
# - The `gitlab_url` option must point to the https endpoint of GitLab.
# - In case you are using self signed certificate set `self_signed_cert` to `true`.
# See #using-https for all necessary details.
#
# You can review (and modify) it as follows:
sudo -u git -H editor /home/git/gitlab-shell/config.yml

# Ensure the correct SELinux contexts are set
# Read http://wiki.centos.org/HowTos/Network/SecuringSSH
restorecon -Rv /home/git/.ssh

Initialize Database and Activate Advanced Features

sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production
#Type yes to create the database. When done you see Administrator account created

Install Init Script

Download the init script (will be /etc/init.d/gitlab):

#download the file
wget -O /etc/init.d/gitlab https://gitlab.com/gitlab-org/gitlab-recipes/raw/master/init/sysvinit/centos/gitlab-unicorn
#give execute permission
chmod +x /etc/init.d/gitlab

chkconfig --add gitlab

#Make GitLab start on boot:
chkconfig gitlab on

#Set up logrotate
cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab

Check Application Status

#Check if GitLab and its environment are configured correctly:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
#Compile assets
sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production
#Start your GitLab instance
service gitlab start

6. Configure the web server(Apache)

We will configure apache with module mod_proxy which is loaded by default when installing apache and mod_ssl which will provide ssl support:

yum -y install httpd mod_ssl
chkconfig httpd on
wget -O /etc/httpd/conf.d/gitlab.conf https://gitlab.com/gitlab-org/gitlab-recipes/raw/master/web-server/apache/gitlab-ssl.conf
mv /etc/httpd/conf.d/ssl.conf{,.bak}
mkdir /var/log/httpd/logs/

Open /etc/httpd/conf.d/gitlab.conf with your editor and replace git.example.org with your FQDN. Also make sure the path to your certificates is valid.

Add LoadModule ssl_module /etc/httpd/modules/mod_ssl.so in /etc/httpd/conf/httpd.conf.

In /etc/httpd/conf/httpd.conf   Change the DocumentRoot to /home/git/gitlab/public/

AND

in Directory --> /home/git/gitlab/public

 

Finally, start apache:

service httpd start

Please take note that, the above configuration is a http setup rather than a ssl setup. the configuration file for ssl is a little buggy but can be found on https://github.com/gitlabhq/gitlab-recipes/tree/master/web-server/apache

Done!

Double-check Application Status
To make sure you didn't miss anything run a more thorough check with:

cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production

Initial Login

Visit YOUR_SERVER:8080 in your web browser for your first GitLab login. The setup has created an admin account for you. You can use it to log in:

root
5iveL!fe

Important Note: Please go over to your profile page and immediately change the password, so nobody can access your GitLab by using this login information later on.

Potential Errors
In case the when start the httpd(apache service),it may throw exception (SSL ERROR) or anything regarding Self-Signed Certificates(ssl) - > it because you may miss the or not generate the Self-Signed Certificates. Here the solution for that

#make sure you installed the mod_ssl 

yum install mod_ssl

#Next, we need to create a new directory where we will store the server key and certificate

mkdir /etc/httpd/ssl

#When we request a new certificate, we can specify how long the certificate should remain #valid by changing the 365 to the number of days we prefer. As it stands this certificate #will expire after one year.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/httpd/ssl/apache.key -out /etc/httpd/ssl/apache.crt

With this command, we will be both creating the self-signed SSL certificate and the server key that protects it, and placing both of them into the new directory.

This command will prompt terminal to display a lists of fields that need to be filled in.

The most important line is "Common Name". Enter your official domain name here or, if you don't have one yet, your site's IP address.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:NYC
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Awesome Inc
Organizational Unit Name (eg, section) []:Dept of Merriment
Common Name (e.g. server FQDN or YOUR name) []:example.com
Email Address []:[email protected]

Now open and edit the file /etc/httpd/conf.d/gitlab.conf

Under section

Uncomment the DocumentRoot and ServerName line and replace example.com with your DNS approved domain name or server IP address (it should be the same as the common name on the certificate):

ServerName example.com:443

Find the following three lines, and make sure that they match the extensions below:

SSLEngine on
SSLCertificateFile /etc/httpd/ssl/apache.crt
SSLCertificateKeyFile /etc/httpd/ssl/apache.key

Now just Restart Apache --> service httpd restart and run

Very Important**CHECK LIST**
      in /home/git/gitlab/config/unicorn.rb change the listen to "YOUR_SERVER_IP_ADDR:8080"
      change to ur ipaddr and server name in /etc/httpd/conf.d/gitlab.conf