Gitlab 500/502 Error After Upgrade

Here is what happen, i got either a 500 or 502 error after an upgrade using Omnibus method. And got the error message "Gitlab is not responding"

The first thing i did was to look for what could have happen. And the first place to see that is on the log file located at

/var/log/gitlab/gitlab-rails/production.log

And the log basically just give me an error as shown below,

Started GET "/users/sign_in" for 175.144.6.68 at 2015-12-12 23:48:54 +0800
Processing by SessionsController#new as HTML
Completed 500 Internal Server Error in 98ms (ActiveRecord: 10.8ms)

NoMethodError (undefined method `push_events=' for #<GitlabCiService:0x0000000463dba8>):
  app/models/project.rb:809:in `builds_enabled='
  app/controllers/application_controller.rb:194:in `add_gon_variables'

But when i do a status check, it gives me this

[root@git gitlab-rails]# gitlab-ctl status
run: gitlab-workhorse: (pid 4934) 1009s; run: log: (pid 4147) 1227s
run: logrotate: (pid 4942) 1008s; run: log: (pid 296) 3434s
run: nginx: (pid 4948) 1008s; run: log: (pid 299) 3434s
run: postgresql: (pid 4957) 1007s; run: log: (pid 301) 3434s
run: redis: (pid 4965) 1007s; run: log: (pid 294) 3434s
run: sidekiq: (pid 4972) 1005s; run: log: (pid 302) 3434s
run: unicorn: (pid 4990) 1004s; run: log: (pid 305) 3434s

I have pretty much no idea what is going on. But after trying out different ways, it seems to boil down to the following,

1. Check what is going on

Firing the following command should give you an idea what is going on with your configure.

sudo gitlab-rake gitlab:check

After that you could try see what is causing it.

2. Forget to turn on postgres before upgrade

Well, because gitlab said to shutdown gitlab before upgrading, hence i did this,

gitlab-ctl stop

which stops everything including postgres. Hence, database migration wasn't possible. Therefore, i fire the following command and see whether that helps

gitlab-rake db:migrate

Now, after this i still got a 502 error but at least i'm not stuck with 500 error!

3. Forget to reconfigure after an upgrade

Well, if its not database migration, then every time you did a migration, remember to do a reconfigure!

gitlab-ctl reconfigure

Once i did this. Wait a while, and puff! The screen is back up!

I'm just grateful everything is ok! Just remember to back up your VM image before doing all these upgrades!