"Backlog limit exceeded error", basically what happen is that your OS audit folder is getting flooded with audit events and is unable to write to /var/log/audit directory as the write are too damn fast. It cause the whole system to freeze and you won't be able to login either. Here are a few solutions,
Disable Audit Log
The easiest way is to disable audit log altogether. This will prevent the problem altogether but leaving you with empty audit log to figure out what is going on in your system
/etc/init.d/auditd stop
chkconfig auditd off
Usually we want the log hence this might not work for everyone.
Increase the buffer size
By increasing the buffer size, we can prevent the system from crashing
auditctl -b 8192
of you can head over to /etc/audit/audit.rules and change the value permanently
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.
# First rule - delete all
-D
# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 8192
# Feel free to add below this line. See auditctl man page
or change the priority at /etc/audit/auditd.conf
#
# This file controls the configuration of the audit daemon
#
log_file = /var/log/audit/audit.log
log_format = RAW
log_group = root
priority_boost = 6
But then, there are times when it grows so big, you want to look into the source of the problem. Hence, might not work for everyone too.
Debugging it
Now, the last solution i have is to look for the issue.
aureport --start today --event --summary -i
which will show us what happen today that crashed my server
-bash-4.1# aureport --start today --event --summary -i
Event Summary Report
======================
total type
======================
9474 AVC
1196 USER_ACCT
1196 CRED_ACQ
1195 USER_START
1194 LOGIN
1194 CRED_DISP
1194 USER_END
83 NETFILTER_CFG
5 CRYPTO_KEY_USER
3 USER_AUTH
2 CRYPTO_SESSION
1 USER_LOGIN
1 DAEMON_START
1 CONFIG_CHANGE
Now notice this AVC. It has a total of 9474! Gosh, what the heck is that.
-bash-4.1# aureport --start today
Summary Report
======================
Range of time in logs: 11/07/2016 00:00:01.323 - 11/07/2016 10:49:26.950
Selected time for report: 11/07/2016 00:00:00 - 11/07/2016 10:49:26.950
Number of changes in configuration: 86
Number of changes to accounts, groups, or roles: 0
Number of logins: 1
Number of failed logins: 0
Number of authentications: 3
Number of failed authentications: 0
Number of users: 5
Number of terminals: 7
Number of host names: 2
Number of executables: 27
Number of commands: 26
Number of files: 1705
Number of AVC's: 9746
Number of MAC events: 0
Number of failed syscalls: 1263
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 7
Number of integrity events: 0
Number of virt events: 0
Number of keys: 0
Number of process IDs: 4475
Number of events: 17107
Google a bit gives me this
For SELinux there are two main types of audit event:
AVC Audit Events - These are generated by the AVC subsystem as a result of access denials, or where specific events have requested an audit message (i.e. where an auditallow rule has been used in the policy).
SELinux-aware Application Events - These are generated by the SELinux kernel services and SELinux-aware applications for events such as system errors, initialisation, policy load, changing boolean states, setting of enforcing / permissive mode, relabeling etc.
Ah! My SELinux is enable! More Info on AVC (https://selinuxproject.org/page/NB_AL) Looking into /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
Now disabling it won't flood my audit log! Since i don't need SELINUX on my server.