Using Out Of Memory Killer on Linux

The Out of Memory Killer (OOM Killer) is a mechanism in the Linux kernel that frees up RAM when it runs out of memory by forcibly killing one of the running processes.

When the system runs out of memory, the Linux kernel calls the OEM killer, which selects one process according to certain rules and kills it. Freed memory is transferred to the kernel, which can provide it to other processes.

If the process is killed by an OOM Killer, the /var/log/messages log will contain the following entry:

$ cat /var/log/messages | grep "Out of"

Out of Memory: Killed process 123 (postgres) score 904 or sacrifice child

The OOM killer kills processes with a SIGKILL signal, preventing them from gracefully terminating (doesn’t allow saving data or calling other processes). This is why the result of an OOM killer often leads to serious consequences.

The frequent occurrence of OOM killer events indicates system instability and requires the administrator to carefully analyze the causes of out-of-memory events.

To terminate a process, Linux calls the out_of_memory function. The out_of_memor function calls the select_bad_process function, which selects a process to terminate. All running processes are evaluated using the badness function. Badness selects a process according to rules that allow you to determine the reputation of each process (oom_score). You can display the reputation of a process by its PID (for example, for process ID 1764):

$ cat /proc/1764/oom_score

728

The higher the reputation of the process, the more likely it is that it will be terminated by OOM Killer. You can completely disable OOM Killer on Linux (not recommended in most cases):

$ sudo cat /proc/sys/vm/panic_on_oom

The default is 0. This means that OOM Killer is enabled.

To disable it:

$ sudo echo 1 > /proc/sys/vm/panic_on_oom

If you don’t want to disable OOM Killer, but want to ensure that a certain critical process is never killed by OOM, you need to increase its reputation.

Print the current reputation of the process:

$ cat /proc/1764/oom_score

You can increase or decrease the reputation of a process by adding a value between -16 and +15 to the oom_adj file. For example:

$ sudo echo -5 > /proc/1764/oom_adj
$ cat /proc/1764/oom_score

If you want to completely disable OOM Killer for a process, you need to set oom_adj to -17.

$ sudo echo -17 > /proc/1764/oom_adj
$ cat /proc/1764/oom_score

However, you need to understand that when you restart the process, its PID will change. And this reputation will be reassigned to another process with the same PID.

You can dynamically get the PID of a process and reduce its reputation:

$ pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done

But it’s much more convenient to set the reputation in the systemd service unit’s service file:

[Service]
OOMScoreAdjust=-500