If MariaDB, PHP-FPM or another service unexpectedly stops running on the server and the website returns a database connection error, a common cause is that the system has run out of memory. In such cases, a component of the Linux kernel known as the OOM killer (Out Of Memory killer) selects a process that is using a lot of memory and terminates it to ensure that the system as a whole remains operational. More often than not, this is the database server.

How much free memory is there?

free -h

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.9Gi       150Mi        45Mi       800Mi       650Mi
Swap:             0B          0B          0B

A low value in the free column is not a problem in itself: Linux uses free memory as a cache (buff/cache) and releases it when necessary. The ‘available’ column shows the memory that is actually available. If this consistently falls below a few per cent of total memory, and there is no swap (Swap: 0B), the server is vulnerable to the OOM killer.

Traces of the OOM killer

Whether a process has been terminated by the OOM killer can be determined from the kernel’s messages:

sudo journalctl -k | grep -i -E "out of memory|killed process"
sudo dmesg -T | grep -i -E "out of memory|killed process"

A real-life example from a test machine with 2 GB of RAM and no swap space, where dnf consumed all the memory during a major system update:

Out of memory: Killed process 16943 (dnf) total-vm:1802676kB, anon-rss:1486780kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:3104kB oom_score_adj:0

The name of the terminated program is shown in brackets; anon-rss indicates the amount of memory actually used (in this case, approx. 1.4 GB). dmesg cannot be read by a standard user (‘read kernel buffer failed: Operation not permitted’), so run it using sudo.

For more information on reading logs: Reading log files on Linux.

What uses memory?

ps aux --sort=-%mem | head -n 15

The %MEM and RSS (in kilobytes) columns show how much memory each process is using. You can monitor this interactively using the top or htop commands (press the M key to sort by memory usage); see: Monitoring processes with the ‘top’ and ‘htop’ commands.

The most common causes:

• MariaDB/MySQL: the innodb_buffer_pool_size is set too high in relation to the server’s memory. On smaller VPSs, allocate no more than 40–50 per cent of the total memory to it if the web server is also running on the same machine.
• PHP-FPM: the value of pm.max_children is too high. Each PHP process can use between 50 and 150 MB of memory, so 20 concurrent processes can easily take up several gigabytes.
• A script running out of control, a memory leak or a sudden spike in traffic (bot, attack).

Creating a swap file

Swap is an area on the hard drive where the kernel can write out rarely used memory blocks (see: What is swap in Linux?). It is slower than memory, so it does not compensate for a lack of RAM, but it buys time during sudden spikes in demand and can prevent the OOM killer from shutting down the database.

Check whether swap is already in place:

swapon --show

If the command does not display anything, create, for example, a 2 GB swap file:

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

To ensure this persists after a reboot, add it to the /etc/fstab file:

echo '/swapfile none swap defaults 0 0' | sudo tee -a /etc/fstab

Check:

swapon --show
free -h

NAME      TYPE SIZE USED PRIO
/swapfile file   2G   0B   -2

In the test, following the creation of the 2 GB swap file mentioned above, the same update – which had previously been halted by the OOM killer – ran without error, and the swap file remained active even after the system was restarted.

How much should it be? On a VPS with 1–4 GB of RAM, 1–2 GB of swap space is usually sufficient. Before doing so, use the df -h / command to ensure there is enough free space on the disk.

Swappiness

The vm.swappiness value (0–100) determines how readily the kernel uses swap. The default value depends on the distribution and the configured profile: on a machine installed from an AlmaLinux 9 cloud image, it was 30 during testing (sysctl vm.swappiness). On a server, it is advisable to set this to a low value so that swap is only used when absolutely necessary:

echo 'vm.swappiness = 10' | sudo tee /etc/sysctl.d/99-swappiness.conf
sudo sysctl -p /etc/sysctl.d/99-swappiness.conf

If the swap is constantly in use

If the free -h command shows that the swap is consistently full and the server is running slowly (with a high wa value in the top output), the server is running low on memory. In this case, reduce the memory usage of the services listed above, or upgrade to a larger plan (see: How can I upgrade the resources of my VPS plan?).

On shared hosting, memory is limited per account by CloudLinux; reaching the memory limit there usually results in a 500 or 503 error, see: CloudLinux resource limits.