Fix a full disk, low memory or a slow server

A server that runs out of disk space stops writing logs, fails deployments and eventually breaks the database. One that runs out of memory kills processes at random, and one that is overloaded answers slowly. They look alike from the outside (errors, timeouts, a 502), so the first job is to tell them apart.

This article shows how to read the numbers SrvYard collects, how to find what is using the disk, the safe ways to free space, what to do about memory and CPU, and how to be told before it happens again.

Before you begin#

• View access to the server for the numbers, and SSH access to look inside (see Add SSH keys to a server and to your account). See Roles and permissions.
• A recent backup before you delete anything you did not create yourself. Freeing space is usually safe, but a wrong rm is not reversible.
• The time the problem began. The charts go back 30 days, so you can see when a line started to climb.

1. Read the Monitoring tiles#

Open the server and choose Monitoring.

1. CPU. The percentage in use, with the number of cores. A value that stays above 80 to 90 per cent means the server is working as hard as it can.
2. Memory. The percentage in use, the amount and the swap. Swap that is in use means the machine ran out of real memory and started to use the disk as a slow substitute. That is when sites feel slow.
3. Disk. The percentage and the amount used of the total. Treat 85 per cent as the line to act at, and 95 per cent as an emergency.
4. Updates. How many packages can be upgraded. A reboot is required under Uptime means a kernel update is waiting. Neither is the cause of a full disk, but old kernels and caches take space.

Also on the page: Load average (1, 5 and 15 minutes; compare it with the number of cores, a load above the core count means work is queueing), Inbound and Outbound network, and Uptime.

2. Look at the trend#

A number tells you where you are; a chart tells you how you got there.

1. Time range. 1h, 6h, 24h, 7d or 30d. Use 7d or 30d for a disk, 1h or 6h for a slow server right now.
2. Memory. The green line is memory and the orange line is swap. A flat green line near the top with swap rising is a memory shortage.
3. Disk. A straight line is fine. A line that climbs steadily is a log, a cache or a backup that grows. A step up on one day is something that was installed or uploaded on that day.

Read the shape, not only the value:

3. Find what uses the disk#

SrvYard does not delete your files for you, so look for the large directories first. Use an SSH session on the server (the Commands tab of a site runs as the site user, which cannot see system folders):

df -h /
sudo du -xh --max-depth=1 /var /home /srv 2>/dev/null | sort -h | tail -n 15

The first line shows the free space. The second ranks the largest folders. Then go one level deeper into the biggest one. The usual suspects:

• Application logs, such as storage/logs/laravel.log in each site and the Nginx logs in /var/log/nginx.
• Old releases of zero-downtime sites, in each site's releases folder.
• Database data and binary logs under /var/lib/mysql or /var/lib/postgresql.
• Uploads and generated files in storage or public/uploads.
• Package caches in /var/cache and old kernels (apt autoremove removes them).
• Local backups or dumps left in a home folder.

4. Free space safely#

Start with the options that are safe and repeatable:

1. Reduce kept releases. On a zero-downtime site, Settings > Deployments has Deployment retention: how many releases stay on the server, between 1 and 50. The live release and the one before it are never removed, and a lower number takes effect, with older releases removed, at the next deployment. For a big application, five releases may be many.
2. Truncate large logs that you do not need: sudo truncate -s 0 /home/srvyard/example.com/shared/storage/logs/laravel.log. Truncating keeps the file and its permissions, so the application keeps writing to it. Then fix whatever fills the log so it does not return in a day.
3. Clean the package cache: sudo apt clean && sudo apt autoremove.
4. Move heavy files off the server. User uploads belong in object storage, and backups belong on a storage provider rather than on the same disk. See Add an S3-compatible storage provider.
5. Resize the disk at your cloud provider if the data is real and it simply grew. Grow the volume first, then the filesystem.

Check the Disk tile again; it updates within a minute or two.

5. Memory and CPU#

For a memory problem:

• Look at which program is large: ps aux --sort=-%mem | head. PHP-FPM, a queue worker, a database or a build are the usual ones.
• PHP-FPM: too many workers for the memory. Reduce the maximum children of the pool, or give the server more memory.
• Queue workers: use the --memory option so a worker restarts itself before it grows too large. See Fix queued jobs that are not being processed.
• Builds: npm run build can need several GB. A failed deployment with exit code 137 means the build was killed for memory. See Find out why a deployment failed and fix it.
• A small swap file gives a short machine room to breathe, but a server that lives in swap needs more memory.

For a CPU problem, run top or ps aux --sort=-%cpu | head and find the program. Common causes: an unindexed database query, a cron job that overlaps itself, a bot crawling every page, or too little cache. Fix the cause where you can; add cores if the load is legitimate.

6. Get warned before it happens again#

Monitors send an email and an in-app notification when a value stays past a threshold, and again when it recovers. Open Monitoring and scroll to Monitors.

1. New monitor. Opens the form.
2. A monitor row. The metric and its rule (here Used disk space, 85 per cent or more for 5 minutes), the state (OK), the last value and how many people it notifies.

1. Metric. CPU usage, Used memory or Used disk space.
2. Threshold. The percentage. Condition chooses greater than or equal to or less than or equal to.
3. For at least. How many minutes the value must stay past the threshold. A few minutes avoids alerts for short spikes.
4. Create monitor. Saves it. Choose who to notify under Notify members, and add up to five other email addresses.

A disk monitor at 80 to 85 per cent gives you days of warning; a memory monitor at 90 per cent for 10 minutes catches a leak. See Alert thresholds and silencing for more.

Troubleshooting#

• The disk is 100 per cent but du adds up to less. Deleted files that a running process still holds open. Restart the process (PHP-FPM, a worker, the database) and the space returns.
• Inodes, not bytes. df -i shows the inode use. Millions of tiny files (cache, sessions) can fill the inodes while space looks free. Clean that folder.
• The server is slow but CPU and memory look normal. Look at the disk: heavy writes and a database on a slow disk are common. Also check network and external services your application waits for.
• The Monitoring page says Stale. SrvYard cannot reach the server. See Fix a server that will not connect.
• I got an email for a recovered monitor I did not fix. The value dropped on its own, for example after a log rotation. Keep the monitor; the cause may come back.
• Everything is fine now but sites were down. Check the application log and the Nginx error log for the time of the problem. See Fix a 502 Bad Gateway on your site.

Related#

• Server metrics and charts: the full tour of the Monitoring tab.
• Alert thresholds and silencing: sensible thresholds and how to mute them.
• Zero-downtime releases and rollbacks: why releases take disk space.
• Install system packages and apply updates: keep the server patched.