Log rotation, and the disk that fills at the worst moment

The most common way a long-running game server dies has nothing to do with the game. The disk fills, the world save fails, and the failure arrives at the worst moment because saves happen on shutdown.

It is entirely preventable and almost nobody prevents it, because a server that has run for three weeks without trouble does not look like a server two days from a full disk.

Where the space actually goes

Four things grow without limit on a default install:

• Console logs. A busy server writes tens of megabytes a day. A chatty plugin writes more.
• Crash reports and heap dumps. A Java heap dump is the size of the heap. One out-of-memory event on an 8GB server writes an 8GB file, and the restart that follows can write another.
• Old world backups, if your backup script writes locally and never prunes.
• Demos and replays on Source servers, which record continuously if configured to.

The last two are the ones that fill a disk overnight rather than over months.

logrotate, and the one option that matters

logrotate is already installed on essentially every Linux server and is the standard answer. A config for a game server is a few lines: the path, how often, how many to keep, and whether to compress.

The option people get wrong is copytruncate. A process that holds a log file open keeps writing to the same file handle even after logrotate renames it, so the rotated file keeps growing and the new file stays empty. copytruncate copies the contents and then truncates the original in place, which keeps the handle valid.

Use it when the process cannot be told to reopen its log. Do not use it when it can, because there is a small window during the copy where lines can be lost.

The journal counts too

If you run under systemd, output also goes to the journal, and the journal has its own retention. SystemMaxUse in journald.conf caps it. Unset, it will happily take 10% of the filesystem.

So a server whose logs are rotated properly can still fill a disk through the journal holding a second copy of everything. Check both. This pairs directly with running the server as a systemd unit rather than a screen session, which is otherwise the better arrangement.

Heap dumps are the one to disable

-XX:+HeapDumpOnOutOfMemoryError is genuinely useful when you are chasing a leak and actively reading the dumps. It is a liability the rest of the time: an out-of-memory event on a server nobody is watching writes a file the size of the heap, the server restarts, and it can do it again.

Turn it on when investigating. Turn it off afterwards. If you leave it on, give it a directory on a partition that is not the one your world lives on.

Monitor free space, not just uptime

An uptime check tells you the server answered. It answers right up until the moment it cannot save. Disk space is the metric that predicts the failure rather than reporting it, and it is usually absent from the monitoring people set up first, which is a gap worth closing when you are setting up monitoring at all.

A simple threshold at 80% gives you days of warning. The failure without it gives you none.

What the failure actually looks like

The server appears normal. Players are connected. Then a save fails, and depending on the game you get a silent error in a log nobody is reading, a corrupted region file, or a world that reverts to the last successful save when it restarts.

The players lose whatever they built since then, and they do not lose it gradually — they lose it all at once, on a server that seemed fine. That is the kind of event a community does not fully recover from, and it is why a backup you have actually restored from is the second half of this.

The afternoon this saves

Write a logrotate config. Cap the journal. Turn off heap dumps unless you are reading them. Point backups somewhere with a retention policy. Add a disk-space alert.

None of it is interesting work and all of it is cheaper than the alternative, which is explaining to a server full of people why yesterday did not happen — the same conversation that follows any other preventable outage.