Diagnosing a Server Crash

The server was up. Now it is not. There is no obvious reason, twenty people are asking in Discord, and the temptation is to start it again and hope.

Starting it again is usually the right first move. Doing only that is how the same crash happens every evening for a fortnight.

There is a method, and it is mostly about ruling out the boring causes before investigating the interesting ones, because the boring causes account for the overwhelming majority of crashes and take about two minutes each to eliminate.

First: preserve the evidence

Before you restart, copy the log somewhere else.

cp logs/latest.log ~/crash-$(date +%F-%H%M).log

Many servers overwrite or rotate their log on startup. Restarting first and investigating afterwards is the single most common way people destroy the only record of what happened, and it takes two seconds to avoid.

If the server is running under systemd the journal keeps history anyway, which is one of the quieter arguments for running it as a service.

Then restart it. Your community waiting while you investigate produces worse decisions than investigating a running server at your leisure.

The three boring causes

Rule these out first. Between them they account for most crashes and each takes under a minute.

1. Out of memory

The most common cause by a wide margin, and the one that leaves the least trace in the game's own log: because the process was killed from outside and had no opportunity to say anything.

journalctl -k | grep -i "killed process"
dmesg | grep -i "out of memory"

If the kernel killed your server, you will find it here. free -h shows what is available now.

What causes it: a memory limit set too low, a leak that grows over hours, another process on the same machine growing, or simply more players than the allocation supports.

A specific Java trap: the heap limit is set in the startup flags, not by the machine. A server with 8GB of RAM and -Xmx2G will run out of heap at 2GB and crash with an OutOfMemoryError in its own log while free -h shows plenty available. Conversely, setting -Xmx larger than the physical memory gets the process killed by the kernel instead.

2. Disk full

df -h

Two seconds. A full disk produces failures that look like absolutely anything — a server that will not save, a database refusing writes, a log that stops mid-line.

The usual culprits are logs nobody rotates and backups nobody prunes. Reading game server logs covers rotation; backups covers retention.

3. A plugin

The most common cause that leaves a proper trace.

Search the log for the crash, then find the topmost line naming something you installed. On Java servers a stack trace makes this trivial: the first line mentioning a package that is not the game itself is your culprit.

The strong hint: did you change anything recently? A plugin updated, a game update, a config edit. A server that ran for four months and started crashing on Tuesday changed on Tuesday.

Reading the crash itself

Read from the top of the error block, not the bottom. Crashes cascade, one failure produces twenty consequent failures, each logging its own error. The last one is the twentieth. The first is the cause.

Note the last normal line. What was the server doing immediately before? A player joining, a chunk loading, a world save, a scheduled task. That is a substantial clue and it is usually two lines above the error.

Distinguish "crashed" from "was killed" from "shut down". A server that logged a clean shutdown was told to stop by something; a control panel, a scheduled task, a stray script. That is a completely different investigation from a crash.

When there is nothing in the log

Genuinely common, and it narrows things considerably rather than being a dead end.

A server that vanishes with no error almost always means the process was killed rather than that it failed. Check, in order:

The kernel OOM killer, as above.

The host. Providers reboot machines, suspend accounts over billing, and perform emergency maintenance. Check your provider's status page and your email before assuming it was you.

A control panel or watchdog. Something else may have decided to stop it. Panel action logs will say.

Hardware. Rare, and it looks like this: random crashes with no pattern, no log entries, and sometimes filesystem errors in dmesg. If you are on shared hosting, a noisy neighbour produces something similar.

When it crashes on startup

Different problem, easier to solve, because the failure is reproducible.

Read the first error, not the last. Startup failures cascade dramatically.

Port already in use. Address already in use means an old instance is still running. Find and stop it.

Permission denied. The server cannot read or write its own files, usually because they are owned by a different user, the classic result of uploading as root and running as a service user. Linux basics covers ownership.

A config it cannot parse. Usually your last edit. A stray character, a missing quote, or Windows line endings in a file that does not tolerate them.

A plugin that will not load. Almost always a version mismatch after an update. Updating without breaking things covers avoiding this in the first place.

Bisecting, when the cause is not obvious

When the log does not name a culprit, find it by elimination.

Move all plugins aside and start with none. If it is stable, the cause is a plugin.

Add them back in halves. Half the plugins, restart, observe. Whichever half breaks it, halve again. Twenty plugins resolve to one in about five restarts.

This is slow and it is deterministic, which beats guessing. The reason to keep a written list of what you have installed and why is that it makes this exercise twenty minutes rather than an afternoon.

Crashes that only happen sometimes

The hardest category, and the approach is different.

Look for the pattern before looking for the cause. Always at the same time of day? Suspect a scheduled task, a backup job, or a restart colliding with something. Always at peak? Suspect resources. Always after several hours? Suspect a leak.

Correlate against your own timeline. Your uptime history shows exactly when the gaps were, which is the fastest way to see a pattern you had not noticed.

Turn up logging deliberately, for a bounded period. Debug logging is unreadable in volume, but for a crash that happens once a day it is worth a day of noise to catch one occurrence properly.

Telling your players something

The part that is not technical and matters more than the diagnosis.

A server that goes down silently and comes back an hour later leaves everyone guessing. A server whose owner says "it crashed, looking into it, back shortly" has had an incident rather than an outage, and the difference is in the message.

Say something within ten minutes. It does not need to be an explanation. "Server crashed, restarting, will update" is enough and costs one sentence.

Say when it is back. People stop checking after the second failed attempt, and without a message they do not know to try again.

Say what it was, briefly, afterwards. "A plugin update broke it, rolled back, stable now." This does two things: it tells people it was understood rather than mysterious, and it makes the next outage less alarming.

Do not over-explain or apologise at length. Servers crash. Everyone knows servers crash. A calm short update reads as competence; three paragraphs of apology reads as a server in trouble.

Player support without a helpdesk covers building this into a habit rather than improvising it each time, and it is the single cheapest thing that separates a community that tolerates downtime from one that drifts away after it.

Preventing the next one

Once it is fixed, three things make the next one cheaper.

Monitoring so you find out in minutes rather than in the morning. The gap between those two is most of the damage.

Automatic restart on failure, so a crash at 4am is a ninety-second gap rather than an eight-hour one. Keeping a game server running covers Restart=on-failure — with a delay, so a crash loop does not hide itself.

Backups you have restored from once. Some crashes take data with them, and finding out then how long a restore takes is the worst possible time to learn.

The order, in short

1. Copy the log, then restart the server
2. journalctl -k | grep killed, was it the OOM killer
3. df -h, is the disk full
4. Read the error block from the top; find the first line naming something you installed
5. Ask what changed recently
6. If nothing is obvious, bisect the plugins
7. If it is intermittent, find the pattern before the cause

Most crashes stop at step two or three, which is the whole reason to do them first.