Reading a crash report in thirty seconds

A crash report looks like a wall of text and is mostly a structured document with four useful parts. Knowing which four turns twenty minutes of scrolling into thirty seconds of reading.

Part one: what was thrown

The first line, and it names the category of failure rather than the cause.

OutOfMemoryError means the heap was exhausted: no plugin is at fault in the way you think, and the trace below shows whoever happened to be allocating when the memory ran out. This is the most misread line in any crash report, because the named plugin is usually innocent.

StackOverflowError means recursion, and the trace will show the same few frames repeating. The repeating frames are the answer.

NullPointerException and friends mean ordinary bugs, and the trace is genuinely informative.

Reading this line first decides which of the next three parts is worth your time.

Part two: the first Caused by

Java wraps exceptions. The outer one is frequently a generic wrapper added by the server software, and the real failure is further down.

Read to the last Caused by: in the chain. That is the original event; everything above it is the path by which it was reported.

People stop at the first line, conclude the server software is broken, and file a bug against the wrong project.

Part three: the first frame that is not the game

The trace is a list of method calls, most recent first. Scroll down until you find something that is not the game engine or the Java standard library.

That package name is your culprit: a plugin, a mod, a library. Server software usually names it explicitly as well, which makes this redundant on Paper and Spigot and necessary on almost everything else.

If every frame is the engine and nothing third-party appears, you have found the rarer case: an actual bug in the server software, or a symptom of something environmental like memory or storage.

Part four: the environment block

Version, Java version, memory, plugin list, uptime at crash.

The uptime is the most diagnostic and the least read. A crash five minutes after boot is a configuration or startup problem. A crash after eleven hours, every time, is accumulation, and accumulation is a leak rather than a fault, which is a completely different investigation.

The reports that contain nothing

Two failures leave no report at all, and recognising the absence is the skill.

A kill. The process was terminated rather than asked to stop, so nothing was written: the log simply ends mid-line. The evidence is outside the game: the container's exit status, or the host's kernel log. This is what a memory limit enforced by killing produces.

A hang. The process is alive and doing nothing. No exception, no exit, no new lines. The only signal is the absence of routine output, which is why knowing what a healthy log looks like matters as much as knowing what an error looks like.

Before reporting it to anybody

If you are about to open an issue against a plugin, three things make the difference between a fix and a closed ticket.

The whole report, not a screenshot of the first line. The maintainer needs the chain and the environment block, and the first line is the part that tells them least.

The version of everything. Game, server software, plugin, Java. A trace without versions is unreproducible, and unreproducible reports get closed.

Whether it is repeatable. "Happens every time I open a shop" is worth ten times "it crashed once". If you do not know, say you do not know rather than guessing.

What to do with it

Match the crash against your change log. Almost every crash has a change behind it within 48 hours, and if you have been recording what you change this step is a minute rather than an archaeology exercise.

Then reproduce it if you can, on a copy rather than on the live server. A crash you can trigger is a crash you can fix; one you cannot is one you are guessing about.

The four crashes that are not bugs

A large share of what looks like a crash report is the server being killed rather than failing, and each has a signature worth recognising because none of them are fixed by reading a stack trace.

Out of memory from the kernel leaves nothing useful in the server log, because the process was removed rather than asked to stop. dmesg names it, and the fix is heap sizing or overhead, not code.

Out of memory from the JVM is different: an OutOfMemoryError with a stack trace that points at whatever happened to be allocating. The frame is coincidental; the heap is the problem.

A port already bound means the previous process did not exit. Usually a restart script that did not wait, and the visible symptom is a server that fails immediately after a restart that appeared to work.

A full disk makes writes fail in whatever order they were attempted, producing a report that looks like corruption in whichever subsystem wrote first. Run df before anything else.

Reading a stack trace in the right order

When it is a real exception, the useful information is not at the top.

Start at the bottom for the Caused by chain and read the last one, because that is the original failure and everything above it is wrapping. Then scan upward for the first frame that is not engine or library code: that names the plugin or resource responsible, and it is almost always two-thirds of the way down rather than at the front.

The top line is the most visible and the least useful. It tells you where the failure surfaced, which is frequently a long way from where it started.

The part worth saying out loud

Most crash reports are read in a hurry by somebody who wants the server back up, which is exactly when the first line gets mistaken for the cause.

Thirty seconds of reading the four parts in order (what was thrown, the last Caused by, the first non-engine frame, the uptime) answers the question most of the time. It is a smaller skill than it looks and it is the difference between fixing something and restarting until it stops, which is the habit that quietly turns into a shrinking restart interval nobody has diagnosed.