JoinGG

The smoke test that takes four minutes

A server that starts is not a server that works. Six checks, written down once, run the same way every time by whoever is there.

SweetMask · 3 min read

A server that starts is not a server that works. The distance between those two states is where most post-deploy problems live, and it is crossed by four minutes of checking that almost nobody does.

The reason nobody does it is that there is no list. Everyone knows they should "test it", nobody has decided what that means, and so the test becomes joining briefly, seeing the spawn, and declaring victory.

Write the list once

The point is to have it written down, so it is the same every time and so somebody else can run it. Five or six items, specific to your server, covering the things that break.

A workable starting set:

Does the console go quiet? Read the startup block, then watch for thirty seconds. A wall of warnings after boot is the most reliable early signal there is, and it is visible before anything else.

Can a fresh account join? Not your account, which has permissions and cached content. The experience that matters is the one a stranger gets, and yours is not it.

Do the three things players do most still work? For a survival server that might be claiming land, opening a shop and teleporting. For roleplay, spawning a vehicle and opening an inventory. Pick the three your players would notice immediately.

Does the permission system still recognise groups? A staff member should be able to use one staff command. This catches the most common consequence of a plugin update, which is a config quietly rewritten.

Does the server appear externally? Query it from outside and confirm the name, map and player count come back right. This is the check that catches a server that is up and invisible, and it is invisible from the console.

Is anything writing? Confirm a save happened and the backup job still has somewhere to write.

Automate the half that can be

Three of the six checks are a script: query the server externally, confirm the process is up, confirm the newest backup is under twenty-six hours old. None of those needs a person and all of them can run every few minutes rather than once after a deploy.

That leaves the human list at three items, which is short enough that it actually gets done. The split is worth making deliberately — the mechanical checks belong with whatever you are already monitoring, and the human ones belong on a card next to the keyboard.

What cannot be automated is the judgement: does the server feel right, is the console quiet, does a normal action still behave normally. Those need somebody who knows what normal looks like.

Run it from a clean install occasionally

Your own client has the content cached, the server in favourites and your account whitelisted. It cannot tell you what a new arrival experiences.

Once in a while (after any content change) join from a machine or account that has none of that. The download size, the missing textures, the first-join messages: all of it is only visible this way, and all of it is what decides whether a stranger stays past the first ten minutes.

Do it before you announce, not after

The order matters more than it seems. A server announced as back and then found broken costs trust twice: once for the outage and once for the false all-clear.

Four minutes of checking before the announcement converts that into a slightly longer maintenance window, which nobody minds. The same logic applies to saying anything during an incident — an accurate later update beats a confident early one.

What makes a list decay

Items get added after each new failure and never removed, and eventually the list is twenty items and takes half an hour, so it stops being run.

Prune it. If a check has passed every time for a year and the thing it covers has not changed, it is no longer earning its place. The list should be short enough that running it is never a decision.

The honest reason to have one

Not discipline. Memory.

At the end of an update, at whatever hour updates happen, nobody reliably remembers which six things to check, and the one forgotten is the one that breaks. A written list removes the need to remember anything, which is the same reason writing down what changed beats intending to remember it.

Write yours this week, while nothing is broken. That is the only time anybody has the perspective to decide what matters.

Tags
operationstestingmaintenance
Share

SweetMask

Published · 3 min read

All articles

Keep reading