Updating a Server Without Breaking It

The server was fine on Friday. On Saturday the game updated, you restarted, and now it will not start. The console is producing a stack trace about a class that does not exist. Twenty people are in Discord asking what happened, and you are reading a plugin's issue tracker hoping someone else has already found this.

Almost every server outage that is not a hardware failure looks like this, and almost none of them are caused by the game update itself. They are caused by something you installed months ago that has not been touched since, meeting a game that has moved on.

Why it is almost never the game

A game update changes the game. If you were running an unmodified server, updates would be close to uneventful — the developers tested it, and the server binary is one of the things they tested.

What they did not test is your plugin from 2023 that hooks a function they just renamed, your fork of a mod whose upstream fixed this two versions ago, and the config file that has an option in it that was removed last spring.

So the risk in an update is proportional to how much you have added, not to how large the update is. A small patch can break a heavily modded server; a major version can leave a vanilla one untouched. This is worth internalising because it changes what you check: not "what does this patch change" but "what have I installed that touches what this patch changes".

The Source and GoldSrc family are the clearest illustration, a Valve update breaking community servers is a recurring event, and the damage lands almost on servers running plugin stacks rather than on the game.

The habit that prevents most of it

Do not update the live server first.

That is the whole practice. Everything below is detail.

You need somewhere that is not the live server where the same software runs. It does not need to be impressive: a second instance on the same box, on a different port, with a copy of the config and the plugin folder, is enough. It runs for ten minutes while you check it and then you stop it.

The check is not "does it start". A server can start cleanly and be broken in ways that matter. The list worth running through, which takes about five minutes:

• Does it start with no errors in the log, and no warnings you have not seen before?
• Can you connect?
• Do the plugins that matter respond? Pick the three your community would notice within an hour.
• Does RCON still work? This one is skipped constantly and is the thing you will want most if something goes wrong later.
• Does the server still answer status queries? A server that plays fine but stops answering queries vanishes from every list simultaneously and looks, to everyone outside, exactly like a server that is down. How queries work explains why this can break independently of the game.

That last one is worth emphasising because it is invisible from inside. You are connected and happy; your listing says offline and your arrivals stop.

Back up before, not after

A backup taken after something breaks is a backup of the broken state.

The rule that matters is that the backup must be taken before the update and verified before the update. An unverified backup is a hypothesis. Game server backups covers doing this properly; the update-specific version is short:

• Take it immediately before you touch anything.
• Confirm the file exists and is a plausible size. This ten-second check catches the case where your backup job has been silently failing since March.
• Know how long a restore takes. If you do not know, you will find out during the incident, which is the worst possible time to learn that it takes forty minutes.

Order of operations

1. Read what changed. Not the whole changelog, search it for the things you have installed. A networking change matters if you run networking plugins. A rendering change usually does not matter at all on a server.

1. Check your plugins for updates first. Well-maintained plugins are frequently updated before the game reaches everyone, because their authors follow the pre-release builds. Updating plugins before the game is often the difference between an uneventful update and a broken evening.

1. Update the copy and test it. Five minutes, list above.

4. Announce a window. Even fifteen minutes. Players tolerate scheduled downtime enormously better than unexplained downtime, and the announcement costs nothing.

1. Update the live server in the trough. Not at peak. Your stats page has your actual quiet hours, and owners are frequently wrong about them by two or three hours. Doing maintenance during your real peak is one of the most expensive avoidable mistakes available, because downtime during peak costs you the arrivals as well as the regulars.

1. Watch it for ten minutes. Most breakage appears in the first few minutes under real load, not in your test.

The two dangerous shortcuts

"It is a small patch." Size of patch and size of breakage are unrelated. The five-minute test is the same either way, and skipping it is a bet you make repeatedly until you lose one.

"I will update everything at once while I am here." Updating the game, four plugins and a config change simultaneously means that when something breaks you have no idea which of six things did it. One change at a time is slower on the day it works and enormously faster on the day it does not.

Automatic updates

Most hosting panels offer to update the game for you. On a modded server, turn this off.

The reason is not that automatic updates are bad — it is that they choose the timing, and the timing is the one variable you actually control. An automatic update at 3am while you are asleep is an outage nobody notices until morning. The same update at 11am, watched, is a non-event.

Vanilla servers with no plugins are the exception and can reasonably be left alone.

When it breaks anyway

It will, eventually. The sequence that gets you back fastest:

Roll back first, diagnose second. Get the server running on the previous version and let people play. Investigating a broken server while your community waits produces worse decisions and worse outcomes than investigating a working one at your leisure.

Read the log from the top, not the bottom. The last error is usually a consequence. The first one is usually the cause.

Disable rather than delete. Move the plugin folder aside and start with none, then add back in groups. Binary search finds the culprit in a few restarts; guessing does not.

Say something within ten minutes. "Update broke something, rolling back, will update you shortly" costs one sentence and completely changes how the outage is experienced. Silence is what turns a technical problem into a community problem, and it is the part player support exists to systematise.

Configs are where the quiet breakage lives

Binary incompatibility announces itself: the server refuses to start and prints a stack trace. Configuration drift does not, and it is responsible for most of the breakage that goes unnoticed for weeks.

A removed option is silently ignored. You set it years ago, the developers deleted it, and your config still contains a line that now does nothing. The server starts, everything looks fine, and a setting you rely on is simply not applied. Nothing tells you.

A new option gets a default you did not choose. The reverse case. An update adds a setting, your config does not mention it, and the default takes effect. Sometimes that default is exactly what you would have picked. Sometimes it changes how the game plays.

Config format changes are the worst version. A file that moves from one format to another usually gets regenerated, and your customisations go with the old one. The server comes up on defaults across the board, and if the defaults are reasonable it can take days for someone to notice that three settings quietly reverted.

The defence is not clever: keep your configs in version control, or at minimum keep dated copies. After an update, diff the new generated config against yours. Five minutes, and it catches every one of the cases above.

It also gives you something to read when a player says the server "feels different" and you are certain nothing changed. Frequently something did, and it was a default.

Read the config notes in the changelog specifically. Developers usually list removed and added options separately from features, precisely because this category of problem is well known to them. It is the section most owners skip.

The maintenance you should be doing anyway

Know what you have installed. A written list; plugin, version, where it came from, why it is there. Most owners cannot produce this, which means every update is an audit under time pressure.

Remove what you are not using. Every plugin is a future breakage. The one nobody has used since last year is pure risk with no return. This is the cheapest reliability improvement available and it takes an afternoon once.

Prefer maintained things. A plugin whose last release was two years ago will break, and when it does nobody is coming. When choosing between two that do the same job, recent activity beats feature count. This is a large part of why choosing your mod framework matters more than it looks.

Know which of your plugins are server-side only. Server-side versus client-side mods determines who is affected when something changes, a server-side break affects everyone at once, a client-side one affects only players who have not updated, and the two produce very different-looking bug reports.

The cost, honestly

The full procedure adds perhaps twenty minutes to an update. Most of the time those twenty minutes buy nothing, and it is tempting to skip them.

The return is not measured per update. It is measured against the one update a year that would have taken your server down through your peak, cost you the players who could not get in, and put a hole in your uptime history that a stranger reads as unreliability months later.

Twenty minutes, against that. It is not a close trade, and the servers that are still running in three years are almost invariably the ones that made it.