Rolling out a config change safely

The riskiest thing most server owners do is change six settings at once on a Friday evening, because it felt like a small change and the server was quiet.

There is a version of that which is safe, and it costs about ten extra minutes.

Change one thing

Not as a rule of thumb — as the whole method. A single change has a single possible cause when something breaks, and a single thing to revert.

Six changes have fifteen pairwise interactions and no obvious starting point. The time saved by batching is spent several times over on the evening you have to unpick it, and unpicking happens under pressure with players connected.

The objection is that six small changes are faster than six restarts. That is true right up until one of them is wrong.

Write down what you changed, before you change it

One line, with the date: which setting, from what, to what, and why.

This is the file you will want in three weeks when something is subtly different and nobody remembers touching anything. Without it, every investigation starts with archaeology, which is the same reason a crash without a record of recent changes takes an evening instead of ten minutes.

It is also the thing that makes a change reversible by somebody who is not you.

Know what the change is supposed to do

Before applying it: what number should move, and in which direction?

If you cannot answer that, you are not making a change, you are trying something. Trying things is legitimate during investigation and is not the same activity as tuning, and conflating them is how servers end up with a configuration nobody can justify line by line.

The answer also tells you when to revert. A setting changed to fix a stutter, which did not fix the stutter, should go back, and it usually does not, because it was not obviously harmful and nobody was measuring.

Apply at the quietest hour, not the most convenient one

A change made at peak affects the most people and is hardest to evaluate, because everything is under load and the signal is buried.

Off-peak gives you a clean baseline and the smallest audience for a mistake. It is also when a restart costs least, and most meaningful config changes need one.

The hour is a question your own logs answer rather than one anybody else can tell you, because your population is not necessarily in your timezone — the same point as picking a restart window from data rather than habit.

Give it long enough to show

Most config changes cannot be evaluated in five minutes. Entity activation ranges, cache sizes, tick budgets: they express themselves under load, over hours.

Change, observe across a full peak, then decide. A change judged immediately after the restart is being judged against the cleanest state the server will have all week, which flatters everything.

Tell somebody, if anybody else runs the server

A change made silently by one admin is a change the other three will spend an evening investigating when something behaves differently.

This costs a line in a staff channel and it is the single biggest source of wasted debugging on servers with more than one person touching the configuration. The person who knows the answer is frequently online and does not know a question is being asked, and consistency of that kind is the same thing that makes enforcement legible on the moderation side.

If you keep the change log described above where staff can read it, this happens for free.

Have the revert ready

Know exactly what you would type to put it back, before you need to. On a config file that means a copy; on a database change it means a dump; on a plugin update it means the previous jar, which you kept.

The instinct during an incident is to fix forward, and the correct move is nearly always to revert first and diagnose afterwards: the same order as the first hour of any incident.

What this buys

Not safety in the abstract. Specifically: the ability to answer "what changed" in under a minute, three weeks later, when the person asking is you.

Servers that update confidently are not the ones with better luck. They are the ones where every change was small, recorded, and reversible, and that is the entire difference between running a server and maintaining one, which shows up eventually in whether it is still there in three years.