JoinGG

The disk is usually the thing

Everybody checks CPU and memory. A tick that jumps to four hundred milliseconds with nothing dominating the profiler is contention, not computation.

SweetMask · 3 min read

When a server stutters, everybody looks at CPU and memory. The disk is checked last if at all, and on a meaningful share of servers it is the answer, because disk waits do not look like disk waits from inside the game.

What it looks like from the game

A tick that takes fifty milliseconds and then one that takes four hundred, with nothing in the profiler dominating. Everything slightly slow, nothing obviously wrong.

That shape is contention rather than computation. The server was not doing more work; it was waiting for something, and on a game server the something is nearly always the disk.

What writes, and when

More than people expect, and mostly on a schedule that guarantees collisions:

World saves are periodic and large. On Minecraft this is region files; on most survival titles it is a world directory rewritten in place.

The database is written on every player action if it is not caching, and logs go continuously, with more under load because more is happening.

Backups, which read everything above at once and write a compressed copy of it.

The collision that matters is the last one against the first. A backup job running while the world saves produces exactly the stutter described, and both are scheduled by defaults nobody changed.

Stagger everything, first

Before buying anything, look at what is scheduled and move it apart. Backups at 04:00, world save on its own interval, database dump at 04:30, log rotation at 05:00.

That costs nothing and resolves a surprising share of unexplained stutters, particularly on servers where the lag appeared after somebody set up automated backups, which is a correlation nobody makes, because the backups are working perfectly.

How to see it

The metric is I/O wait: the proportion of time the CPU spent idle because it was waiting for storage. A CPU at 30% with high I/O wait is not a busy server, it is a blocked one, and reading only CPU utilisation hides that completely.

iostat or any monitoring agent will show it. What matters is watching it during a stutter rather than on average — the same reason a profile taken at a quiet moment describes a server that is fine.

The write that is not yours

Two things write to a game server's disk without appearing in any of its own scheduling.

The first is the operating system — package updates, journal flushes, and on some distributions an automatic update that runs at a time you did not choose. The second is whatever the host runs for its own purposes, which on a managed plan can include their snapshot of your machine.

Neither shows up when you audit your own cron entries, which is why a server can be perfectly staggered and still stutter at the same time every Tuesday. Check the system timers as well as your own before concluding the schedule is clean.

The shared-host case

On a rented box the disk is shared, and your neighbours' backups are on their schedule. You cannot stagger against something you cannot see, and the symptom is a stutter that correlates with nothing you do.

That is the honest limit of tuning on shared storage, and at that point the answer is either a plan with dedicated storage or a different provider, which belongs in the same conversation as every other resource you thought you had bought and are actually sharing.

What actually helps, in order

Stagger scheduled work. Free, effective, first.

Stop writing what you do not need. Log volume is the most compressible thing on most servers, and pruning it reduces both the write load and the backup read load.

Move backups off the machine as they are made, rather than writing a local archive and copying it afterwards. That halves the disk traffic of a backup and is better practice for other reasons anyway.

Separate the database from the world, if you have more than one disk.

Then, and only then, buy faster storage. It is the expensive answer to a problem that four free changes usually fix.

The one-line check

Next time somebody says the server feels laggy, look at I/O wait before anything else. It takes ten seconds and it either eliminates the most-missed cause or finds it immediately, which is a better first move than any amount of configuration tuning aimed at a component that was never the constraint.

Tags
performancestoragetroubleshooting
Share

SweetMask

Published · 3 min read

All articles

Keep reading