JoinGG

Surviving a traffic flood as a small server

Four different things run out and they need different answers. Why a bigger box rarely helps, and the firewall rule that outlives the attack.

SweetMask · 3 min read

A small server does not get attacked by a sophisticated adversary. It gets hit by a bored teenager with a booter, a misconfigured crawler, or a bot sweeping every address on the internet, and all three produce the same symptom, which is a server that stops answering while its logs show nothing wrong.

The response people reach for is a bigger box. That almost never helps, because the resource that ran out was rarely CPU.

Work out what actually ran out

Four different things fail under load and they need different answers:

Bandwidth. Your uplink is saturated. Nothing on the box can fix it; only the provider or something upstream can.

Connection tracking. The kernel's table of tracked connections filled, and it started dropping everything including legitimate traffic. The server is idle and unreachable at the same time — a distinctive and confusing combination.

Application threads. Every worker is occupied by a slow or stalled request, so new ones queue. Common when something is hammering an expensive endpoint.

CPU. The one people assume and the least common in a flood.

The diagnosis is quick: high bandwidth with low CPU is the first; low both with dropped packets is the second; normal traffic volume with everything slow is the third.

The cheapest real defence is not being expensive

Most floods are only effective because each request costs the server something substantial. A request that is answered from a cache in 35 milliseconds is a request an attacker has to send a thousand of to hurt you; one that triggers a two-second render is a request they need five of.

So caching is a security control as much as a performance one. This is the same reasoning as what the hardware is actually doing under normal load, applied to an abnormal one — the cost per request is the whole game.

Rate limit by identity, not by address

Blocking IP addresses works against one attacker and not against a botnet, and it reliably catches innocent people behind shared connections.

Where you can identify the client (a declared crawler, an API key, a logged-in session) limit on that instead. It survives address rotation and it lets you be generous with things you recognise and strict with things you do not.

And when you do limit, answer rather than drop. A rejected request that returns a status code tells a well-behaved client to slow down; a dropped packet tells it nothing and it retries, which is worse for both of you.

What to do while it is happening

In order:

  1. Confirm which resource ran out, using the four-way split above. Everything else depends on this.
  2. Stop the bleeding at the cheapest layer that can. If it is one endpoint, disable that endpoint. If it is one user agent, limit that user agent.
  3. Do not restart repeatedly. A service restarting into the same flood generates an unreadable log and contends for the disk, which is the same mistake as letting a crash loop run.
  4. Say something. Players who see a dead server assume it closed. Fifteen minutes of honesty prevents a week of rumour, and that is as true here as it is for any other incident.

What to do afterwards

Write down which resource ran out and what fixed it. The next flood will be the same shape, and the note is what turns an hour of panic into ten minutes of applying something you already know.

Then check the boring things: is the game port rate limited, is the query port still reachable, did any rule you added in a hurry stay behind and quietly block something legitimate? That last one is the most common lasting damage from a flood: a server that survived the attack and has been invisible to listings ever since, because of a firewall rule nobody removed.

The honest limit

If your uplink is saturated, nothing on your machine helps. That is a conversation with your provider, and the answer is either upstream filtering or a different provider.

Knowing which side of that line you are on is most of the value here. Everything above the line is yours to fix cheaply; everything below it is somebody else's, and spending a weekend tuning a kernel that was never the constraint is how people lose time to a problem they could not have solved.

Tags
securitynetworkingincidents
Share

SweetMask

Published · 3 min read

All articles

Keep reading