The firewall that ate fragmented UDP

A server that had been fine for weeks started taking thirteen seconds to answer a query that had always taken a hundred milliseconds. Not intermittently. Every time, and only for one kind of query.

The interesting part is not the fix. It is that nothing, anywhere, recorded a problem.

The symptom

The service was a game server answering the Source query protocol — three separate requests that a server browser makes to build a listing:

Query	Reply size	Behaviour
A2S_INFO	~500 bytes	Instant
A2S_PLAYERS	~900 bytes	Instant
A2S_RULES	~2.4 KB	Thirteen seconds, or never

One of those is not like the others, and it is not the one people reach for first.

The first instinct — mine included — was latency. It looks like latency. A server browser shows one number, that number was enormous, and everything downstream of “the server is slow” is a plausible story: DNS, routing, a struggling host, a noisy neighbour. I spent time on several of those, and all of it was wasted, because the server was not slow. It was answering some things instantly and other things not at all, and the thirteen seconds was a client-side retry timer, not a round trip.

That is the first lesson, and it is the one worth keeping: an average is the enemy of a diagnosis. The moment the numbers were split by query type, the shape of the fault changed completely.

Size, not time

2.4 KB does not fit in one UDP datagram on a 1500-byte MTU link. It goes out as fragments — the first carrying the UDP header and the port numbers, the rest carrying nothing but IP headers and payload.

That distinction is the whole fault. A non-first fragment has no port numbers in it. There is nothing in the packet that says “this belongs to UDP port 27016”. The only way anything can know is to hold the fragments, reassemble them, and look at the result.

So the question stopped being “why is it slow” and became “what in this path handles fragments”, and that question has a much shorter answer.

Why nothing showed up

Here is the part that cost the most time, and the part most worth writing down.

I checked the obvious counters. The NIC reported no drops. The kernel reported no drops. The firewall’s own counters reported no drops. Every interface was clean. softnet_stat was clean. On the evidence available, nothing was being lost — and yet a 2.4 KB reply left one machine and did not arrive at another.

A packet dropped during reassembly is not dropped by any of the things that keep a counter. The defragmentation layer is handed fragments, waits for a complete set, and if its policy says no, the whole thing evaporates. There is no interface to attribute it to, and by the time it is discarded there is no longer a packet in the sense that a counter understands. It is an accounting blind spot, not a hardware fault.

If a counter says nothing is wrong, that is evidence about the counter.

The capture that lied

Worse, my first packet captures agreed with the counters: they showed the server never answering the large query at all.

They were wrong, and they were wrong because of me. A filter of the form udp port 27016 cannot match a non-first fragment, because — as above — there are no ports in one. I had written a filter that was structurally incapable of seeing the thing I was looking for, and then treated its silence as data. Traffic carrying a VLAN tag made it worse: the offsets a naive filter assumes are all shifted by four bytes.

A capture with no filter at all, taken at both ends simultaneously, showed the server answering perfectly and the fragments dying in between.

Two rules came out of that, and both are now standing orders in my own notes:

• Capture without a filter first, then narrow. A filter is a hypothesis, and testing a hypothesis with a tool shaped by that same hypothesis proves nothing.
• Capture at both ends. One end tells you what you received. Two ends tell you where it stopped.

What it actually was

The hypervisor’s per-guest firewall.

Enabling it inserts an extra bridge in front of each guest and pulls connection tracking into the path. Connection tracking on a bridge needs whole packets to make decisions about, so it loads a defragmentation module. Fragments arriving at that bridge are held, reassembled, judged — and, on a layer-2 bridge, cannot be re-fragmented afterwards. A reassembled 2.4 KB frame has nowhere to go on a 1500-byte link, so it is quietly discarded.

Small replies fit in one packet, never fragment, and sail through. Large replies do not. That is precisely the symptom, and once the mechanism is stated the symptom is not mysterious at all — it is the only symptom it could have produced.

The firewall had been enabled a few days earlier as part of unrelated work, on the reasonable-sounding principle that more filtering is better. It was my change. Nothing about the guest, the game, or its mods had altered.

What that says about the original theory

The user whose service it was told me early on, in stronger words than these, not to start with the mods — it had been fine for weeks with exactly that configuration, and the only thing that had changed was my own networking work.

That was correct, and I should have weighted it far more heavily than I did. “What changed?” beats “what looks suspicious?” every single time, and the person who runs the thing usually knows the answer. A component that has been stable for weeks is not a good suspect just because it is complex.

The fix, and the trade

Disabling the per-guest firewall on that guest restored the service immediately: the large query came back in 49 ms.

What that costs is worth stating plainly rather than glossing over. The per-guest firewall was providing a second filtering layer between guests on the same host. Removing it means guest-to-guest traffic is now governed by the network firewall and by each guest’s own rules, and not by anything on the hypervisor.

For this estate that is an acceptable trade — the guests are not mutually hostile, the perimeter is unchanged, and the alternative was a service that does not work. But it is a trade, and a post-mortem that describes a fix without naming what it gave up is marketing rather than engineering.

The general principle it replaced: filtering belongs where it can see whole flows. A layer-2 bridge that has to reassemble in order to decide, and then cannot put the packet back together to forward it, is in the wrong place to be making that decision.

What to take away

• Split the average. One slow number hid a service that was fast for most things and broken for one. The split took thirty seconds and changed the entire investigation.
• Reply size is a variable. If small works and large does not, stop thinking about time and start thinking about MTU, fragmentation and reassembly.
• Clean counters are a claim, not a proof. Reassembly drops are invisible to nearly everything that counts packets.
• Your filter can hide the fault. Especially with fragments, VLAN tags, or anything where the headers are not where a simple expression assumes.
• Ask what changed, and believe the answer. The owner of the system told me on day one. The fault was mine, in a change I had made, and I looked everywhere else first.

Nine minutes of reading; it took rather longer than that to find.