docs(blog): add 019-oldest-wins-and-the-loser-gets-wiped · xeiaso.net/ircd.rs@a6a089b

+317

1 changed file

Expand all

docs

site

content

blog

019-oldest-wins-and-the-loser-gets-wiped.md

+317

docs/site/content/blog/019-oldest-wins-and-the-loser-gets-wiped.md

View file

··· 1 + +++ 2 + title = "Oldest wins, and the loser gets wiped" 3 + description = "When a netsplit heals, two divergent copies of the same channel and the same nick have to become one — with no coordinator, no quorum, and no second chance to ask anyone. The whole consensus protocol is a less-than sign. The catch is what happens to the side that loses." 4 + date = 2026-06-15 5 + weight = 19 6 + +++ 7 + 8 + Two servers in the same IRC network lose sight of each other. A flaky link, a 9 + dropped TCP connection, a server that wedges for ninety seconds — doesn't matter 10 + why. For that window each side keeps running. People keep talking. On the left 11 + half of the split, alice creates `#foo` and gets ops because she made it. On the 12 + right half, a *different* alice — same nick, different person, because the left 13 + alice is unreachable so the nick looks free — registers and joins her own `#foo`, 14 + and gets ops there, because she made it too. 15 + 16 + And then the link comes back. 17 + 18 + Now there are two of everything. Two channels named `#foo`, each with its own op 19 + list, its own ban list, its own topic. Two users claiming the nick `alice`. The 20 + network has to collapse all of that back into one coherent picture, and here's the 21 + constraint that makes it genuinely hard: there is no coordinator. No server is in 22 + charge. Every server in the network has to look at the merge and *independently* 23 + arrive at the exact same answer, with no vote, no round-trip, no leader to break 24 + ties. If two servers disagree about who holds `#foo`'s ops, they stay desynced 25 + forever, and a permanently split-brained channel is how you get a channel nobody 26 + can moderate. 27 + 28 + This is a distributed-consensus problem. The kind people reach for Raft to solve. 29 + 30 + IRC solves it with a less-than sign. 31 + 32 + ## The whole protocol fits in one comparison 33 + 34 + The trick is that you don't need a vote if every node is already holding the same 35 + numbers. Each channel has a creation timestamp, minted once on the server where it 36 + was first created and then carried, unchanged, everywhere the channel is known. 37 + When two copies of `#foo` meet, the rule is: the *older* one wins. Lower 38 + timestamp, smaller number, that side keeps its ops. That's it. Every server runs 39 + the identical comparison on the identical two integers and lands on the identical 40 + winner without talking to anyone. 41 + 42 + Here's the actual arbitration in `leveva`, trimmed to the decision: 43 + 44 + ```rust 45 + // leveva/src/channel.rs:1345 46 + pub fn reconcile_channel_ts(&self, name: &str, their_ts: u64, their_sid: &Sid) 47 + -> ChanTsOutcome 48 + { 49 + let mut g = self.by_name.lock().expect("channels mutex poisoned"); 50 + let Some(chan) = g.get_mut(&casemap::fold(name)) else { 51 + return ChanTsOutcome::Unknown; 52 + }; 53 + let our_ts = chan.created_at; 54 + if their_ts == our_ts { 55 + return ChanTsOutcome::Merged; // genuine same-age merge, nobody loses 56 + } 57 + let they_win = their_ts < our_ts; // <- the entire consensus protocol 58 + chan.created_at = our_ts.min(their_ts); 59 + // ...then deop the losing side... 60 + } 61 + ``` 62 + 63 + One line — `they_win = their_ts < our_ts` — is the whole decision. Everything else 64 + in the function is bookkeeping that follows from it. There's no negotiation step 65 + because there's nothing to negotiate — the inputs are fixed, the function is 66 + pure, and `<` is total. Feed every server the same pair of timestamps and they 67 + cannot disagree. 68 + 69 + > **Mara** (hacker): The reason this works where it'd fall apart for, say, a 70 + > bank ledger is that the timestamp is *immutable and birth-stamped*. It's not 71 + > "when did this server hear about the channel," it's "when was the channel 72 + > created, ever, anywhere." That value rides the wire with the channel and never 73 + > gets rewritten. So the comparison isn't comparing two servers' opinions, it's 74 + > comparing two readings of one fact. Consensus is free when everyone's looking 75 + > at the same fact. 76 + 77 + I want to be honest that "side" is doing some work in that sentence. For a 78 + two-server merge — the overwhelmingly common netjoin — it's exact: each member is 79 + homed on one server or the other, and the losing server's members get deopped. 80 + For three or more servers merging at once, `leveva` approximates "side" by each 81 + member's home server, which means a member who happened to be on the *winning* 82 + side but is homed on a third server can get caught in the deop. That's a 83 + divergence from a perfectly-correct merge, it's written down, and I'm not going 84 + to pretend the approximation is free. 85 + 86 + ## Carrying a number the ancestor never knew about 87 + 88 + The timestamp has to cross the wire, and there's a wrinkle: the daemon `leveva` 89 + descends from doesn't have this feature at all. The faithful C port — the 90 + [oracle from the last several posts](@/blog/018-the-strangler-eats-its-own-port.md) 91 + — has no concept of a channel-creation timestamp it broadcasts for merge 92 + arbitration. So if I invent a new server-to-server verb and send it raw, an older 93 + peer that doesn't understand it treats the unknown verb as a protocol error and 94 + drops the link. That's the opposite of what I want. 95 + 96 + My way out is `ENCAP`, the "encapsulated command" envelope. A peer that doesn't 97 + speak the inner subcommand routes the whole thing to `m_nop` — literally the 98 + no-op handler — and ignores it instead of choking: 99 + 100 + ```text 101 + :<sid> ENCAP * CHANTS <chan> <unix-nanoseconds> 102 + ``` 103 + 104 + That line goes out right after each channel's membership burst when two servers 105 + link. A `leveva` peer that advertised the `channel-ts` capability reconciles on 106 + it; anything older shrugs it off. It's the same backward-compatibility move every 107 + long-lived protocol eventually grows — wrap your extensions so the old 108 + implementations can skip them — and it's the reason a `leveva` network can carry 109 + a feature its own ancestor never had without breaking links to it. 110 + 111 + > **Cadey** (coffee): There's something funny about building the extension 112 + > mechanism *into* the thing whose whole point was to be byte-identical to a 113 + > thirty-year-old daemon. The oracle taught me exactly what the old wire tolerates 114 + > and what makes it hang up. Turns out "what it tolerates" is a door you can walk 115 + > new features through. 116 + 117 + ## The merge isn't a merge, it's an eviction 118 + 119 + Here's where the gentle word "merge" starts to lie. 120 + 121 + When your side loses, you don't get folded politely into the winner. You get 122 + *wiped*. Your channel ops are stripped. And — this is the part I got wrong the 123 + first time — so is everything else your side decided about the channel while it 124 + was off on its own: the ban list, the invite-exception list, the ban-exception 125 + list, and the auto-reop list. All of it, gone, replaced wholesale by the winner's. 126 + 127 + I originally didn't do that. When I first built the channel merge I kept the 128 + loser's `+R` auto-reop list around, *unioned* with the winner's, on the theory 129 + that it was harmless and saved me a step. The maintainer — which is to say, me, 130 + two days later, reading the TS6 merge rules properly — overruled it. The spec is 131 + unambiguous: the losing side drops *all* its list modes and adopts the winner's, 132 + full stop. Keeping the reop list was me sneaking a "this is probably fine" 133 + divergence into the one place where "probably fine" gets you a channel that two 134 + halves of the network moderate differently. 135 + 136 + So the loser's lists get wiped at merge time: 137 + 138 + ```rust 139 + // leveva/src/s2s/chants.rs — the "we lost" branch of apply_encap_chants 140 + if link.bursting { 141 + changes.extend(ctx.channels.wipe_list_masks(chan)); // +b +e +I AND +R 142 + link.lost_merges.insert(chan.to_string()); 143 + } 144 + ``` 145 + 146 + Wiping the reop list created a real problem, though, and it's worth sitting with 147 + because it's the kind of thing that only shows up once you take the rule 148 + seriously. Auto-reop exists to re-op the right people after a merge. But I just 149 + deleted the loser's copy of the list that says who the right people are. The fix 150 + is ordering: the winner's reop list arrives in the burst *after* the timestamp 151 + line, so I can't re-op anyone at merge time — the policy that decides who to re-op 152 + isn't here yet. I record that this channel lost, and once the whole burst has 153 + landed and the winner's `+R` list is in place, an end-of-burst sweep does the 154 + re-op against the *winner's* policy: 155 + 156 + ```rust 157 + // leveva/src/s2s/chants.rs:185 158 + pub(super) fn sweep_lost_merge_reop(link: &PeerLink, ctx: &ServerContext) { 159 + for chan in &link.lost_merges { 160 + if let Some(members) = ctx.channels.members(chan) { 161 + crate::command::enforce_reop(ctx, chan, &members, None); 162 + } 163 + } 164 + } 165 + ``` 166 + 167 + That's the whole shape of the correction: I tried to preserve something for 168 + convenience, the rule said wipe it, wiping it broke a feature that depended on it, 169 + and the real fix was to do the dependent feature *later* against the winner's 170 + data instead of earlier against the loser's. There was no shortcut. There rarely 171 + is, with this kind of thing. 172 + 173 + ## Same rule, nastier victim 174 + 175 + Nicks work the same way, and the stakes feel more personal because a nick is a 176 + person's name and a channel is a room. 177 + 178 + Every registered user has a signon timestamp — when they connected, in 179 + nanoseconds — and it rides the wire the same way the channel timestamp does, in a 180 + `USERTS` line sent just ahead of the user's introduction. When two users collide 181 + on a nick, oldest signon wins. The decision is a single pure function, and I like 182 + it enough to show the whole thing: 183 + 184 + ```rust 185 + // leveva/src/s2s/collision.rs:58 186 + pub fn resolve(incoming_ts: Option<u64>, holder_ts: Option<u64>) -> Resolution { 187 + match (incoming_ts, holder_ts) { 188 + (Some(rt), Some(ht)) => match rt.cmp(&ht) { 189 + Ordering::Less => Resolution::NewcomerWins, // older arrival takes the nick 190 + Ordering::Equal => Resolution::SaveBoth, // unresolvable — see below 191 + Ordering::Greater => Resolution::NewcomerLoses, // younger arrival keeps its UID 192 + }, 193 + // A missing timestamp on either side can't be compared → first claimant keeps it. 194 + _ => Resolution::NewcomerLoses, 195 + } 196 + } 197 + ``` 198 + 199 + Losing here means your nick is taken away and you are renamed to your UID — the 200 + internal, globally-unique identifier like `0XYZAAAAA` that every user has under 201 + the hood. You don't get disconnected; you get un-named. Your friends' clients see 202 + you blink from `alice` to a string of base-36 noise. It's brutal, and it's brutal 203 + *on purpose*, because the alternative — two users both answering to `alice` on the 204 + same network — is worse than either one of them losing the name. 205 + 206 + The implementation has one wrinkle that the channel side doesn't, and it comes 207 + from `leveva` being real async Rust instead of a single-threaded C event loop. A 208 + local client's connection runs in its own task, and that task caches its own nick. 209 + I can't just reach into the registry and rename it; the task wouldn't know. So 210 + renaming a *local* loser means sending a message to its task and asking it to 211 + rename itself: 212 + 213 + ```rust 214 + // leveva/src/registry.rs:45 215 + ForceNick { wire: Vec<u8>, new: String }, 216 + ``` 217 + 218 + Its serve loop receives that, writes the `NICK` line out to the client so its UI 219 + updates, and calls `set_nick` on its own session so its cached copy matches 220 + reality. A remote loser is simpler — you tell its home server to do the 221 + rename and trust the rename to propagate. The same eviction, two delivery paths, 222 + depending on whose task owns the doomed nick. 223 + 224 + > **Mara** (hacker): This is the tax for not being a C daemon with one global 225 + > event loop and a big array of clients you can mutate in place. In `leveva` every 226 + > connection is its own little owner of its own state, which is exactly what you 227 + > want ninety-nine percent of the time and is mildly annoying the one percent 228 + > where you need to reach across and edit someone else's state. The mailbox 229 + > message is the price of not sharing mutable globals. 230 + 231 + ## What happens when there's no oldest 232 + 233 + You can see the unresolvable case coming if you stared at the `resolve` function: 234 + what if the two timestamps are *equal*? Two users who registered in the same 235 + nanosecond, on two halves of a split, both claiming `alice`. There is no older 236 + one. The whole protocol was "compare two numbers," and the numbers are the same. 237 + 238 + The answer `leveva` takes is the one the older daemons settled on, and it's 239 + bleak in a way I find honest: if you can't decide who deserves the nick, *neither* 240 + gets it. Both sides lose. The holder is evicted to its UID and the newcomer is 241 + evicted to its UID, and the contested nick is left vacant network-wide for whoever 242 + grabs it next. A tie doesn't get awarded arbitrarily to whoever the code happened 243 + to process first — that'd be a coin flip dressed up as a decision, and a coin flip 244 + isn't deterministic across servers. Mutual destruction is. 245 + 246 + > **Cadey** (aha): I keep coming back to how the *tie* is the most principled 247 + > case. Everywhere else the rule is "older wins," which can feel unfair if you're 248 + > the one who lost by a few milliseconds. But the tie is the one spot where the 249 + > system admits it genuinely cannot know, and instead of faking a winner it burns 250 + > the contested thing down for everyone. There's no lie in it. 251 + 252 + ## How you trust this without an oracle 253 + 254 + Here's the thing I have to admit about this whole feature, and it's a first for 255 + this series. There is no oracle behind it. 256 + 257 + Every post before this one leaned on a differential test: run the faithful C copy 258 + and the Rust side against the same input, diff the bytes, and if they match I know 259 + I'm right because the C is the definition of right. That harness is the reason I 260 + could claim "100% Rust" and mean it. But the C daemon doesn't *do* timestamp-based 261 + merge — it has an entirely different nick-collision mechanism and no channel-TS 262 + arbitration to diff against. So for the first time I'm building something the 263 + oracle can't grade. I'm out past the edge of the map I spent eight phases drawing. 264 + 265 + So where does confidence come from when there's nothing to diff against? Three 266 + places, none of them as airtight as a byte-for-byte match, and I'd rather say that 267 + plainly than oversell it. The decision functions are pure and small, so they get 268 + property tests that hammer them with arbitrary timestamps and assert the 269 + invariants directly — `resolve` is total, equality is the *only* path to the 270 + both-lose outcome, exactly one side wins otherwise. The cross-task eviction, which 271 + property tests can't reach because it spans real tasks, gets an end-to-end test 272 + that boots two daemons, collides a nick, and checks the right user ends up 273 + renamed. And the design itself isn't invented — it's the timestamp-merge rule that 274 + IRC servers have used for decades, so "is this the right behavior" has a real 275 + answer I can check against, even if I can't check it byte-for-byte. 276 + 277 + It's less certainty than I'm used to. That's the trade for building something the 278 + reference never had: I gave up the oracle's grade the moment I started adding 279 + features it doesn't have, and that was always going to be the price of `leveva` 280 + being its own daemon instead of a transliteration. 281 + 282 + ## The brutality is the feature 283 + 284 + So the netsplit heals and the merge runs and somebody loses everything. Ops, ban 285 + lists, sometimes a name. Written out cold it reads merciless, and I've mostly made 286 + peace with that, because the thing that actually scares me in a distributed system 287 + isn't a harsh rule. It's ambiguity — two servers quietly disagreeing about who 288 + runs a channel and neither one ever finding out. A harsh rule every server applies 289 + the same way is safe. A kind rule that occasionally leaves two servers holding 290 + different answers is the bug you ship by accident and find six months later. 291 + 292 + Nobody voted. Nobody asked. Every server looked at the same two numbers and reached 293 + the same verdict in the same instant, alone, and that's the part I still find a 294 + little eerie — there's no coordination anywhere in it. The cruelty isn't a design 295 + goal I went looking for. It's just what's left over when you need a deterministic 296 + answer and two equal timestamps refuse to give you enough to be kind about it. 297 + 298 + --- 299 + 300 + The channel arbitration is in 301 + [`leveva/src/channel.rs`](https://tangled.org/xeiaso.net/ircd.rs/blob/master/leveva/src/channel.rs) 302 + (`reconcile_channel_ts`) and 303 + [`leveva/src/s2s/chants.rs`](https://tangled.org/xeiaso.net/ircd.rs/blob/master/leveva/src/s2s/chants.rs); 304 + the nick-collision decision is the pure `resolve` in 305 + [`leveva/src/s2s/collision.rs`](https://tangled.org/xeiaso.net/ircd.rs/blob/master/leveva/src/s2s/collision.rs), 306 + and the cross-task eviction is in 307 + [`leveva/src/s2s/unick.rs`](https://tangled.org/xeiaso.net/ircd.rs/blob/master/leveva/src/s2s/unick.rs). 308 + The phase plan is [PLAN.md](https://tangled.org/xeiaso.net/ircd.rs/blob/master/PLAN.md). 309 + [Post eighteen](@/blog/018-the-strangler-eats-its-own-port.md) is where `leveva` 310 + became its own greenfield daemon with the faithful port demoted to oracle; this is 311 + the first post about a feature that oracle can't grade. [Post 312 + three](@/blog/003-the-oracle.md) is the differential harness I'm working without 313 + here. 314 + 315 + And the standing reminder, true on every post here: this is an unsanctioned 316 + experiment on someone else's code. Leave the IRCNet team out of it. The merge that 317 + takes your ops and the tie that takes your name are both mine.

Configure Feed

Configure Feed