The RNG Bug That Wasn't an RNG Bug
Two SecureRandom client tests were failing. Every instinct said the bug was in the random-number code. It wasn't. It was a busy-spinning restart starving lower-priority services out of CPU time entirely.
Two lower-priority SecureRandom client tests weren't running correctly, right on the heels of the entropy subsystem work from the last entry. Every instinct I had said: the bug's in the new random-number code, obviously, what else would it be. Every instinct I had was wrong, which is exactly the kind of thing worth writing down honestly instead of quietly editing out of the story.
What was actually happening
The real cause was nowhere near the RNG code. A higher-priority application slot that had been restarted was sitting in an unconditional busy spin — actively burning CPU in a loop, at high priority, with no yielding, no waiting, nothing that would let a scheduler consider it "idle" even for a moment. From the scheduler's point of view, that slot always had work to do, right now, more urgently than anything else. So it never gave up the CPU voluntarily, and the lower-priority SecureRandom client services underneath it functionally starved. They weren't failing because anything was wrong with them. They were failing because they were never getting scheduled at all.
Why this looked exactly like an RNG bug
From where I was sitting, the SecureRandom client tests were failing, so the SecureRandom client code was the obvious first suspect. That's a completely reasonable first assumption, and it's also a great example of why scheduler and service-composition behavior can hide behind a completely unrelated subsystem's symptoms. The bug wore the RNG subsystem's clothes. It had nothing to do with the RNG subsystem.
The fix, and a rule I want to be honest about
The actual fix was replacing that inappropriate busy-spin quiescent state with a proper blocking wait suited to how that composition of services was actually supposed to behave — let the restarted slot wait properly instead of spinning at full priority forever.
Here's the rule I want to be explicit about, because it's tempting after a satisfying fix like this one to retroactively explain every other weird thing that's ever happened as "oh, that was probably this same bug." I'm not going to do that. Other historical faults in this project that I never fully diagnosed stay unknown. I'm not rewriting old mysteries to make this one bug the tidy explanation for everything, just because it would make for a more satisfying narrative. It fixed exactly what it fixed.
The actual lesson here, the one worth keeping: a subsystem can look completely broken while being completely innocent, if something upstream of it is quietly eating all the CPU time it needed to prove otherwise. Worth remembering literally every time something "doesn't work" going forward. Which, as it turns out, came up again almost immediately — for a much, much dumber reason.
Originally published on The Weekend Report: https://theweekendersproject.com/report/the-rng-bug-that-wasnt-an-rng-bug