Headroom for the replay
A flaky leaderboard test traced back to tests repeatedly rediscovering deterministic scores and an arbitrary rate limit cap.
End-to-end tests for leaderboard deduplication began failing intermittently with HTTP 429 rate limit errors. The flakiness appeared when running multiple test suites concurrently against a shared database instance, but investigating the root cause revealed a double friction: the server rate limiter had an arbitrary cap that left insufficient headroom for rapid replay, while the test suite was needlessly burning through its own rate limit budget by repeating identical benchmark submissions on every single test case.
// tests/leaderboard-idempotency.test.ts
let husScore: number;
let bilBokScore: number;
test.beforeAll(async ({ request }) => {
const seed = await (await request.get('/api/daily-seed?lang=no')).json();
({ score: husScore } = await getScoreForWordHistory(request, seed, ['hus']));
({ score: bilBokScore } = await getScoreForWordHistory(request, seed, ['bil', 'bok']));
});The cost of redundant discovery#
The repeated submissions were an unintended side effect of a previous reliability improvement. When the suite moved away from hardcoded test scores, it switched to discovering scores at runtime by posting candidate words directly to the server before making assertions. That runtime lookup made the tests immune to differences in server seed secrets, but each test case in the group repeated the discovery independently. Because puzzle word scores are deterministic for the active date, language, and server seed, submitting those identical word lists multiple times produced the exact same totals while consuming real rate limit tokens.
The score was deterministic for the entire day, but the tests rediscovered it on every single assertion.
Under sequential test execution on a dedicated database, that redundant token spend went unnoticed. But in the local environment and during parallel execution, test files share an IP address and a single rate limit bucket. When another test file submitted its own valid daily puzzle scores concurrently, the accumulated consumption exhausted the remaining allowance and triggered sudden 429 rejections. Hoisting the benchmark discovery into a shared setup hook queries the server once for the whole suite, cutting out redundant submissions and leaving ample headroom for concurrent test files.
Headroom for rapid play#
The test flakiness prompted a closer look at the endpoint rate limiter itself. An earlier revision had raised the limit to prevent rapid replay attempts from hitting a wall, but that cap was still an arbitrary estimate rather than a measured boundary. The intended product behavior is that an active human player should never encounter a rate limit during normal play, even when rapidly testing alternative word strategies. Quadrupling the limit headroom reinforces its role as an abuse backstop rather than an operational ceiling, ensuring enthusiastic players are never blocked.
Pairing higher endpoint headroom with leaner test fixtures stabilized the suite across both local development databases and continuous integration. Lowering thresholds should wait for real abuse evidence in production logs, while automated test fixtures should avoid spending shared network budget on facts that can be established once.