The Cheapest Part Was Already Installed

Three CI runs hung on the same day, all in the same step, before a single test had run. One sat for close to three hours before someone noticed and cancelled it. Nothing in the job itself had changed. What had changed was invisible from inside the repository: the Ubuntu package mirror that step happened to hit was slow that day, and nothing bounded how long a slow install was allowed to take.

The step doing the hanging installed Chromium’s system dependencies with apt-get, needed because this job’s Storybook interaction tests launch a real browser and the job ran on a bare runner with none of Chromium’s libraries preinstalled. One run’s own log showed the actual throughput: eighteen megabytes fetched in eleven and a half minutes, worse than twenty-seven kilobytes a second. GitHub’s own ceiling on a job with no explicit timeout is six hours, so a mirror that slow does not fail loudly. It just sits there, indistinguishable from a job doing real work, until someone happens to look.

Both problems had the same fix already sitting one job down in the same file: the end-to-end test job already ran inside a prebuilt container image carrying Chromium’s system dependencies preinstalled, needing only the browser binary itself, fetched fast from Playwright’s own CDN regardless of how the Ubuntu mirror was behaving that day. Moving the flaky job into the same container turned an apt-get install into a no-op:

container:
  image: mcr.microsoft.com/playwright:v1.62.0-noble
  options: --ipc=host

An explicit fifteen-minute timeout went on the job as a second, independent line of defense, so a future stall of any kind fails loudly on its own schedule instead of waiting on a human to notice a job that has been running for hours. Whether the combination holds is a question this job’s own next run answers on its own, the same way the previous three had already answered it the other way.