The bump job of the same afternoon died on a repository the release does not use:

E: Failed to fetch https://dl.google.com/linux/chrome-stable/deb/dists/stable/main/binary-amd64/Packages.gz  Hash Sum mismatch
E: Some index files failed to download.
Error: Process completed with exit code 100.

It was installing python3 and curl. A GitHub runner comes with google-chrome, microsoft-prod, azure-cli and docker repositories configured, and apt-get update fails as a whole when any one of them serves an index that does not match its own hashes - which is what a mirror looks like while it is being republished.

releases/apt-install.sh installs the packages instead. It retries the update, clearing the cached lists first - a Hash Sum mismatch is a cached index disagreeing with the server, so re-reading it reports the same thing - and if it still fails it moves the third-party lists aside and updates from the distribution archive alone, which is where every package a release job installs comes from. Both steps say what they did: a silent change of package sources would be worse than the failure. A mirror that never comes back still fails the job, saying it is the mirror.

Every apt-get update + apt-get install pair in Release All, Release All Missing, the Sandstorm, meteor-spk and Flatpak workflows, and the emulated build container, goes through it. tests/releaseAptInstall.test.cjs drives it with a fake apt-get that mismatches on demand and a fake sudo that records rather than runs - a test must not move the package sources of the machine it runs on.

The second run of the same afternoon died one step later than the first, in build-amd64, on the FerretDB binary:

curl: (22) The requested URL returned error: 503
Warning: Problem : HTTP error. Will retry in 10 seconds. 5 retries left.
...
curl: (56) Connection died, tried 5 times before giving up

Nothing was wrong with WeKan, and nothing was wrong with nodejs.org either - the bundled Node.js downloaded and verified in the same step, seconds earlier. It was github.com, and --retry 5 --retry-delay 10 gives it fifty seconds.

releases/fetch.sh is now what downloads a file in a release. It retries 5xx, 429, 408 and the connection errors on a backoff that adds up to about fifteen minutes, and every download in Release All, Release All Missing, the preflight scripts and the emulated build containers goes through it - with the Dockerfile carrying it alongside resolve-node-source.sh, which now asks it which Node.js builds exist.

The distinction it adds is the one a longer --retry cannot: a 404 is not an outage. Several callers here legitimately ask “is this published for this CPU?” and get “no” - the preflight that skips an architecture with no Node.js build yet, the MongoDB Database Tools that are not built for every platform, the .sha256sum a source may not publish. Those fail immediately and quietly. Everything else waits.

And an existence check now has three answers instead of two: present, absent, or the server would not say. That third one used to be indistinguishable from “absent”, which is how an outage could silently drop a platform from the Docker image or skip an architecture whose binary was published all along - a ::warning:: nobody reads until somebody on ppc64le asks where their image went. It now stops the job and says to re-run it.

tests/releaseDownloads.test.cjs runs the script against a local server that 503s, 404s and 429s on demand, and reads the workflows for a download that still goes straight to curl.

The last WeKan test run failed one test in all three browsers - a test that had passed for a month:

02-cards-open-view.e2e.js:66 copy-link button produces a URL that
opens the card in full-screen view
  Error: Token login failed: You've been logged out by the server.

Driving the running server over DDP with a token seeded the way the fixtures seed one shows what it is:

session A: ok           tokens: [CfgBWImyytla]
session B: ok           tokens: [CfgBWImyytla]  <- two sessions, one token
after B logged out      tokens: []              <- logout removed it
session C (same token): ERROR You've been logged out by the server.

A seeded test user has one resume token, and Meteor.logout() deletes it on the SERVER — for every session using it. The login helper called it when a page was logged in as somebody else, so switching users in one page stranded every other page of that test. Only the copy-link test logs a second page in, which is why it was the one that failed.

The helper now ends the previous session in the CLIENT instead: it drops the three Accounts keys and reloads, which the helper already knows how to do for its own first load. The token is untouched, and the page still arrives with no user on it. logout() stays as its own helper, because logging out is a real thing to test — 05-admin-users logs out and back in with a password.

Two pages are two browsers, so they now get two tokens: db.addResumeToken() adds one to an existing user, and the second tab uses it. That tab also stopped waiting for networkidle before looking for the card — a card is rendered when the subscriptions land, which is not a network event a browser can be idle about, and on a loaded machine the wait ended before the card existed.

tests/e2eSessionTokens.test.cjs pins both rules, including a scan of every spec for two logins sharing one token.