x86_64 and aarch64 become four, with i686 and armhf. That is the whole set: an AppImage BEGINS with a runtime binary the kernel executes, and runtimes exist for exactly those four. ppc64le, s390x, riscv64 and loong64 have none, so they cannot have an AppImage however the job is written - they are served by the bundle zip and the .deb, and the matrix now says so where somebody would otherwise try.

The two are not equally safe and are not treated as though they were. i686 builds on the x86_64 runner, which runs 32-bit x86 natively: no emulation, no multiarch, appimagetool-i686 runs as-is. armhf builds on the aarch64 runner, which runs 32-bit ARM only if the kernel has CONFIG_COMPAT and the 32-bit loader is installed - GitHub promises neither - so it is continue-on-error, the same treatment TSC gives its emulated armhf job. It cannot decide whether the other three reach the release.

The smoke test needed the distinction too. It starts the AppImage and waits for port 8080, which a 64-bit runner cannot do for a 32-bit build it cannot execute. A check that cannot RUN is not a failed check: when the binary will not execute at all AND it is not this machine’s architecture, it warns, says the AppImage is uploaded unchecked, and moves on. A build that DOES start and then does not answer still fails - that is the bug the step exists for.

BuildKit attaches MINIMAL provenance on its own, which is where the unknown on unknown rows beside each real platform on quay.io come from - that is the OCI marker for “not a runnable image”, so docker pull never selects one. Minimal is a build id and a timestamp, which answers nothing anybody asks of a supply chain. mode=max records what does: which commit, which base image, which build arguments, which frontend.

The hazard is not getting it wrong once, it is that WeKan pushes images from FOUR places - docker-publish.yml, two sites in release-all.yml, and releases/docker-build.sh - and a fifth added later would silently fall back to the default. So the test finds the call sites by searching the repository instead of from a list, and fails when a pushing one lacks the flag.

SBOM stays off, deliberately: it enumerates every OS package and npm dependency, so the attestation grows from tens of kilobytes to megabytes per platform. Turning it on is a decision rather than a default to drift into.

The --load build in docker-build.sh must NOT ask for it - the docker exporter cannot carry an attestation at all - and that is a test of its own, so the flag is not added there by symmetry one day.

One more thing the test pins, because it was written the wrong way twice while this was being done: a # comment sitting among a continued command’s arguments. docker buildx build \ followed by a comment line comments out the REST OF THE JOINED LINE, so the command becomes a bare docker buildx build with every platform, tag and flag swallowed. bash -n accepts it - it is valid syntax, just a different command - and a YAML run: block is a shell script, which is where it happened the first time.

The armhf snap failed all three attempts in v10.77 while s390x, ppc64el and riscv64 each built on attempt 1. The Launchpad build log says why, one line into the mongodb part:

Stage package not found in part 'mongodb': libcurl4.
Build failed

Ubuntu 24.04’s 64-bit time_t transition renamed the library, and libcurl4 is not a real package on any architecture in noble - the binary is libcurl4t64 everywhere. On the 64-bit architectures the renamed package keeps the old name alive, Provides: libcurl4 (= 8.5.0-2ubuntu10), and the armhf libcurl4t64 has no Provides line at all, because there the ABI really changed. That is what makes this class of bug reach exactly one architecture - which is also what makes it look like flakiness on that architecture.

It had already happened, to libssl3 and libgoogle-perftools4 in v10.71, and the comment above this list describes it exactly; libcurl4 was the same transition and was left behind. All fourteen stage-packages were checked against the noble armhf main+universe indices this time, and it was the only one that did not resolve.

The workflow’s own guidance made it worse. snapcraft ends a missing stage-package as a Stopped build with no artifact, which from the outside is indistinguishable from Launchpad cancelling a build, so the message offered “an OOM in the Meteor npm install, or a transient build-farm reset - re-run” for a failure that will never succeed on a re-run. The step now looks for that line first, names the missing package, says it is not transient, explains the Provides asymmetry, and gives the one-line archive query that checks a name.

s390x, ppc64el and riscv64 each printed “Remote build <arch> succeeded on attempt 1” and downloaded their .snap - and each job then ended FAILED, because the step that saves the build log could not upload it:

The path for one of the files in artifact is not valid:
/snapcraft-wekan-f82a93c2…_s390x_2026-08-09T11:39:05.txt.
Contains the following character:  Colon :

snapcraft names a remote-build log after the recipe and an ISO timestamp, and upload-artifact refuses a colon because NTFS cannot hold one. The colons become dashes now, so the timestamp is kept rather than the name thrown away - and the upload is continue-on-error, because a diagnostic upload must never be able to fail the build it is diagnosing. The snap is the deliverable, and the renaming is not the last thing that could ever make a log unuploadable.

Three things wrong at once in the v10.77 table, and every one of them silent - nothing errored, the table was simply not what it claimed to be.

Every row appeared twice. provenance-table.sh globbed provenance/**/*.tsv provenance/*.tsv, and with globstar a **/ matches zero or more directories, so the first pattern already covered the second and every file was read twice. It is one pattern now, and rows are also deduplicated on the WHOLE line - a retried step records an identical line and nothing tells the copies apart. Deliberately not sort -u -k1,1 -k2,2: two rows sharing a bundle and a binary but differing elsewhere are not a duplicate, they are a real disagreement about which Node.js went into a bundle, and hiding one at random is worse than showing both.

amd64 was missing - the platform most people download. Its step runs cd .build before mkdir -p provenance, so the rows went to .build/provenance while the upload looked at provenance/ from the workspace root and found nothing. Every other build job records from the workspace root, which is why only this one was affected. That turned out to be half the story; the entry below is the other half.

Six platforms said Version latest. amd64, arm64, win64, win32, mac-arm64 and mac-x64 passed the literal string; only the extra-architecture job asked what latest actually was, with its own inline curl. “Which FerretDB did v10.77 ship” is the one question that column exists to answer. All seven sites now call releases/ferretdb-latest-tag.sh, which asks once per job and caches, authenticates when there is a token so a shared 60/hour limit is not what makes it fail, checks the answer is shaped like a tag before printing it into a markdown cell, and prints nothing and exits 0 when it cannot find out - a release note must never fail a build that produced a good bundle.

The table’s prose also linked wekan/node; the binaries come from wekan/node-patches, which is what the rows themselves already linked.

The amd64 build failed with exit 127, after the bundle had been zipped and checksummed:

a95d331b…  wekan-10.78-amd64.zip
bash: releases/record-provenance.sh: No such file or directory
bash: releases/ferretdb-latest-tag.sh: No such file or directory

The step runs cd .build first, so nothing relative in it means what it looks like it means - and that is two bugs, of which only the second had ever been visible. bash releases/record-provenance.sh resolves against .build/, which has no releases/, so it had printed “No such file or directory” on every run since it was written and the || true on the end swallowed it. amd64 had therefore never recorded any provenance at all, and the only symptom was its absence from the table - so the missing-directory fix above repaired the half that showed and left the half that did not.

What made it loud is that the FerretDB tag lookup added alongside it is an assignment. Under set -e, and every GitHub run: is bash -e, VAR="$(cmd)" ends the step when the command fails - so a line that had been quietly doing nothing for months became a hard failure of the whole amd64 build.

Every path in that step is absolute now, and every FERRET_TAG assignment across all seven sites ends || true: the bundle is the deliverable, and which version string reaches a markdown cell is not worth failing a good build for. Only amd64’s step cds - the other six run from the workspace root, which is exactly what the v10.77 run showed, since all six uploaded provenance and amd64 did not.

The guard added with it is the general form rather than this one line: it walks back from every provenance call to its step header and requires an absolute path whenever a cd runs inside that step.

Both AppImages built. Both passed the smoke test that starts them and waits for WeKan to answer. Both uploaded as artifacts, and both were downloaded by the attaching job with matching SHA256 digests. Then:

gh release upload 'v10.77' assets/* --clobber
failed to run git: fatal: not a git repository

That job does not check the repository out - it has no reason to, it only downloads artifacts - so gh had no git remote to infer the repository from. Every other gh call in the workflow already passed --repo; those two did not.

Every call passes it now, including the ones in jobs that DO check out: a call relying on an inferred remote breaks the day somebody removes the checkout, and the error it gives names neither gh nor the workflow.

Worth recording for whoever reads that failed run, because the log looks far worse than the bug: ::error:: lines appear in the build jobs for the bundle-file check and the port check, and in the attaching job for “no architecture produced an AppImage - every build job failed”. None of them fired. They carry the escape codes of the ##[group] header - they are the SCRIPT being echoed, not output - and a few lines below them ls -lh assets shows both AppImages sitting there at 225M and 227M. One line in the whole run was a real error.

The same one-line bug as the AppImage workflow above, from the same cause: the job that attaches the bundles does not check the repository out - it only downloads artifacts - so gh had no git remote to infer the repository from and gh release upload failed with “fatal: not a git repository”. Both flatpaks had built and uploaded as artifacts; nothing reached the release. Every gh call in that workflow passes --repo now too.

And a second one beside it: assets/* was not the bundles. The artifacts also carry the ostree repository the flatpak was exported through - config, objects/, refs/, summaries/, summary, summary.idx - which is build scaffolding, and attaching it would have put a few hundred directories on the release beside the two files anyone wants. The upload names the bundles and their checksums instead.

No architectures could be added here, and the workflow header now says why rather than leaving it to be rediscovered. A flatpak runs against a RUNTIME, not the host’s libraries, so an architecture exists only if freedesktop.org publishes org.freedesktop.Platform for it: x86_64 and aarch64, the i386 and arm runtimes having been discontinued. That is the difference from the AppImage work above, which could grow from two architectures to four - an AppImage carries its own runtime binary, and those exist for i686 and armhf as well.

WeKan ships binaries other projects publish - FerretDB, the MongoDB Database Tools, Node.js - and fetches each from a release URL. There are many such URLs, across workflows, release scripts and compose files, and they must all agree: one that names a fixed version stops receiving that project’s security fixes silently, and nothing about the build fails when it does.

That is not hypothetical. A Quay scan of the v10.77 image reported Go advisories in the FerretDB binary baked into the bundle - stdlib 1.25.9 wanting 1.25.11, golang.org/x/sys v0.38.0 wanting 0.44.0. The source was already fixed and v1.48.0 was already published carrying go1.25.11 and x/sys v0.46.0; the image had simply captured an older latest at build time. Had any of these URLs been pinned instead, the rebuild would not have fixed it either.

The URLs are found by searching the repository rather than from a list, so a build site added later is checked too, and three shapes are accepted: latest/download/<asset>, the bare latest API endpoint that release-all.yml asks which version latest resolved to so the provenance table can record it, and ${VAR} whose default is separately asserted to be latest - that is FERRETDB_RELEASE in the compose files, which an operator may pin for their own reasons but which must not freeze everybody who does not.

Node.js is a different mechanism with the same effect, so it is pinned too: NODE_VERSION is the bare major 24, and releases/resolve-node-source.sh answers with that CPU’s newest 24.x from whichever of nodejs.org, unofficial-builds or wekan/node-patches has one.

What makes latest safe rather than merely convenient is the provenance table above: rebuilding an old release would embed a different FerretDB than it shipped with, and the only reason that is a trade rather than a hole is that every release RECORDS the versions and SHA256s it actually shipped. So that is asserted here as well - releases/record-provenance.sh exists, and the CHANGELOG still carries the table it produces.