From #6583: “the migration re-ran yesterday (even though it had already run successfully a few weeks ago). The .migration-to-ferretdb-done file is time stamped yesterday … That explains why I’m seeing old data.”

That is the worst version of this bug. The instance had been migrated and had been serving from FerretDB for weeks; the marker went missing, the old staleness guard put it back on database=mongodb, and the migration ran again — importing the MongoDB copy it had been made from, over the database holding the work since. discard_partial_ferretdb could delete that database outright, because “partial” was assumed rather than checked.

Two locks on that door now, and the same fact opens both: the importer writes migration-progress.json as it goes and resumes from it, so a FerretDB with data and no checkpoint beside it is a finished database, in use — never a migration to continue.

• migration-control checks that before it probes, reads or deletes anything. If it finds one it marks the migration done, starts FerretDB, stops MongoDB and exits. Nothing is imported.
• discard_partial_ferretdb checks it again before removing a SQLite, and says so when it declines. The MongoDB data is never touched either way.

bin/migration-pending already answered the same question through bin/database-role, so neither branch should be reachable — which is why they are there. The cost of being wrong in this direction is somebody’s data.

Three ways a snap could stay on MongoDB for good, all of them reported. The snap runs on FerretDB on every platform — MongoDB is bundled to be READ during a migration, and is not what WeKan runs on — so each of these is a bug.

A database nothing could open. A MongoDB server starts only on data whose featureCompatibilityVersion is at most one major behind it, so the readers covered FCV 6.0/7.0 (mongod 7), 4.0/4.2 (mongod 4.2) and 3.x (the 3.2 tools) — and nothing covered 4.4 or 5.0. That is not a hypothetical rung: the WeKan snap shipped MongoDB 5 in February 2023 and 6.0.6 only in May, so a site that stayed on it has 5.0 files, and every reader refused them. Those instances got .mongodb-data-too-old and an explanatory page while their boards sat in a database nobody could read. mongod 5.0 is bundled now, as a fourth read-only reader, tried between 7 and 4.2 — and through cpu-exec, because MongoDB 5.0 requires AVX on x86_64 and a CPU without it should read the database under emulation rather than die on a SIGILL.

“WeKan changed to old MongoDB data.” When the migrated FerretDB copy had fallen behind the MongoDB beside it, the snap answered by switching itself to database=mongodb. That is the mail this came from: the site is put back on the database the snap is migrating away from — and when the detector guessed wrong (#6583), onto a copy that was weeks behind. The repair is the merge, not the switch: the documents MongoDB has and FerretDB does not are copied into FerretDB — inserting what is missing, overwriting nothing — and WeKan carries on there. WeKan’s history is append-only, so the work done on MongoDB after the migration lands in the card History instead of a database nobody opens. Switching to MongoDB is now only the fallback for when the merge cannot run, because serving a copy that is behind is exactly the complaint.

A failed migration that never tried again. A failure set migrate=off so it would not loop, and nothing ever set it back on. The snap stayed on MongoDB until an admin read snap logs and typed a command, and most never do. A failure is recorded now — how many attempts, when, and which snap revision — and retried by itself: immediately after the next snap refresh, since the next release is the most likely thing to have fixed it, and otherwise after a wait that doubles from an hour up to a day. The same record replaces migrate=off on the unreadable-database path, which is what makes this release’s 5.0 reader reach the instances that were already given up on. snap set wekan migrate=off still stops it completely — an admin saying “not now” is a decision, not a failure.

None of this deletes anything: the MongoDB data stays in $SNAP_COMMON and a snap set wekan database=mongodb is still the way back.

snap set wekan database=mongodb|ferretdb is removed, and so is snap run wekan.database. WeKan runs on FerretDB — every platform, every install — and MongoDB is in the amd64/arm64 snaps to be read while a migration is owed, not to be run on.

A setting could say something the data did not support, and each way it could was a report:

• set to mongodb, it kept a site on the database the snap migrates away from, for good, because nothing ever set it back — including the instances a failed migration or a wrong staleness guess had put there;
• set to ferretdb with no FerretDB present, it would have served an empty site, so the guard against that had to exist anyway;
• and every script had its own copy of “which database is this, then”.

snap-src/bin/database-role replaced it: one helper, asked by wekan-control, mongodb-control, ferretdb-control, migration-pending, attachment-repair and the configure hook, that answers from the data — is there a FerretDB with something in it, and has the migration that fills it finished? An interrupted migration is told from a finished one by the importer’s own checkpoint, so a partial FerretDB resumes and a finished one whose marker went missing is not migrated over again (#6585). A snap that still carries the old setting is told once that it is ignored, and it is unset.

The explanatory page stopped being a dead end too. When the MongoDB files cannot be read by this snap but a FerretDB copy is there, that copy is now served instead of the page — older beats unreadable — and the page’s first instruction, which used to be a command to type, says so. The rest of it now opens with the fact that the snap keeps trying by itself.

Migration-to-FerretDB.md is the whole design in one page: what moves (all text data to SQLite, CollectionFS and Meteor-Files attachments to the filesystem, the card History with it), which MongoDB versions can be read, when it runs, what happens when it fails, and how two copies are reconciled. The Admin Panel, Snap and CPU-platform docs point at it instead of describing a setting that is gone.

wekan-ondra and wekan-gantt-gpl are snap names — they exist because a snap name cannot be changed once people have it installed — and as Docker images they were only ever a second name for the same image. The release tagged them on all three registries for two versions; it does not any more, and docker pull wekanteam/wekan (or quay.io/wekan/wekan, or ghcr.io/wekan/wekan) is the image, as it always was.

Six extra repositories across three registries, each with its own visibility and its own push permission, is six new ways for a release to fail in order to publish a copy of something already published — and v10.88 failed exactly that way, an hour into an emulated build:

ERROR: failed to push quay.io/wekan/wekan-ondra:v10.88:
  unauthorized: access to the requested resource is not authorized

Quay grants push per repository and that repository had just been created by the release itself.

The -t lines are commented out, not deleted, with what it would cost to uncomment them written beside them — a line that vanishes is a line somebody re-adds next year — and the same for the names in the two verification loops and the push preflight. The manual docker-variant.yml stays for publishing one out of band; it is workflow_dispatch only and no release calls it.

Nothing is deleted from any registry: ghcr.io/wekan/wekan-ondra up to v6.99.2, quay.io/wekan/wekan-gantt-gpl to v4.41 and wekanteam/wekan-gantt-gpl to v5.62 keep working for whoever pinned them. They stop gaining versions. The snaps keep both names and are still built and published, which is the point of having them.

The v10.89 run failed four more jobs, all of them the same two mistakes one step further along.

The Windows jobs. They check this repository out to path: src, so the scripts were addressed as src/releases/… — correct until the bcrypt step does pushd "$TMP", after which a relative path resolves against a temp directory:

bash: src/releases/npm-retry.sh: No such file or directory

The location is fixed now BEFORE anything moves — SRC="$PWD/src" at the top of the step, then "$SRC/releases/…" — in all eighteen blocks that need it, and the same for the UCS job’s univention/.

The downloads that are not in a workflow. snapcraft.yaml builds the snap in its own container, and sandstorm-src/build-deps.sh runs on the runner; both still used a bare curl, and github.com’s 503s took them out:

:: curl: (56) Connection died, tried 5 times before giving up
:: caddy: no linux/arm64 archive in Caddy 2.11.4 - nothing left to try.
==> [4/7] FerretDB v1 (amd64) at deps root
curl: (56) Connection died, tried 5 times before giving up

Both go through releases/fetch.sh now — the snap parts reach it through CRAFT_PROJECT_DIR, since snapcraft mounts the project into the build — so the caddy, MongoDB, mongod 4.2/5.0 and OpenSSL downloads, the meteor-spk and Node.js tarballs and the FerretDB binary all wait an outage out. The Caddy version lookup stays a plain curl: when it fails the pinned version is used, which is what it is for. curl https://install.sandstorm.io | sudo bash became a download and a run, because a pipe cannot be retried.

tests/workflowRepoScripts.test.cjs grew the two checks that would have caught these: a repo-script path that is relative in a step which changes directory, and a bare download in the snap build or the Sandstorm deps.

The v10.88 run lost seven jobs to two mistakes of the same kind: a step that needs something and does not check whether it is there.

The scripts were not on disk yet. Moving the downloads and the package installs behind releases/fetch.sh and releases/apt-install.sh turned steps that needed nothing into steps that need this repository:

bash: /home/runner/work/wekan/wekan/releases/apt-install.sh:
        No such file or directory
bash: D:\a\wekan\wekan/releases/npm-retry.sh: No such file or directory

The first is build-extra-arches, where “Install dependencies” was the FIRST step of the job, before actions/checkout — fine while it was a plain apt-get. The second is the Windows jobs, which check this repository out to path: src, so $GITHUB_WORKSPACE/releases is a directory that does not exist there; they already called the other scripts as src/releases/…. The same two shapes were in the Flatpak job (no checkout at all), Release All Missing’s extra-arches and its charts job (path: wekan), and the UCS job (path: univention).

tests/workflowRepoScripts.test.cjs now reads every workflow and reports a step that runs releases/… before its job checks out, or through a prefix that does not match where that job put the repository. It also checks that every script a workflow names exists here.

And the push that was never going to work. The docker job built every architecture, emulated, for the best part of an hour, and threw it all away on the last line:

ERROR: failed to push quay.io/wekan/wekan-ondra:v10.88:
  unauthorized: access to the requested resource is not authorized

The credentials were fine — the login check passed. Quay grants push per repository, and wekan-ondra had just been created, so the account that pushes wekan and wekan-gantt-gpl had no rights on it. A registry will say whether it would grant a push token in one request, so the job now asks — for all nine images, before building anything — and fails in seconds with what to change, naming the per-repository setting. A registry that does not answer is a warning: that is the network, not the rights.