FerretDB v1 translates one MongoDB query into five different SQL dialects, so “WeKan starts on MariaDB” says very little. The question that decides whether a backend can be trusted with a board is whether {n: {$gt: 5}} returns the same documents, in the same order, as it does on SQLite — and nothing was asking it. ./build.sh → Tests → All databases (sequential) now does, and build.bat has the same entry, running the same script rather than a second implementation that would drift.

It builds FerretDB v1 from source first: the FerretDB subdirectory is cloned from git@github.com:wekan/FerretDB if it is not there, updated if it is, and built through its own build.sh, which installs the Go toolchain and the module dependencies when they are missing — so the tests run against the newest code, not a downloaded release. Then, for each backend whose database image has a build for THIS CPU — asked of the registry with docker manifest inspect, so no table can go stale — it starts that database, runs the freshly built FerretDB against it, runs the whole catalogue and stops everything. Sequentially, because they all use the same FerretDB port and a database under test should not be competing with three others.

The catalogue is 100 cases in 15 groups, taken from FerretDB v1’s own handler sources rather than from MongoDB’s manual: every query, update and bitwise operator, the aggregation stages and accumulators, projection, sorting, paging, count, distinct, indexes and uniqueness, capped collections — which is how the OpLog exists at all — and the commands whose answers may legitimately differ. The seed data is deliberately awkward, because tidy data lets a broken translation pass. Answers are normalised and compared byte for byte against SQLite, so document ORDER counts; two backends failing the same way is agreement about a limitation, one answering where another fails is a difference.

Everything lands in ../log/<datetime>/ with every other test run, including db-conformance-report.md. SAP HANA is opt-in behind WEKAN_CONFORMANCE_HANA=1 — amd64-only, ~16 GB of RAM, SAP’s licence — so a menu choice cannot start it by accident.

Three things the first real run, on arm64, found.

The FerretDB it built could not start: panic: commit.txt value ... != vcs.revision value .... Go stamps the VCS revision into the binary and FerretDB’s build/version panics when it disagrees with the committed commit.txt, which only its generator refreshes — so every commit made after the last refresh built a binary that panicked, whatever the change was. Fixed in the fork: its build.sh build regenerates them now, as the release build always did.

Ports: FerretDB listened on 27017, which is where a dev server’s database lives and where the compose files publish FerretDB. So this could not run beside anything else and, worse, could have pointed the tests at somebody else’s database and rewritten it. It listens on 37017 now and publishes the database server on 35432, both moved on if something is already listening, both settable with WEKAN_CONFORMANCE_PORT and WEKAN_CONFORMANCE_DB_PORT, and its containers are named per run so a stack started with docker compose up is never reused or stopped.

And Ctrl-C only killed whatever was in the foreground — a registry lookup, a sleep — after which the loop carried on and reported the interrupted lookup as “NO linux/arm64”, which is a lie about the image. An interrupt ends the run now, and the image check has three outcomes rather than two: has it, does not have it, could not ask. A FerretDB that will not start also prints the last lines of its log, because the reason is usually one line.