Updating a Unit to 0.1.0

Published

October 26, 2026

Modified

October 26, 2026

This guide is for whoever manages a unit: it takes a unit already in use to 0.1.0, through the one-time steps that follow, in the order they must be done. A short version for a new unit is at the end.

The commands assume a unit installed as scripts/dtk.service ships: the checkout at /home/pi/dtk, the service running as pi. Run them over SSH as pi.

1. Before the update

1.1 Note the current revision

git -C /home/pi/dtk rev-parse HEAD
git -C /home/pi/dtk submodule status
sudo cat /var/lib/dtk/backups/last-deployed-SHA

update.sh returns to the commit in last-deployed-SHA if it fails, and records it as the rollback target. With --no-pull, which the 0.1.0 install uses (2.1), it refuses to start, with nothing changed, when that file is absent, empty, or names a commit not in the checkout: the checkout is already the new code and cannot stand in for it. In that case, before 2.1, write the commit the first command printed:

echo <commit> | sudo tee /var/lib/dtk/backups/last-deployed-SHA

/var/lib/dtk/backups/ is root’s alone (mode 0700), so reading it needs sudo.

1.2 Check free space

df -h / /var/lib/dtk

update.sh rebuilds the frontend image, reinstalls the backend environment and writes a compressed database dump to /var/lib/dtk/backups/. It keeps the five newest dumps and deletes older ones.

1.3 The database backup

You do not need to take a backup by hand. update.sh takes two pg_dumps to /var/lib/dtk/backups/pre-update-<YYYYmmdd-HHMMSS>.sql.gz. The first is a trial while the unit still serves, which checks pg_dump and free space before any downtime. The second is taken after the backend has stopped, so nothing saved before the stop is missing from it; once it is verified, the trial is deleted. Each is refused if it is under 100 bytes. The dumps hold every account’s password hash, so they are written mode 0600 in a root-only directory. The dumps already on the unit:

sudo ls -lt /var/lib/dtk/backups/

1.4 Stop any interim upload timer

Some units were given an upload timer by hand before the backend could upload. These timers are unit-specific and are not part of the software. On the Rionegro unit the timer is b2push-interim, and it runs the script /home/pi/b2push.sh, which pushes the images and also takes the daily pg_dumpall of the catalogue to the bucket. Find the timer and stop it, so that after the update only the backend writes to the bucket:

systemctl list-timers --all
sudo systemctl disable --now b2push-interim.timer

Use the name list-timers shows on your unit. The name of the service the timer starts is not recorded; systemctl list-timers shows it in its ACTIVATES column. If a run of it is in progress, wait for it to end before step 2.

If you abandon the update before step 3, enable the timer again, so the unit goes on uploading as it did before:

sudo systemctl enable --now b2push-interim.timer

1.5 Pin the unit’s Tailscale name

The update renames the unit from digitool to captua (2.6) when apt can reach its sources: the alias that keeps digitool.local answering needs avahi-utils, which the update installs. A unit that cannot reach apt keeps its name until an update that can. Tailscale takes the machine’s name from its hostname unless the name is set, so a unit reached as ssh pi-dtk or by its Tailscale name would change name in the tailnet with it. Set the Tailscale name explicitly first, to the name the unit has in the tailnet now:

tailscale status --self
sudo tailscale set --hostname=<the unit's current Tailscale name>

The software does not touch Tailscale. Skip this on a unit without Tailscale.

2. Run the update

2.1 Put the 0.1.0 code on disk first

For the 0.1.0 install, fetch and check out the new code by hand, and only then run update.sh with --no-pull. This is the only path for this install, on every unit. Bash reads a script from disk as it runs, so an update.sh that fetches its own update runs the old copy’s failure handling, and none of the protections of the new script apply to that run.

Units run main, which moves only at releases. A unit on dev (the bench unit and the Rionegro unit, at the time of writing) is moved to main here, once. As pi:

git -C /home/pi/dtk fetch origin
git -C /home/pi/dtk rev-parse --verify --quiet refs/heads/main

Check that the commit in last-deployed-SHA (1.1) is in the release’s history:

git -C /home/pi/dtk merge-base --is-ancestor "$(sudo cat /var/lib/dtk/backups/last-deployed-SHA)" origin/main && echo ok

It must print ok. If it does not, the unit runs code newer than the release, or code the release does not contain: stop, and decide what to do with that code before going on. update.sh --no-pull checks ancestry against the commit checked out, not origin/main: it refuses a checkout that does not descend from last-deployed-SHA (2.2).

If the second command above prints nothing, there is no local main yet; create it tracking origin/main:

git -C /home/pi/dtk checkout --track origin/main

If it prints a commit, a local main exists; check it out, bring it to origin/main and set its upstream:

git -C /home/pi/dtk checkout main
git -C /home/pi/dtk merge --ff-only origin/main
git -C /home/pi/dtk branch --set-upstream-to=origin/main

Then, in both cases:

git -C /home/pi/dtk rev-parse --abbrev-ref --symbolic-full-name '@{u}'
git -C /home/pi/dtk submodule update --init --recursive
git -C /home/pi/dtk submodule status
git -C /home/pi/dtk status --porcelain --untracked-files=no
git -C /home/pi/dtk submodule foreach --quiet --recursive 'git status --porcelain --untracked-files=no'
  • The first command must print origin/main: later runs of update.sh fetch from it, and refuse to fetch without an upstream.
  • Every line of submodule status must begin with a space (each submodule on its pin).
  • The last two commands must print nothing. update.sh does not put back a checkout with modified tracked files if it fails, and rollback-update.sh refuses one, so the tree must be clean before 2.2.

Check also that no tracked file is marked assume-unchanged or skip-worktree, which hides changes from git status. Both commands must print nothing:

git -C /home/pi/dtk ls-files -v | grep '^[a-zS] '
git -C /home/pi/dtk submodule foreach --quiet --recursive 'git ls-files -v | grep "^[a-zS] " || true'

update.sh --no-pull checks the submodule pins, the flags, modified tracked files and ancestry itself, and refuses with nothing changed (2.2). It does not check the branch or its upstream. The checks here are what to run before it, so that it does not refuse.

checkout refuses when local changes would be overwritten; set them aside by hand first, since update.sh does that only when it fetches.

After a rollback to a version before 0.1.0. The rollback puts back that version’s update.sh, which is not the 0.1.0 one. The next update must again take this path: fetch and check out the code by hand, then run the new update.sh with --no-pull (8.4). The ordinary path in “Later updates” below would run the old script.

2.2 Run update.sh outside the SSH session

Make sure nobody captures while update.sh is running: it stops the service part-way. Only one update or rollback runs at a time: both take the lock /run/dtk-update.lock, and a second run is refused with the holder’s process id.

sudo systemd-run --unit=dtk-deploy --collect /home/pi/dtk/scripts/update.sh --yes --no-pull

--no-pull means that the code was fetched by hand, not that the unit has no network: the update still installs avahi-utils from apt for the rename (2.6). On a unit with no network at all, add --offline, and the rename waits.

Never run update.sh directly in an SSH session. If the connection is lost, the shell sends the script SIGHUP and it stops part-way. Under systemd-run it runs as its own transient unit and continues whatever happens to the connection.

--yes is required here. update.sh asks for confirmation when the update may need internet access, and with no terminal and no --yes it aborts.

When --no-pull refuses

With --no-pull, update.sh installs whatever is checked out, so it checks the checkout first. Each refusal comes before the script changes anything, and each message ends with the commands that show and fix the problem.

The message begins Meaning Fix
✗ --no-pull: a submodule is not on the commit HEAD records (<commit>), / or its staged pointer differs from it (or this could not be read). A submodule’s checked-out commit, or a pointer staged in the superproject, is not the gitlink HEAD records. git -C /home/pi/dtk reset -q -- <submodule path> for a staged pointer, then git -C /home/pi/dtk submodule update --init --recursive.
✗ --no-pull: tracked files are marked assume-unchanged or skip-worktree A tracked file, in the superproject or a submodule, carries a flag that hides its changes. The message lists the files. In the repository that owns each file (the superproject, or the submodule the listed path begins with): git -C <repository> update-index --no-assume-unchanged --no-skip-worktree <file>, with <file> the path inside that repository. Then look at what git status --short reports.
✗ --no-pull: the checkout has modified tracked files (changed or staged), A tracked file is changed or staged, in the superproject or a submodule. Copy aside anything worth keeping, commit it, or discard it. The message gives the reset --hard HEAD commands that discard every local change to tracked files.
✗ --no-pull: the code checked out (<commit>) does not descend from the version / this unit last deployed (<commit>)., or ✗ --no-pull: could not check whether the code checked out (<commit>) / descends from the version this unit last deployed (<commit>). The checkout is older than, or unrelated to, the commit in last-deployed-SHA, or git could not tell. Installing it would put older code over a database migrated for the deployed version. To go back to an earlier version, use the rollback (step 8). To install a newer version, check out a commit that contains the deployed one.

Each refusal says Nothing has been changed. The script prints its git commands with sudo -u pi; as pi, leave that prefix out. The fourth prints sudo /home/pi/dtk/scripts/rollback-update.sh.

Later updates (0.1.x onward). The script on disk is then already the 0.1.0 one, so the ordinary path applies: sudo systemd-run --unit=dtk-deploy --collect /home/pi/dtk/scripts/update.sh --yes, which fetches and fast-forwards main itself. This holds only while the checkout is on 0.1.0 or later. After any rollback to a version before 0.1.0, the script on disk is the old one again, and the next update takes the 2.1 path: fetch and check out by hand, then the new update.sh with --no-pull (8.4).

2.3 What update.sh does

In order:

  1. Records what is running: the last deployed commit, the frontend image (kept under a pre-update-<timestamp> tag) and the backend’s pixi.lock. These are what it puts back if the update fails.
  2. Without --no-pull only (later updates): fetches the branch’s upstream and fast-forwards to it, then syncs the submodules. It refuses a branch that cannot be fast-forwarded. Locally modified tracked files are copied to /var/lib/dtk/local-mods-<timestamp>/ before they are reset; untracked files are never deleted.
  3. Rebuilds the frontend image and runs pixi install --locked for the backend. The containers and the backend keep running, but the code and the pixi environment on disk are already the new ones.
  4. Runs provision-chdk-usb.sh, provision-rclone.sh --advisory and provision-hostname.sh --prepare --advisory. The second installs the pinned rclone if needed and keeps what offsite.env already records. Under --advisory it never writes a read key. The third installs what the rename needs (2.6).
  5. Takes the trial dump while the unit serves (1.3), and records the rollback target in /var/lib/dtk/backups/pre-update-SHA.
  6. Stops dtk and checks that no backend process is left, then renames a digitool unit to captua (2.6).
  7. Starts the database alone and takes the dump with the backend stopped.
  8. Writes the unresolved record (7.3) and applies the migrations with pixi run db-upgrade, then stops the database.
  9. Starts dtk and waits up to 300 seconds for the backend to answer /health.
  10. Writes the new commit to last-deployed-SHA, removes the unresolved record, and relaunches the kiosk (2.6).

What update.sh does when one of these steps fails is in step 7.

2.4 Follow it

journalctl -u dtk-deploy -f

The script has finished when it prints Update complete! or one of its failure messages. With --collect, systemd removes the transient unit when the script ends; journalctl -u dtk-deploy still reads its log.

Update complete! is printed only after the backend has answered /health (step 9 above). Read its start as well:

curl -s http://localhost:8000/health
journalctl -u dtk --since "15 min ago" | grep -E "Volume layout:|Cannot start|Traceback"

/health answers {"status":"ok"} only once the backend has finished its start; through nginx the same route is http://localhost/api/health. Cannot start: database schema is out of date or Cannot start: the database is unreachable means the backend refused to serve. For the Volume layout: lines, see 3.4.

2.5 Check that nothing uploads yet

The backend starts its upload schedules at boot only when DTK_OFFSITE_REMOTE and a valid DTK_OFFSITE_TENANT are both set. A unit not yet provisioned for 0.1.0 has no tenant, so nothing is uploaded. Confirm it before step 3 with the request in 6.3: images.enabled must be false. If it is true, the uploader keys each object by the file’s name on disk, so an image copied into the bucket under a contract name would be uploaded again at a second key, and adopt-map would then refuse its entry as key_differs. On a unit whose offsite.env already records a remote and a tenant, create the upload hold (4.0) before step 2, so the backend holds uploads from the update’s restart on:

sudo grep -E '^DTK_OFFSITE_(REMOTE|TENANT)=' /var/lib/dtk/offsite.env

2.6 The unit name: digitool becomes captua

From 0.1.0 every unit is called captua and answers at http://captua.local. The browser calls the API at /api on the page’s own origin, through nginx, so the unit also works when opened by its IP address or by any other name that reaches it, and the kiosk opens http://localhost instead of http://localhost:3000.

update.sh renames a unit only when its hostname is exactly digitool and .env has HOST_IP=digitool.local. A unit named otherwise by hand is left alone. The work is done by scripts/provision-hostname.sh, in two calls:

  • Before the stop (--prepare): it installs avahi-utils from apt if it is missing, the alias script /usr/local/sbin/dtk-mdns-alias, its unit dtk-mdns-alias.service, the NetworkManager hook /etc/NetworkManager/dispatcher.d/90-dtk-mdns-alias, and the drop-in /etc/systemd/system/dtk.service.d/hostname-repair.conf. It tries apt under --no-pull too, so the 0.1.0 install (2.1) renames a unit that can reach apt. Only update.sh --no-pull --offline skips apt. When avahi-utils cannot be installed, the log says The rename to captua waits for avahi-utils, the unit keeps the name digitool until an update that can reach apt, and the update goes on.
  • With dtk stopped (--switch): it writes 127.0.1.1 captua digitool to /etc/hosts, runs hostnamectl set-hostname captua, restarts avahi-daemon and starts dtk-mdns-alias, which keeps digitool.local answering through 0.1.x. It then checks that captua.local and digitool.local both resolve, to one of the unit’s own addresses. If either does not, or resolves to another device, it puts the name, /etc/hosts, avahi and the alias back and leaves .env as it was. If both do, it sets HOST_IP=captua.local in .env and adds http://captua.local to CORS_ORIGINS, keeping http://digitool.local.

Neither call can make the update fail. A rename that does not happen prints ⚠ The rename to captua did not finish, and the unit keeps working as digitool.

A power cut in the middle of the rename is handled at the next start of dtk: the drop-in runs provision-hostname.sh --repair. If the hostname had already changed, it writes the 127.0.1.1 captua digitool line again, starts the alias and checks both names as the rename does. When both resolve to the unit it finishes .env and records the rename as done; when captua.local answers from another device it undoes the rename; when the names do not answer yet (no network at boot) it finishes .env and leaves the record pending for the next start. If the hostname is still digitool, it undoes the rename; if it has since been set by hand to another name (--name), it keeps that name and removes the record. An undo puts /etc/hosts back only once the hostname is digitool again; if hostnamectl fails, everything stays as it is for the next start to try. A unit called captua whose .env still has HOST_IP=digitool.local is finished the same way, whatever the record says. The repair also removes the temporary files a power cut can leave next to .env, /etc/hosts and the record.

A failed update, and a rollback, restore the code but not the name. The code before 0.1.0 works under it: its frontend calls http://${HOST_IP}/api, now http://captua.local/api, which resolves, and CORS_ORIGINS lists both names, so a page opened as digitool.local can call it.

Once the new version answers /health, update.sh ends the kiosk’s cage (only pi’s), and the tty1 autologin relaunches it from the new start-kiosk.sh, on http://localhost. That is a new origin for the browser, so the kiosk asks for a login once. update.sh --resolve-forward (7.4) relaunches it the same way once it has started dtk, and a rollback relaunches it on the restored version’s address. Chromium’s profile lock, ~/.config/chromium/SingletonLock, names the hostname it was taken under, and Chromium refuses a profile locked under another name. The rename points the running kiosk’s lock at captua (and an undo back at digitool), so that the next Chromium, started by whichever start-kiosk.sh is on disk after an update, a restore or a rollback, can take it over. The kiosk user is read from the tty1 autologin that install-kiosk-service.sh sets up. Before it starts Chromium, the 0.1.0 start-kiosk.sh also removes SingletonLock, SingletonCookie and SingletonSocket, unless a Chromium of pi’s is still running.

Two units on one network that are both called digitool cannot both become captua: the second one’s check finds captua.local answering from the first and undoes its rename. Give it its own name by hand, as in SD Card Distribution, troubleshooting; its update then leaves it alone.

3. Give each volume its stem, then restart

3.1 Set the stems

In the interface, open the project, edit each volume (Edit Volume) and fill in FILE NAME STEM. The rules:

  • Lowercase letters and digits joined by single hyphens, at most 27 characters.
  • The stem equals the volume’s reference code in Fisqua. A stem that differs from the volume’s recorded reference code is refused. In Edit Volume the message is The stem has to be the volume's Fisqua reference code, "<code>"; "<stem>" is different. Use "<code>" as the stem. A PATCH to the volume gets 409 with The reference code '<code>' is not the file name stem '<stem>'; a volume's reference code is its file name stem. An import rejects the item with filename_stem_differs_from_reference_code (“The filenameStem is not the same as the referenceCode.”) and goes on with the rest.
  • Unique on the unit, and not the directory name of a project.
  • Refused while <projects root>/<stem>/ already holds files that are not this volume’s.
  • A volume with no stem can be given one at any time, including after it has captured. Once a volume has a stem and has captured, the stem cannot be changed or cleared.

On a volume that already has images, the stem is fixed the moment it is saved. Check each stem against the volume’s reference code in Fisqua before you save it.

Setting a stem moves no file. The backend writes new captures into the volume to <stem>/captures/ at once. Do not capture into a volume that has a rename map until its adopt-map and raise-capture-counter have been applied (5.1, 5.3): until then the backend can give a capture a number the bucket already holds.

3.2 Preview the move (optional)

python -m app.core.volume_layout --dry-run prints, as JSON, what the next start would move, and writes nothing. It needs the database up and the backend stopped, so stop dtk, start the database alone, run it, and stop the database again (verify on the bench):

sudo systemctl stop dtk
sudo docker compose -f /home/pi/dtk/docker-compose.yml -f /home/pi/dtk/docker-compose.pi.yml up -d db
sudo docker compose -f /home/pi/dtk/docker-compose.yml -f /home/pi/dtk/docker-compose.pi.yml exec -T db pg_isready
cd /home/pi/dtk/backend
pixi run python -m app.core.volume_layout --dry-run > ~/volume-layout-dry-run.json
sudo docker compose -f /home/pi/dtk/docker-compose.yml -f /home/pi/dtk/docker-compose.pi.yml down --remove-orphans

Repeat pg_isready until it reports that the server is accepting connections. The command exits with 1 when failed is not empty and with 2 when it refuses (a journal it cannot complete). Do not start dtk again here; 3.3 does.

3.3 Restart so the backend moves the files

sudo systemctl restart dtk

Do not restart until every stem is right: from this start on, the files of each volume with a stem are in <stem>/, and rolling back no longer returns the unit to a usable state (8.3).

At every start, before it serves, the backend moves the existing files of each volume with a stem from the name-based layout into <stem>/captures/, <stem>/previews/ and <stem>/thumbnails/. It works one volume at a time, through a journal in /var/lib/dtk/volume-layout.journal, and completes an interrupted batch at the next start. Once every volume with a stem is in place, it moves nothing.

The backend already moved the files of volumes that had a stem before the update, when update.sh started it.

3.4 Read the Volume layout: lines

journalctl -u dtk --since "15 min ago" | grep "Volume layout:"

The summary line is written only when something was moved, adopted or completed. A restart after giving stems to volumes with files, with no summary line, needs checking.

Line Meaning
moved N image(s) and pointed M row(s) at files already in volume directories under <root> The summary. M counts rows whose old path was gone and whose file was found in captures/ with the capture-time sha256.
image N not moved, left at <path>: <error> The backend could not move that file. It left the file and its row as they were and went on with the next image. The row is still out of place, so the backend tries again at its next start.
image N not moved (<reason>): <path> The plan skipped the image. Reasons below.
<dir> already holds what is not volume N's, so its files are not moved there: [...] <stem>/ holds entries that belong to no image of the volume. None of the volume’s files are moved until those entries are removed.
<root>/<stem> or one of its directories resolves outside the projects root <root>; ... A link under <stem>/ points outside the projects root. Nothing of that volume is moved.
sidecar not yet beside its master, still at <path> The backend moved a RAW master but not its preview. It tries the preview again at every start.
file no record references, left where it is: <path> A file no row names, in a volume directory or one files were moved out of. Nothing deletes it. Inside <stem>/, it blocks later moves of that volume (stem_dir_occupied) until it is removed.
temporary file of an interrupted write, left where it is: <path> Left in place; it does not block the move.
generated thumbnail no row references, left where it is: <path> Left in place; it does not block the move.
no capture checksum is held for image N at <path>; no relocation record is written for its move to <path> The file was moved, but the manifest has no checksum for it, so the rename is not recorded there.
no manifest holds a capture record for capture(s) [...] of volume N The volume’s own manifest lacks capture records that no other manifest holds either.
copied N missing record(s) into volume N's manifest Missing capture records were copied in from other manifests.
completing the interrupted batch of volume N A batch interrupted by a crash or power loss was finished.
not run, the projects root is not usable: ... Nothing was moved. The backend serves.
refusing to serve A journal that names another projects root, or a journal or sidecar list that cannot be read. The backend does not start until it is resolved. For another root, activate that storage again and restart.
failed An unexpected error. The backend serves unless the failure left a journal.

The skip reasons:

Reason Meaning
stem_dir_occupied <stem>/ holds what is not the volume’s (see the table above).
stem_dir_is_project_tree <stem>/ is a live project directory.
stem_dir_outside_projects_root <stem>/ resolves outside the projects root.
no_stem The stored stem is not a valid stem.
outside_storage, outside_projects_root The row’s path is not under the active projects root. Uploads made before this layout, under /var/lib/dtk/uploads/, stay there and are reported this way.
missing The row’s file is gone and no file in captures/ matches its capture-time sha256.
row_changed The row changed while the move was being planned.

The journal also holds lines logged at INFO, such as the one for freed files pointed at their new names: since backend aa0a10a (#134), the backend sends the INFO lines of its own app.* and capture.* loggers to stderr, which the journal records, as <LEVEL> <logger>: <message> (app/core/logging_setup.py). The level is set by DTK_LOG_LEVEL, INFO by default.

4. Provision the offsite copy, without restarting

4.0 Create the upload hold

While /var/lib/dtk/offsite-hold exists, nothing is sent offsite:

  • the backend uploads and lists no image, and stops a run already under way after the object in progress;
  • the backend takes no catalogue dump;
  • freeing a volume from the card is refused. The free’s dry run and restore are not held.

The backend checks for the file at the start of each run, so creating or removing it takes effect at the next run, with no restart. It also holds through a restart caused by a crash, a reboot or a power cut. The commands in step 5 do not read it. Only the file’s existence counts. Its path can be changed with DTK_OFFSITE_HOLD_FILE. GET /system/catalogue-backup reports it as held and hold_file. The send to Fisqua and the Fisqua status poll are held too: each answers “held” at its start, with no call to Fisqua and no manifest built (9.3).

Create it before provisioning, or before step 2 on a unit whose offsite.env already records a remote and a tenant (2.5):

sudo touch /var/lib/dtk/offsite-hold

Remove it as the last step (6.2).

4.1 The key file

Put the unit’s key material in a file only root can read, with an editor:

sudo install -m 0600 /dev/null /root/dtk-offsite-key
sudo nano /root/dtk-offsite-key

with these lines:

KEY_ID=...
APP_KEY=...
BUCKET=...
READ_KEY_ID=...
READ_APP_KEY=...

provision-rclone.sh prints the same file as a heredoc (sudo sh -c 'cat > /root/dtk-offsite-key' <<'END'). Typed into an interactive shell, a heredoc’s lines, keys included, are saved in that shell’s history; an editor keeps them out of it.

Key What it is
KEY_ID, APP_KEY The unit’s write key, captua-<unit>-write: listBuckets, listFiles and writeFiles. No readFiles and no deleteFiles, so a stolen unit can neither download nor erase the archive.
BUCKET The unit’s bucket (table below).
READ_KEY_ID, READ_APP_KEY Optional. The unit’s read key, captua-<unit>-read: listBuckets, listFiles and readFiles. It must differ from the write key. Without it, restoring a freed volume is unavailable.

Both keys are scoped to the unit’s bucket and its tenant prefix. provision-rclone.sh asks only for writeFiles and listFiles on the write key and listFiles and readFiles on the read key; the keys issued for 0.1.0 also carry listBuckets, as the interim Rionegro key did. Each 0.1.0 key was tested from a workstation with the b2 command-line tool: it authorised, listed its tenant prefix, and was refused outside it. rclone on a unit has not used them yet, and no key without listBuckets has been tried, so whether rclone needs listBuckets is untested (verify). The file’s mode is 0600 and its owner root.

For the link to Fisqua, the same file holds FISQUA_TOKEN=<token>, the token Fisqua issued for this unit: exactly 64 lowercase hexadecimal characters (9.2).

Units and tenants

Unit Tenant (--tenant) Bucket (BUCKET) Keys
rionegro ahr neogranadina-masters captua-rionegro-write, captua-rionegro-read
lab ampl ampl-masters captua-lab-write, captua-lab-read
sbmal sbmal ampl-masters captua-sbmal-write, captua-sbmal-read
cobos (staging only) cobos ampl-masters captua-cobos-write, captua-cobos-read

Never print a key id or an application key in a log, a ticket or this guide.

Never pass a key as an assignment in front of sudo: it becomes part of sudo’s command line, which any local user can read from /proc/<pid>/cmdline.

4.2 Run the provisioning script

sudo DTK_OFFSITE_KEY_FILE=/root/dtk-offsite-key \
    /home/pi/dtk/scripts/provision-rclone.sh \
    --tenant <tenant> \
    --dump-recipient age172wkzadqepd565chppztfw8jefpcga8s7t6eqrfhnxnv2nlfkdrqygrl3m
  • --tenant is the prefix every object of the unit is keyed under, and the unit’s slug in Fisqua. The script refuses a value that is not 1 to 63 lowercase letters, digits and single hyphens, starting with a letter and not ending with a hyphen, or that is one of platform, www, api, admin or app. Replacing a tenant already recorded also needs --change-tenant.
  • --dump-recipient is the age public key the catalogue dumps are encrypted to; the one above is the recipient for every unit. The script refuses a private key. The private key never goes on the unit.
  • --fisqua-env production or --fisqua-env staging turns on the link to Fisqua, with FISQUA_TOKEN in the key file (9.2).
  • The tenant, the dump recipient, the Fisqua environment and the Fisqua token are kept on later runs that do not give them, including the run inside every update.

Before it probes the bucket, the script checks that api.backblazeb2.com answers. Any HTTP answer counts, an error status included; only a connection that cannot be made skips the probes, and the script then says so. The probes are what test the credentials: the script lists the tenant prefix once with the write key and, when this run wrote a read key, once with the read key. A listing that fails prints a warning and does not stop the run. With no tenant recorded, the write probe lists the bucket and the read probe is skipped.

The script writes the rclone config at /var/lib/dtk/rclone.conf (the upload remote, and [b2read] when a read key is given) and writes /var/lib/dtk/offsite.env whole, both 0600 and owned by pi. It never replaces an existing credential. It does not start, stop or restart dtk.

If it reports /var/lib/dtk/rclone.conf already exists and was NOT replaced, the key material was not used and offsite.env was not written, so nothing was recorded, the tenant and the recipient included. Resolve it before step 5: the script prints the command that moves the file aside. Then run the command in 4.2 again, with --tenant and --dump-recipient; the re-run line the script prints leaves both out, and nothing recorded them.

When the run has succeeded, remove the key file:

sudo rm /root/dtk-offsite-key

4.3 Check what was recorded

sudo sed 's/^DTK_FISQUA_TOKEN=.*/DTK_FISQUA_TOKEN=(not shown)/' /var/lib/dtk/offsite.env

The sed prints the file with the Fisqua token replaced, so the token does not reach the screen. The file holds DTK_RCLONE_BIN, DTK_OFFSITE_REMOTE, DTK_OFFSITE_TENANT and DTK_OFFSITE_DUMP_RECIPIENT; DTK_OFFSITE_READ_REMOTE when a read key was given; DTK_RCLONE_CONFIG when the run established the rclone config’s path; and DTK_FISQUA_TOKEN and DTK_FISQUA_ENV on a unit with the link to Fisqua (9.2).

4.4 Why nothing uploads yet

The backend reads .env and /var/lib/dtk/offsite.env once, when its settings are built at import; a key in offsite.env wins over the same key in .env. The upload and dump schedules are started at boot. The running backend therefore keeps the settings it started with, which have no tenant, and uploads nothing until it is restarted in step 6.

Each command in step 5 is a new process, which builds its settings at its own start and so reads the new offsite.env. That is what lets step 5 list the bucket and record keys before the first upload.

A backend restarted before step 6, by a crash, a reboot or a power cut, reads the tenant. The hold (4.0) is what keeps it from uploading then.

4.5 Offsite bandwidth

The backend runs the image uploads, the catalogue dump uploads and the restores of freed files under one bandwidth cap, DTK_OFFSITE_BWLIMIT, passed to rclone as --bwlimit. Bucket listings and calls to Fisqua are not capped. The default is 750k, about 750 KB/s (verify the unit rclone reads), the setting the Rionegro unit has used over its ~12 Mbit/s link. For another site, measure the uplink in Mbit/s, divide by 8 for MB/s, and take about half. The bench unit ran with the default until the cobos-bench-005 run (9.3), and was set to 1.5M after it, from a home uplink of ~25 Mbit/s measured on 2026-10-25.

Set it in /home/pi/dtk/.env; provision-rclone.sh does not write it:

DTK_OFFSITE_BWLIMIT=1.5M

The backend reads it at start. During an update, set it before step 6, and the restart in 6.1 applies it. Outside an update, restart dtk: sudo systemctl restart dtk.

5. One-time commands

Run each command first without --apply, which writes nothing and prints the report as JSON, and then with --apply. Run them as pi, from backend/, where the backend finds ../.env:

cd /home/pi/dtk/backend
pixi run python -m app.core.image_upload <command> ... [--apply]

Exit codes: 0 with the report; 2 with error: <reason>, or with a usage message for a mistyped command; 1 with a Python traceback for any other failure. The database must be up, so leave dtk running.

The reports can be long. Send each to a file and read it there:

pixi run python -m app.core.image_upload adopt-map /path/to/map.json > ~/adopt-map-<stem>-dry.json

adopt, adopt-map and backfill-preview-digests hash the files on the card in the dry run as well as with --apply, so both runs take time on a large volume, and an SSH disconnect stops them. Under systemd-run they survive a disconnect. Send the report to a file, truncated at each run, and errors to the journal (verify the form on the bench):

sudo systemd-run --unit=dtk-adopt --collect --uid=pi \
    --working-directory=/home/pi/dtk/backend \
    --property=StandardOutput=truncate:/home/pi/adopt-map-<stem>.json \
    --property=StandardError=journal \
    /home/pi/.pixi/bin/pixi run python -m app.core.image_upload adopt-map /path/to/map.json --apply
journalctl -u dtk-adopt -f

5.1 adopt-map, once per rename map

Only if the archive’s images were copied into the bucket under contract names. The rename map is the contract 1.1 JSON file that lists, for one volume, the key each capture and preview was copied to.

pixi run python -m app.core.image_upload adopt-map /path/to/map.json
pixi run python -m app.core.image_upload adopt-map /path/to/map.json --apply

Each entry that passes every check is recorded as the frozen key of its image, verified, with no upload, whether or not the file is still on the card. An entry that fails any check is listed under refused, with its reason, and nothing is recorded for it.

The whole map is refused, with error: <reason> and nothing written, when:

  • the file cannot be read or is not JSON, or is not a JSON object;
  • its contract, version or kind is not fisqua-captua-exchange, 1.1 and rename-map;
  • it has no volume or destination object, or entries is not a list;
  • no volume on the unit has both its volume.captuaCollectionId and its volume.stem;
  • DTK_OFFSITE_REMOTE is unset or not <remote>:<bucket>, or its bucket is not destination.bucket;
  • destination.root is not a key prefix;
  • DTK_OFFSITE_TENANT is unset or invalid, or is not the first segment of destination.root;
  • a bucket listing does not answer.

Read refused and refused_counts in the dry run. Resolve every refused entry whose image is still on the card before the backend starts uploading in step 6: the uploader would otherwise upload that file at a second key, its name on disk. Apply every map before 5.2, so that adopt does not record a key a map names differently.

5.2 adopt <prefix>, for objects not in a map

pixi run python -m app.core.image_upload adopt <prefix>
pixi run python -m app.core.image_upload adopt <prefix> --apply

<prefix> is relative to DTK_OFFSITE_REMOTE. The command lists it, hashes each image of a volume with a stem that has no key yet, and records a listed object as the image’s key when exactly one object carries the same SHA1. Ambiguous matches and keys already held are reported, not recorded. It only matches files still on the card.

The command does not check the tenant, but give it a prefix under the tenant (<tenant>/...). A free lists each key’s directory and lists nothing outside the tenant prefix, so a volume with a key adopted outside it can never be freed.

5.3 raise-capture-counter <stem> <counterAfter>

For each volume whose map issued capture numbers outside the unit, raise the counter to the map’s volume.counterAfter, so the next capture takes the number after it:

pixi run python -m app.core.image_upload raise-capture-counter <stem> <counterAfter>
pixi run python -m app.core.image_upload raise-capture-counter <stem> <counterAfter> --apply

The counter never decreases: a value below the current one is refused.

5.4 mark-freed <stem>

For each volume whose files were removed from the card by hand:

pixi run python -m app.core.image_upload mark-freed <stem>
pixi run python -m app.core.image_upload mark-freed <stem> --apply

It marks as freed each absent file whose verified object a fresh listing shows with the same SHA1. It deletes nothing. An absent file the bucket does not account for is listed under unaccounted and left unmarked.

Run it only with the volume’s storage mounted at the active projects root. The command judges a file absent when it is not at its path, so a volume on an external drive that is not mounted reads as entirely absent.

5.5 backfill-preview-digests <stem>

For each volume with RAW masters:

pixi run python -m app.core.image_upload backfill-preview-digests <stem>
pixi run python -m app.core.image_upload backfill-preview-digests <stem> --apply

It records the sha256 of each RAW preview captured before previews were listed in the capture record, by appending a preview_digest record to the volume’s metadata/manifest.jsonl. A free refuses a preview with no digest (no_preview_digest), so run this before the first free. The digest is computed from the file on the card: a preview removed before its digest was recorded is reported as missing (or freed) and cannot be backfilled.

6. Restart and check

6.1 Restart

sudo systemctl restart dtk

The backend now reads the tenant and starts both schedules. The backend runs each one first DTK_OFFSITE_BACKUP_DELAY seconds after the start (300 by default), and again DTK_OFFSITE_BACKUP_INTERVAL seconds after each run ends (1800 by default).

6.2 Remove the hold

As the last step, remove the hold created in 4.0:

sudo rm /var/lib/dtk/offsite-hold

The backend starts uploading at its next scheduled run, with no restart.

6.3 Check

  • Each volume’s received indicator. On the volume’s page, the offsite line is Uploading: N of M files verified, P pending while the upload is under way, and Stored offsite: N of N files verified once every object is verified. N files need attention means objects were refused: the bucket holds a different file at the key (remote_differs), or the file’s sha256 is not the one its capture record holds (fixity). A refused object is not retried. GET /collections/{id}/offsite returns the same counts.

  • The unit’s offsite state. The Configuration page’s Offsite copy section and the indicator in the dashboard read GET /system/catalogue-backup. From the unit:

    read -rs -p "Password: " PW; echo
TOKEN=$(PW="$PW" python3 -c 'import json,os; print(json.dumps({"username": "<admin>", "password": os.environ["PW"]}))' \
  | curl -s -X POST http://localhost:8000/auth/login -H "Content-Type: application/json" -d @- \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')
unset PW
curl -s http://localhost:8000/system/catalogue-backup -H "Authorization: Bearer $TOKEN"

    read -rs keeps the password off the screen and out of the shell history, and the request body goes through a pipe, so the password is never on a command line.

    held is false; enabled and images.enabled are true; images.misconfigured is null; last_verified_at fills in once the first dump is verified. A missing or invalid dump recipient is not shown here: enabled stays true and last_verified_at does not advance, and each run writes catalogue_backup_failed to the system log.

6.4 Free one volume first

Free nothing until a catalogue dump has been verified: last_verified_at in 6.3 is not null. Then choose one volume whose offsite line is Stored offsite. As an admin, open it, choose Space on the SD card, then Free space on the SD card. Captua first asks the backend for the dry run (GET /collections/{id}/offsite/free) and lists in the dialog what would be freed, and each file that would stay on the card with its reason. The dry run changes nothing. Confirm with Free space only when that list is what you expect.

Before it removes any file, the backend recomputes the SHA1 of every file, then lists the bucket again, each key’s directory once. A file that fails either check stays on the card and is listed with a reason. The thumbnails stay, so the volume can still be browsed. A freed file can be restored by an admin with Restore to the SD card, which needs the read key from 4.1.

7. If update.sh fails

7.1 What update.sh does on a failure

  • Before the service stop. It puts the checkout, the submodules, the frontend image and the pixi environment back to the version still running, and does not restart the service. The database has not been touched.
  • After the stop, before the new version is started. It puts back the code and the submodule pins, the frontend image and the pixi environment. If a migration had begun, it restores the database from the dump taken with the backend stopped, and from no other. It then starts the previous version and reports it online only once the backend answers /health. When the new commit is on no remote branch, it is kept as a local branch failed-update/<timestamp>.
  • After the new version has been started. Nothing is restored automatically: the new version may have saved data. The script stops dtk, leaves the unit stopped on the new code and the migrated database, writes the unresolved record, and prints the rollback command with its warning that everything written since the dump is discarded.

Reinstalling the previous pixi environment needs the conda registry when pixi.lock changed in the update, unless every package is still in pixi’s cache. The script says so when it cannot.

After a restore, the local branch names the previous commit and the submodules are on its pins. For the 0.1.0 install, that is main pointing at the commit the unit ran before. Before trying again, point it back at origin/main and put the submodules on its pins, since update.sh --no-pull does not sync them:

git -C /home/pi/dtk checkout -B main origin/main
git -C /home/pi/dtk submodule update --init --recursive
git -C /home/pi/dtk submodule status

Every line of submodule status must begin with a space. Then check the tree as in 2.1 before running 2.2 again.

7.2 A failed 0.1.0 install

update.sh can leave an unresolved record (7.3) with the previous code already back on disk: when it could not restore the database, refused to, or moved the code back only part-way. If the update.sh on disk then predates the record, as the pre-0.1.0 one does, it would not refuse to run. In that case, and only then, the script prints DO NOT RUN scripts/update.sh ON THIS UNIT UNTIL THE ROLLBACK HAS SUCCEEDED. Run only the recorded rollback command, under systemd-run as in 7.3.

7.3 The unresolved record

When an update cannot leave code and database in agreement, update.sh writes /var/lib/dtk/backups/unresolved-deployment and creates the flag /etc/dtk/unresolved-deployment. A systemd drop-in, /etc/systemd/system/dtk.service.d/unresolved.conf, makes ConditionPathExists=!/etc/dtk/unresolved-deployment a start condition of dtk, so systemd does not start dtk while the flag exists, on a restart or a reboot. install-service.sh installs the drop-in, and update.sh installs it whenever it writes a record.

While the flag exists, sudo systemctl restart dtk exits with 0 and does not start dtk; systemctl status dtk shows the start condition as unmet.

While the record exists, update.sh refuses to run. It prints the record, its age, and whether dtk has been started since it was written, and names the way forward.

sudo cat /var/lib/dtk/backups/unresolved-deployment
kind What happened What to do
started-failed update.sh started the new version and then failed. The code is the new version’s and the database is migrated for it. Either run the recorded rollback command, which returns code and database to the previous version and discards everything written since the dump, or, once the cause is fixed, update.sh --resolve-forward (7.4). On the first 0.1.0 install, once files have moved into <stem>/captures/, the stem guard refuses the rollback (8.0a); use --resolve-forward.
migrated-not-restored update.sh failed after the migration began, and the code was not moved back, or only part-way. The database may be migrated or partly migrated. The recorded rollback command, once whatever stopped the code from moving is cleared; or --resolve-forward if the code is exactly the failed commit and the database is at its head.
restore-failed The database was dropped for a restore, and loading the dump failed or was cut off. The database may be empty or partly restored. Also written by a rollback cut off after its drop. Only the recorded rollback command. --resolve-forward refuses this kind.
restore-refused update.sh failed after the migration, and refused the restore before anything was dropped (a backend was running, or the start guard could not be put in place). The code is back on the previous commit; the database is as the migration left it. Only the recorded rollback command. --resolve-forward refuses this kind.

The record names the rollback command in its rollback-command line, beginning with sudo. Run it under systemd-run, as update.sh runs: drop the leading sudo and prefix the rest with sudo systemd-run --unit=dtk-rollback --collect. The rollback ignores SIGHUP only from its database step on, so a lost SSH connection can stop it earlier. For example:

sudo systemd-run --unit=dtk-rollback --collect \
    /var/lib/dtk/rollback-update.sh --project-root=/home/pi/dtk --yes /var/lib/dtk/backups/pre-update-<YYYYmmdd-HHMMSS>.sql.gz
journalctl -u dtk-rollback -f
``` A successful rollback removes the record and the flag. If the rollback refuses because the checkout has modified files, clear them first.

**A flag without a record.** `update.sh` refuses, and `dtk` does not start. Find out which version the code is on and whether the database matches it (the failed run's output, `systemctl status dtk`). When they agree, remove the flag: `sudo rm /etc/dtk/unresolved-deployment`.

### 7.4 `update.sh --resolve-forward [--no-start]`

Use it to keep the version that failed, once the cause is fixed and you would rather not discard what it saved. Use the rollback (step 8) to return to the previous version.

```bash
sudo systemd-run --unit=dtk-deploy --collect /home/pi/dtk/scripts/update.sh --resolve-forward

It changes nothing until every check passes:

  • HEAD is exactly the record’s failed-commit;
  • no tracked file is modified;
  • every submodule is on its pin, and the checked-out submodule commits and any staged pointers match the gitlinks HEAD records;
  • dtk and every backend process are stopped;
  • with only the database container running, the database’s alembic_version equals the migration head of the backend checked out.

Then it writes the commit to last-deployed-SHA, removes the record and the flag, starts dtk, waits for /health and relaunches the kiosk (2.6). With --no-start it records the version and leaves the unit stopped, so that a fixed commit can then be deployed with an ordinary update. It refuses the restore-failed and restore-refused kinds, and refuses to run when there is no record.

8. Rollback

8.0 Before you roll back

If /var/lib/dtk/backups/unresolved-deployment exists, run the rollback command it records, under systemd-run as in 7.3: the rollback refuses any other commit or dump while the record exists. Otherwise:

sudo ls -lt /var/lib/dtk/backups/
sudo cat /var/lib/dtk/backups/pre-update-SHA
journalctl -u dtk-deploy | grep -A1 "Backing up the database with the backend stopped, to:"

The last command gives the dump this update took with the backend stopped. Check that it is in the listing and that pre-update-SHA names the commit last-deployed-SHA held in 1.1, or the one written into it there. Name that dump on the command line rather than relying on the newest one:

sudo systemd-run --unit=dtk-rollback --collect \
    /home/pi/dtk/scripts/rollback-update.sh --yes /var/lib/dtk/backups/pre-update-<YYYYmmdd-HHMMSS>.sql.gz
journalctl -u dtk-rollback -f

--yes is required for the same reason as in 2.2: with no terminal, the script aborts at its confirmation. With --yes it drops and restores the database without asking. The rollback takes the same lock as update.sh (2.2).

If the rollback stops part-way, run it again from the copy it keeps outside the checkout:

sudo systemd-run --unit=dtk-rollback --collect \
    /var/lib/dtk/rollback-update.sh --project-root=/home/pi/dtk --yes /var/lib/dtk/backups/pre-update-<YYYYmmdd-HHMMSS>.sql.gz

At the first start after a rollback, start.sh runs pixi run start, which may reinstall the pixi environment for the restored lock file; that needs the network.

8.0a The stem guard

Before the script changes anything, it checks whether the restored catalogue would point at files that are not there:

  • It reads every image path in the backup and looks for the file on disk, resolving a relative path as the backend does, against /home/pi/dtk/backend. A master freed to offsite storage is not counted. A volume with any image missing is at risk.
  • It asks the current database for volumes with a stem, and counts only those whose stem directory exists. It looks under the projects root named in /var/lib/dtk/storage-override.json and under the default root (PROJECTS_ROOT, or <DTK_DATA_DIR>/projects); a relative root is resolved under /home/pi/dtk/backend, as the backend resolves it. A volume with a stem directory that the backup does not have is at risk: it would be gone from the catalogue, its files left in the directory.
  • If the backup’s image or record rows cannot be read, every volume with a stem directory is listed, and the message says some of them may be safe.

If any volume is at risk, the script refuses with nothing changed and writes the list to /var/lib/dtk/backups/rollback-stems-at-risk, root’s and mode 0600. The file begins with # header lines (the dump, whether the loss was accepted, how the dump was checked), then a tab-separated table with the columns id, name, filename_stem, images_missing, images_total and example. The message begins:

✗ Restoring this backup may cut volumes off from their files:

and ends:

  Nothing has been changed. To roll back anyway, accepting this, run the
  same command again with --accept-lost-stems.

With --accept-lost-stems, the script goes on, records accepted: yes in the list and lost-stems-accepted: with the list’s path in the record it writes before dropping the database.

The script starts the database container itself if it is not running. It refuses when that fails or a query fails: ✗ Could not check the current database for volumes with a stem directory. Make the database reachable and run it again: start dtk, or, if the unresolved flag keeps dtk from starting, start the database container alone (sudo docker compose -f /home/pi/dtk/docker-compose.yml -f /home/pi/dtk/docker-compose.pi.yml up -d db). Three states are not refusals: a database that does not exist, which an interrupted restore leaves, holds no volume; a database with no volumes table holds none either; and a schema from before stems has no stem directories. In each, the backup’s own images are still checked.

The first 0.1.0 install. Once 0.1.0 has started and moved files into <stem>/captures/ (3.3, and for volumes that already had a stem, the start at the end of update.sh), the pre-update backup names those files at their old paths. A rollback to it would cut those volumes off, so the guard refuses it. After a failed update with an unresolved record of kind started-failed, use update.sh --resolve-forward (7.4), not a rollback. A record of kind migrated-not-restored means the new version never started, so no file moved for it. After a successful update, install a fixed version with an ordinary update.

No command recovers a volume after an accepted loss yet. A restore-stem command is planned after 0.1.0.

8.1 What it restores

  1. It checks that the checkout is clean and on its pins and that the pre-update commit is present, and stops with the unit still running if not. It keeps a copy of itself at /var/lib/dtk/rollback-update.sh.
  2. It stops dtk.
  3. It checks out the commit in /var/lib/dtk/backups/pre-update-SHA and syncs the submodules to that commit’s pins. With no recorded commit, the code is left as it is.
  4. It drops the database and restores the newest pre-update-*.sql.gz, or the dump named on the command line.
  5. It puts back the frontend image recorded for the restored version, when the record names it and the image is still on the unit.
  6. It removes the unresolved record and flag, starts dtk, relaunches the kiosk, and reports, component by component, whether the pixi environment and the frontend image correspond to the restored version. It does not rebuild the pixi environment; the summary gives the command.

It does not touch /var/lib/dtk/offsite.env, the rclone config, the files on the card or the bucket, nor the unit’s name (2.6), .env, /etc/hosts or avahi.

Move offsite.env aside before rolling back past step 4. Backends from ba1688e (16 October 2026) up to, but not including, 2dda6ac (24 October 2026) read offsite.env but do not encrypt the catalogue dump, so they upload it in plain text. No released version is affected (0.0.0-pre.3 predates the range); the dev superproject pins from 60f05b1 (17 October) to f429ef7 (24 October) are. A rollback target in that range, with offsite.env in place, uploads the plaintext catalogue:

sudo mv /var/lib/dtk/offsite.env /var/lib/dtk/offsite.env.aside

8.2 What it loses

Everything written to the database after the dump: captures (their files stay on the card, named by no row), stems given in step 3, keys recorded by adopt-map and adopt in step 5, counter raises, uploads recorded in step 6, and freed state.

Never roll back after a volume has been freed. The freed files are no longer on the card, the restored database names them as present, and the records that would let an admin restore them from the bucket are gone.

8.3 When it is safe

The cleanest point is before step 3. By then the backend has moved files only for volumes that had a stem before the update; on a unit where no volume had one, it has moved nothing.

Do not roll back after the restart in 3.3. From then on the files of each volume given a stem are in <stem>/, and the restored rows name their old paths. The earlier version cannot find the moved files, and nothing moves them back. The stem guard (8.0a) refuses such a rollback.

8.4 The next update

The rollback writes the restored commit to last-deployed-SHA and moves the local branch to it. The update.sh on disk is then the restored version’s, not the 0.1.0 one, so the next update repeats the 0.1.0 install:

  1. 2.1: bring main back to origin/main (git -C /home/pi/dtk checkout -B main origin/main), sync the submodules, and check the tree.

  2. If offsite.env was moved aside (8.1), create the hold and put offsite.env back:

    sudo touch /var/lib/dtk/offsite-hold
sudo mv /var/lib/dtk/offsite.env.aside /var/lib/dtk/offsite.env

    offsite.env stays aside only while the rolled-back backend in the plaintext range runs.

  3. 2.2, with --no-pull.

  4. Steps 3 to 6 again: the rollback discarded everything written after the dump (8.2).

When the unit is updated to 0.1.0 again, the backend moves files at its start:

  • For a volume whose stem is in the restored database, a row whose old path is gone is pointed at the file of the same name in <stem>/captures/ when the file’s sha256 matches the capture-time checksum. A file with no such checksum is reported as missing.
  • The stem guard (8.0a) refuses a rollback that would cut a volume off from its files. After a rollback accepted with --accept-lost-stems, giving a lost stem again is refused while <stem>/ holds files that none of the volume’s rows name. No command recovers a volume after an accepted loss yet; restore-stem is planned after 0.1.0.

9. Fisqua

9.2 Provisioning

Add the token to the key file of 4.1 and give the environment in 4.2:

FISQUA_TOKEN=<64 lowercase hexadecimal characters>
sudo DTK_OFFSITE_KEY_FILE=/root/dtk-offsite-key \
    /home/pi/dtk/scripts/provision-rclone.sh \
    --tenant <tenant> \
    --dump-recipient age172wkzadqepd565chppztfw8jefpcga8s7t6eqrfhnxnv2nlfkdrqygrl3m \
    --fisqua-env production
  • On a unit already provisioned in 4.2, the key file can hold the FISQUA_TOKEN line alone, since the original was removed there. The script then keeps the recorded remote and its credential and writes offsite.env again with the token and the environment.
  • The script reads the token from the key file and from nowhere else, refuses one that is not 64 lowercase hexadecimal characters without printing it, and refuses it when no tenant or no environment is given or recorded.
  • It writes DTK_FISQUA_TOKEN and DTK_FISQUA_ENV to offsite.env. Both are kept on later runs that do not give them. A changed value of either is applied with a note.
  • Under --advisory, as update.sh runs it, the script does not take the token.
  • The script makes no call to Fisqua. The backend learns of a wrong token at its first call: Fisqua answers 401, and the backend stops sending and polling (9.4). The backend makes its first call to Fisqua only when the first handed-over volume is fully verified offsite, so a wrong token can go unnoticed for hours or days after provisioning.

The backend reads offsite.env at start. During an update, run the script before step 6, and the restart in 6.1 applies it. Outside an update, restart dtk. Remove the key file once the script has run:

sudo rm /root/dtk-offsite-key

Outside an update, restart dtk as well:

sudo systemctl restart dtk

9.3 What a handover does

When an operator sends a volume for review, the backend queues a capture manifest for it in the same transaction, then wakes the image upload schedule, so that the backend starts an upload run at once instead of at the end of its interval. After each image upload run, the backend runs a send pass:

  1. The backend holds a queued manifest until the volume is received: every object of the volume verified in the bucket.
  2. The backend then builds the manifest, under a fixed manifest id, and sends it in the same pass.
  3. From Fisqua’s answer, the backend marks the manifest delivered or refused. After a transient failure the manifest becomes due again after DTK_FISQUA_RETRY_BASE seconds (300), twice as long after each further failure, at most DTK_FISQUA_RETRY_MAX (6 hours), and never before the time Fisqua’s Retry-After gives. The backend sends it at the first send pass after that. The backend runs a send pass after each image upload run, and runs those every DTK_OFFSITE_BACKUP_INTERVAL (30 minutes by default), or at once when a handover wakes the schedule. The backend does not stop retrying.

A volume whose recorded reference code is not its stem is refused at once, not sent, with the problem reference_code_differs. A volume the manifest builder refuses is refused with the builder’s problems. Taking a volume back from review withdraws its unsent manifests; nothing is sent to Fisqua. A new handover replaces the volume’s unsent manifests. While the hold exists (4.0), no manifest is built or sent.

On the bench unit on 2026-10-25, volume cobos-bench-005, with the default cap of 750k (4.5): the backend started the upload 2 seconds after the handover, verified its 12 objects in 5 minutes 43 seconds, and sent the manifest 0.08 seconds after the last verification. After a take-back and a new handover, Fisqua registered the new manifest within 3 seconds.

The backend first asks Fisqua for the status of its volumes DTK_OFFSITE_BACKUP_DELAY seconds (300) after start, and then every ten minutes (DTK_FISQUA_POLL_INTERVAL, 600). It asks about a volume while Fisqua has not given a final state for the volume’s newest sent manifest, and while Fisqua offers a decisions set for the volume that the backend has not yet read (9.5). The final states are published, refused and superseded.

9.4 The volume page

In the volume view’s top bar, under the offsite copy line, a line shows the delivery to Fisqua of the volume’s newest manifest that was neither superseded nor withdrawn. When there is more to say (a configuration stop, an incident, problems, notes, earlier sends), selecting the line opens a panel. The texts of the line:

Text Meaning
Not sent to Fisqua yet No manifest is queued for the volume, or every one was superseded or withdrawn.
Waiting for the complete offsite copy before sending to Fisqua Queued; the volume is not yet received.
Captua is sending the capture manifest to Fisqua Built and waiting for the next send pass (ready), or being sent (sending). During a configuration stop for token, not_found or redirect, the line stays on this text, and the panel shows the warning.
Captua could not send the capture manifest to Fisqua; it will try again <when> A transient failure; retried.
Not sent to Fisqua: Captua cannot build the capture manifest Refused by the unit, with the problems listed.
Fisqua refused the capture manifest Refused by Fisqua, with its problems.
Sent to Fisqua; waiting for its status Delivered; no status yet.

The panel’s Earlier sends (<n>) list shows the volume’s other manifests with the same texts. Two texts appear only there: Not sent: a later send for review replaced it for a superseded manifest and Not sent: the volume was withdrawn from review for a withdrawn one.

Once Fisqua answers, the line shows its status, from Fisqua received the volume to Published in Fisqua, or Fisqua does not know this volume yet (usually the manifest did not arrive; the backend keeps asking). The panel lists the problems and notes from Fisqua, with Fisqua’s text prefixed From Fisqua:. Captua words four rules itself: unlisted-object and missing-object (when Fisqua names the object), bad-request and payload-too-large. Fisqua could not find some files in the offsite copy that Captua had already verified there. This needs investigating: ask whoever manages the scanner to check the offsite copy. The missing files are listed below. is an incident: check the bucket for the keys listed.

Configuration stops. The backend stops sending and polling for the whole unit when Fisqua answers 401 or 403 (reason token), 404 (not_found) or any redirect (redirect). A 404 with rule no-decisions on a decisions read is not a stop (9.5). The backend also stops on a 400 whose problems say Fisqua cannot read the manifests this software writes (rules version, envelope or shape, reason version); that manifest is refused for good. The journal has:

ERROR app.core.fisqua_outbox: Fisqua answered HTTP <status> (<reason>) for this unit's configuration (fingerprint <8 hex>); sending and polling stop until DTK_FISQUA_ENV, DTK_FISQUA_BASE_URL or DTK_FISQUA_TOKEN is changed

For reason version the backend writes this line instead; <seconds> is DTK_FISQUA_RETRY_MAX (21600 by default):

ERROR app.core.fisqua_outbox: Fisqua answered HTTP 400 (version) for this unit's manifests (fingerprint <8 hex>); sending and polling stop until the software is updated, or until a retest every <seconds> s finds that Fisqua reads its manifests

The volume page and the Configuration page say why. For a token stop they read Fisqua does not accept the token configured on this scanner, so Captua has stopped sending volumes to Fisqua and checking their status. Ask whoever manages the scanner to correct DTK_FISQUA_TOKEN and restart Captua. For reason version they read Fisqua does not read the capture manifests this version of Captua writes, so Captua has stopped sending volumes to Fisqua and checking their status. Ask whoever manages the scanner to update Captua or Fisqua.

The stop is kept in /var/lib/dtk/fisqua-config-error.json and survives restarts. The backend clears the stop when the environment, the tenant (through the base URL built from it, unless DTK_FISQUA_BASE_URL is set), DTK_FISQUA_BASE_URL, the token or the installed code changes; the journal then has INFO app.core.fisqua_outbox: The Fisqua configuration was changed since Fisqua refused it; sending resumes. The backend reads offsite.env at start: during an update, the restart in 6.1 applies a change to it; outside an update, apply it with sudo systemctl restart dtk. An update that changes the backend commit or the release also clears a token stop; if the token is still wrong, the backend’s next call stops the unit again. The backend also removes a version stop after a retest that Fisqua reads (below).

The version stop. While it holds, the backend builds no manifest except the one a retest sends (below). Manifests already built but never sent when the refusal came are put back to be built again under new ids, so their line reads Waiting for the complete offsite copy before sending to Fisqua. The installed code, for this stop, is the backend commit and the release in VERSION: an update that changes either clears it, and an update that changes neither does not. When it cannot read its commit, the backend removes the stop at every restart. A change of environment, tenant, base URL or token clears it too, and so does a retest that Fisqua reads (below). Once the stop is removed, the backend hands each volume whose newest manifest was refused for version over again under a new manifest id, as long as none of its records is captured, and logs INFO app.core.fisqua_outbox: Volume <id> handed to Fisqua again under a new manifest id after a version refusal.

The retest. While a version stop holds, the backend sends one manifest every DTK_FISQUA_RETRY_MAX seconds (21600, six hours), counted from the time it recorded the stop or from the last retest, whichever is later. The backend sends the first retest six hours after it records the stop. The backend sends the retest at the first send pass after that time, and runs a send pass after each image upload run (9.3). A retest needs a volume handed over and received offsite: the backend takes, oldest first, a manifest waiting to be built or built and never sent, or a volume whose newest manifest was refused for version and none of whose records is captured, which it hands over again under a new manifest id. With no such volume, or no call left in the per-minute budget, the backend sends nothing and tries again at the next send pass. What the backend does next depends on Fisqua’s answer:

  • Fisqua accepts the manifest, or refuses it for its content rather than its version: the backend removes the stop. In the same pass it hands over again each volume whose newest manifest was refused for version, then builds and sends what is due, within the per-pass limit.
  • Fisqua refuses it again for version: the backend records the stop again with the time of this refusal, so the next retest is one interval later.
  • The send fails transiently: the stop holds. The backend sends the same manifest, under the same id, at the next retest, and never before the time Fisqua’s Retry-After gives.
  • Fisqua answers with a configuration error (token, not_found or redirect): that stop replaces the version stop. The backend does not retest it; it holds until the configuration or the installed code changes (above).

When Fisqua is changed to read the manifests this software writes, the backend resumes sending within six hours plus the time to the next send pass, with nothing done on the unit, provided some volume is handed over and received offsite and none of its records is captured. Otherwise the stop holds until such a volume exists, or until an update or a configuration change removes it. The backend retests nothing while the offsite upload hold (4.0) exists. Removing /var/lib/dtk/fisqua-config-error.json by hand is not needed. When an update or a configuration change removes the stop instead, the backend builds again, under the new code and new manifest ids, the manifests the retests built and never sent, and the last retest’s manifest when its answer was lost or its send failed.

The Configuration page’s Offsite copy card has a Fisqua section with the environment, the last send and the last status check. GET /system/catalogue-backup returns the same under fisqua: configured, token_invalid, environment, config_error, held, last_send and last_poll.

9.5 Fisqua’s decisions

When Fisqua has decided a volume’s records, the status poll reads the decisions and applies them, for the volume’s current handover only:

  • An approved record becomes approved; a rejected record becomes rejected, with Fisqua’s reason and note. The reviewer recorded for either is Fisqua: <decidedBy>, or Fisqua when Fisqua names no one. Fisqua’s decision replaces a local one.
  • Fisqua’s set replaces the rotation correction of each current image of a record it is applied to: an image the set gives no correction loses any local one. No file is touched.
  • Skipped and reported, with nothing changed for that record: a record not in the volume, a record that is captured (the volume was taken back, or the record retaken), a record whose current images are not the ones the decided manifest listed (images_changed), a decision that is neither approved nor rejected, and a rejection the local reject route would refuse (note_too_long, a reason not in the list, other with no note).
  • A set for an earlier manifest is not applied. A set waits while a newer handover of the volume is not yet delivered.
  • When Fisqua’s status offers a set and Fisqua answers the read with 404 and rule no-decisions, the backend applies nothing, writes INFO app.core.fisqua_outbox: Fisqua has no decisions to give for volume <id> yet; read again at the next poll, and reads the set again at the next poll. The link is not stopped. Any other 404 is a configuration stop (9.4).

9.6 The journal lines

The backend writes one line per run that did something, as <LEVEL> <logger>: <message>:

INFO app.core.image_upload: Image upload run done: uploaded=<n> already_present=<n> unverified=<n> refused=<n> conflicts=<n> skipped=<n>
INFO app.core.fisqua_outbox: Fisqua outbox pass done: promoted=<n> not_exportable=<n> delivered=<n> refused=<n> failed=<n> waiting=<n>
INFO app.core.fisqua_outbox: Fisqua accepted manifest <manifest id> of volume <id>: <state>
INFO app.core.fisqua_outbox: Fisqua status poll done: asked=<n> calls=<n> answered=<n> changed=<n> recovered=<n> decisions_fetched=<n> decisions_applied=<n> decisions_stale=<n>
INFO app.core.fisqua_decisions: Fisqua decisions <id> applied to volume <id>: approved=<n> rejected=<n> unchanged=<n> skipped=<n> rotations=<n> rotations_skipped=<n>
INFO app.core.fisqua_outbox: Fisqua has no decisions to give for volume <id> yet; read again at the next poll

During a version stop (9.4), the backend writes a line when it sends each retest, and one for its outcome; <seconds> is DTK_FISQUA_RETRY_MAX:

INFO app.core.fisqua_outbox: Fisqua version stop retest: sending manifest <manifest id> of volume <id>
INFO app.core.fisqua_outbox: Fisqua version stop retest: manifest <manifest id> accepted; the stop is cleared and sending resumes
INFO app.core.fisqua_outbox: Fisqua version stop retest: manifest <manifest id> refused for its content, not its version; the stop is cleared and sending resumes
INFO app.core.fisqua_outbox: Fisqua version stop retest: manifest <manifest id> refused again for version; the stop holds and the next retest is in <seconds> s
INFO app.core.fisqua_outbox: Fisqua version stop retest: sending manifest <manifest id> failed; the stop holds and the next retest is in <seconds> s
INFO app.core.fisqua_outbox: Fisqua version stop retest: manifest <manifest id> was answered with a configuration error (<reason>); that stop replaces the version stop
INFO app.core.fisqua_outbox: Fisqua retest manifest of outbox row <row id> is built again under the updated software

A retest refused again for version is also preceded by Fisqua refused manifest <id> of volume <id> and the ERROR line of the version stop (9.4); a failed one by Sending manifest <id> of volume <id> to Fisqua failed (attempt <n>): <error>. After a failed send the next retest waits for Fisqua’s Retry-After when that is later than <seconds>. The last line is written at the first send pass after an update or a configuration change has removed the stop, for the last retest’s manifest when its answer was lost or its send failed.

The accepted line ends no state given when Fisqua’s answer names no state.

The word after run, pass or poll is how the run ended. The levels:

  • An image upload run is written at WARNING when it refused an object or ended aborted or failed; done, stopped and held stay at INFO.
  • A send pass is written at WARNING when it refused or failed a manifest, or ended error or config_error; done, stopped and rate_limited stay at INFO. A pass that finds a stop already recorded writes no summary line, unless a retest leads the backend to remove a version stop in that pass; the backend writes the retest’s own lines.
  • A status poll is written at WARNING when it ended error, or cooling_down after a Retry-After on a decisions read; done and rate_limited stay at INFO. A poll that meets a configuration error, or a Retry-After on the status read, writes no summary line.

Failures have their own lines: Sending manifest <id> of volume <id> to Fisqua failed (attempt <n>): <error>, Fisqua refused manifest <id> of volume <id>, Fisqua manifest <id> for volume <id> is not exportable: <rules>, Asking Fisqua for the status of <n> volume(s) failed: <error>, Reading Fisqua's decisions for volume <id> failed: <error>.

9.7 When nothing arrives in Fisqua

  1. GET /system/catalogue-backup (6.3): fisqua.configured must be true. If token_invalid is true, the token is not 64 lowercase hexadecimal characters. If configured is false and token_invalid is false, check that DTK_FISQUA_TOKEN is present (the command in 4.3 shows it as (not shown)), and check DTK_FISQUA_ENV and DTK_OFFSITE_TENANT. During an update, set any change before step 6, and the restart in 6.1 applies it; outside an update, restart dtk with sudo systemctl restart dtk. If config_error is set, see the configuration stops in 9.4. For reason version, the backend retests every six hours with nothing done on the unit, provided some volume is handed over and received offsite and none of its records is captured; otherwise the stop holds until such a volume exists, or until an update or a configuration change removes it, and the backend retests nothing while the hold exists. config_error.at is the time the backend last recorded the stop, which a retest refused for version renews, and config_error.retest_at is the time of the last retest whose send failed (null otherwise). Both are UTC, written without an offset. The next retest is due six hours after the later of the two. If held is true and step 5 is complete, remove the hold (6.2); during an update, leave it.
  2. The volume’s line in 9.4. Waiting for the complete offsite copy… (the start of the text) means the upload has not verified every object: read the volume’s offsite line and the Image upload run lines.
  3. The journal: journalctl -u dtk --since "1 hour ago" | grep -E "Fisqua|Image upload run".
  4. The backend asks again every ten minutes about a delivered volume that Fisqua reports as Fisqua does not know this volume yet, and resends nothing on its own. To send it again, take the volume back from review and send it for review again: the backend queues a new manifest under a new id and sends it once the volume is received. The backend queues a new manifest only while some of the volume’s records are in review; taking the volume back returns only those records to captured, and the backend queues nothing for a handover with no captured record.

10. Checks after an update

Run these on the bench unit after every update, and on a unit in service after its update:

curl -s http://localhost:8000/health
sudo cat /var/lib/dtk/backups/last-deployed-SHA
git -C /home/pi/dtk rev-parse HEAD
cat /home/pi/dtk/VERSION
git -C /home/pi/dtk submodule status
sudo ls /var/lib/dtk/backups/unresolved-deployment /etc/dtk/unresolved-deployment
systemctl cat dtk | grep ConditionPathExists
sudo /home/pi/dtk/scripts/check-stack-health.sh
journalctl -u dtk --since "15 min ago" | grep -E "Volume layout:|Cannot start|Traceback|offsite uploads are on hold|Fisqua"
hostname
getent hosts captua.local digitool.local
grep -E '^(HOST_IP|CORS_ORIGINS)=' /home/pi/dtk/.env
systemctl is-enabled dtk-mdns-alias; systemctl is-active dtk-mdns-alias
sudo cat /etc/dtk/hostname-switch
  • /health answers {"status":"ok"}.
  • VERSION names the release installed. last-deployed-SHA and HEAD name the same commit, and every line of submodule status begins with a space (on its pin).
  • Neither unresolved file exists (ls reports both missing).
  • The drop-in’s ConditionPathExists=!/etc/dtk/unresolved-deployment is listed.
  • check-stack-health.sh reports the database healthy and port 5432 listening on the host, on any address, the frontend running and nginx running.
  • The journal has no Cannot start and no Traceback; any Volume layout: lines are read as in 3.4. offsite uploads are on hold appears at each run while the hold exists.
  • GET /system/catalogue-backup (6.3) reports held as expected for the point of the procedure, and, on a unit with the link, fisqua.configured true with no config_error.
  • The journal, on a unit with the link, has no Fisqua answered HTTP line since the restart. The Fisqua accepted manifest line (9.6) appears only after a handed-over volume is received offsite; a Fisqua outbox pass line can also report a volume refused as not exportable. Neither appears while the hold exists.
  • On a volume handed over since the update, the Fisqua line under the offsite copy line reads Sent to Fisqua; waiting for its status or a status from Fisqua (9.4).
  • On a unit renamed from digitool: hostname prints captua; getent prints an address for both names; HOST_IP=captua.local and CORS_ORIGINS lists http://captua.local and http://digitool.local; dtk-mdns-alias is enabled and active; /etc/dtk/hostname-switch reads state=done. From another machine on the same network, http://captua.local, http://digitool.local and the unit’s IP each load the app and log in. On a unit that kept its name, the update’s journal says why (2.6).

A new unit

  1. Provision the card with sudo ./scripts/setup.sh, which names it captua (sudo DTK_HOSTNAME=captua-2 ./scripts/setup.sh for a second unit at a site), then sudo ./scripts/install-service.sh, which also installs the unresolved-update drop-in (see SD Card Distribution).
  2. Write the key file and run provision-rclone.sh with --tenant and --dump-recipient, as in step 4. For a unit with the link to Fisqua, add --fisqua-env and a FISQUA_TOKEN line in the key file (9.2).
  3. Start or restart dtk and check as in 6.3.

A new volume must be given its stem when it is created, so step 3 does not apply, and with nothing in the bucket before the unit, step 5 does not apply either.