implement ota updates

docs/production-builds.md

@@ -0,0 +1,449 @@
+# Production builds in Gitea Actions
+
+This is the practical guide to turning a commit into something that can be
+deployed or installed: the backend + web image, the signed Android APK, and what
+to do about the desktop agent. It assumes the runner from
+[`gitea-runner.md`](gitea-runner.md) is already registered and online.
+
+The short version:
+
+* a **push to `master`** produces a snapshot — `latest` plus the short commit SHA;
+* a **`v*` tag** produces a release — the same image tagged with the version, and
+  an APK attached to the Gitea release.
+
+A tag is the only thing that means "production". `latest` moves with every push
+and should never be pinned in a production compose file.
+
+* [What the workflows produce](#what-the-workflows-produce)
+* [One-time setup](#one-time-setup)
+* [Cutting a release](#cutting-a-release)
+* [The backend + web image](#the-backend--web-image)
+* [The Android APK](#the-android-apk)
+* [Over-the-air updates](#over-the-air-updates)
+* [The desktop agent](#the-desktop-agent)
+* [Versioning and rollback](#versioning-and-rollback)
+* [Downloading artifacts without the web UI](#downloading-artifacts-without-the-web-ui)
+* [Troubleshooting](#troubleshooting)
+
+---
+
+## What the workflows produce
+
+| Workflow | Trigger | Artifact |
+| --- | --- | --- |
+| `test.yml` | push to `master`, every pull request | No release artifact. It is the gate: backend, desktop, `:core`, and the web typecheck + build |
+| `lint.yml` | push to `master`, every pull request | No release artifact. `ruff check` over `backend/` and `desktop/` |
+| `docker.yml` | push to `master`, `v*` tags | Image `git.timka.su/timka123/kadence:<short-sha>` and `:latest`; on a tag also `:<version>` |
+| `android-release.yml` | push to `master`, `v*` tags | APK artifact `kadence-release-apk` and metadata `kadence-update-metadata`; on a tag also the release asset `kadence-<version>.apk` |
+
+There is no build job for the desktop agent. See
+[The desktop agent](#the-desktop-agent) for what that means in practice.
+
+## One-time setup
+
+### The runner
+
+Use the docker-mode runner from [`gitea-runner.md`](gitea-runner.md). The
+Android job needs a real Android SDK and Gradle cache, and the Docker job needs
+access to the host's Docker daemon through the mounted socket. A host-mode runner
+answering `ubuntu-latest` is the most common reason a production build behaves
+differently from the one in the docs.
+
+### Secrets
+
+All secrets live under **Repository → Settings → Actions → Secrets**.
+
+| Secret | Used by | What it must be |
+| --- | --- | --- |
+| `REGISTRY_TOKEN` | `docker.yml` | A personal access token with `package:write`. Gitea's built-in job token cannot publish packages |
+| `REGISTRY_USER` | `docker.yml` | Optional. The registry username paired with the token; defaults to the repository owner |
+| `KADENCE_KEYSTORE_BASE64` | `android-release.yml` | `base64 -w0 your-release.keystore` |
+| `KADENCE_KEYSTORE_PASSWORD` | `android-release.yml` | The store password |
+| `KADENCE_KEY_ALIAS` | `android-release.yml` | The key alias, normally `kadence` |
+| `KADENCE_KEY_PASSWORD` | `android-release.yml` | The key password |
+
+The workflow uses Gitea's built-in `GITEA_TOKEN` to attach the APK to a tagged
+release. That token is supplied by Actions; it is not one of the secrets above.
+
+Without `REGISTRY_TOKEN` the Docker workflow still builds and smoke-tests the
+image, then skips the push with a warning. That is useful for a test repository,
+not for production.
+
+Without the keystore secrets the Android job still builds an APK, but it is
+**debug-signed**. It installs on your own phone and cannot be upgraded by a
+properly signed release later. For production, set all four keystore values:
+`KADENCE_REQUIRE_SIGNING` then makes a missing or misnamed secret fail the build
+instead of silently falling back to the debug key.
+
+### The keystore
+
+If you do not already have one, create it once and keep it somewhere safe — in a
+password manager or another encrypted backup, never in the repository:
+
+```bash
+keytool -genkeypair -v \
+  -keystore kadence-release.keystore \
+  -alias kadence \
+  -keyalg RSA -keysize 4096 -validity 10000 \
+  -dname "CN=Kadence, O=Kadence"
+
+base64 -w0 kadence-release.keystore > kadence-release.keystore.b64
+```
+
+Put the contents of `kadence-release.keystore.b64` into
+`KADENCE_KEYSTORE_BASE64`. Losing this file means future APKs cannot upgrade the
+installed app; Android treats them as a different application.
+
+### The tag convention
+
+Use annotated `v` tags with a semantic version:
+
+```bash
+git tag -a v0.2.0 -m "Kadence 0.2.0"
+git push origin v0.2.0
+```
+
+`v0.2.0`, not `0.2.0`. Both publishing workflows filter on `v*`, and the Docker
+workflow uses the tag as the image tag verbatim.
+
+## Cutting a release
+
+1. **Start from a green `master`.** The release workflows do not wait for the
+   test workflow, so check that the latest `master` run is green before tagging.
+
+   ```bash
+   git checkout master
+   git pull --ff-only
+   git status --short
+   ```
+
+2. **Tag the commit you mean to ship.** The tag is what the workflows check out;
+   an unpushed commit is not part of the build.
+
+   ```bash
+   git tag -a v0.2.0 -m "Kadence 0.2.0"
+   git push origin v0.2.0
+   ```
+
+3. **Watch both workflows.** In Gitea: **Actions → `docker`** and
+   **Actions → `android`**. The Android job is the slower one on a cold cache
+   because it installs the SDK and fetches the Glyph Matrix AAR.
+
+4. **Verify the image.** It should exist under both the version and the commit:
+
+   ```bash
+   docker pull git.timka.su/timka123/kadence:v0.2.0
+   docker image inspect git.timka.su/timka123/kadence:v0.2.0 \
+     --format '{{index .RepoDigests 0}}'
+   ```
+
+5. **Verify the APK.** Download it from the release page
+   (`https://git.timka.su/timka123/kadence/releases/tag/v0.2.0`) and check the
+   certificate before putting it on a phone:
+
+   ```bash
+   apksigner verify --print-certs kadence-v0.2.0.apk
+   ```
+
+   The certificate must be the release key, not `CN=Android Debug`.
+
+6. **Deploy or install.** The next two sections cover each artifact.
+
+## The backend + web image
+
+### What the job does
+
+`docker.yml` builds the single image from the repository `Dockerfile`: Vite
+builds the web app, then the Python 3.12 runtime serves both the API and the
+built static files. The job then starts the image against SQLite with the
+scheduler and Telegram bot disabled and waits for `/api/health` before it pushes
+anything. A tag that cannot boot is not published.
+
+The pushed names are:
+
+| Tag | Meaning |
+| --- | --- |
+| `:<short-sha>` | The exact commit, for every build |
+| `:latest` | The newest successful build from `master` or a tag; moves on every push |
+| `:vX.Y.Z` | A tagged release |
+
+### Deploying the image
+
+The repository's `deploy/deploy.sh` currently builds on the server from the
+checked-out branch. That is the simplest path, and it means the deployment does
+not need registry credentials.
+
+To deploy the image that CI actually built instead, pin it in the environment:
+
+```bash
+ssh sw_pl_1
+cd /opt/kadence
+
+# Only needed if the registry package is private.
+docker login git.timka.su
+
+export KADENCE_IMAGE=git.timka.su/timka123/kadence:v0.2.0
+docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
+docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build
+
+curl -fsS https://kadence.timka.su/api/health
+```
+
+`--no-build` is deliberate: without it, `up` may build the local checkout and
+quietly ignore the image you just pulled. The production overlay keeps the app on
+`127.0.0.1` and leaves TLS to nginx; see `docker-compose.prod.yml` and
+`deploy/nginx/kadence.timka.su.conf`.
+
+For a fully pinned deployment, use the digest from step 4 instead of the version
+tag:
+
+```bash
+export KADENCE_IMAGE=git.timka.su/timka123/kadence@sha256:<digest>
+```
+
+### Before you roll a release
+
+Take a database backup. The backend uses `create_all` rather than a migration
+tool, so it adds tables and columns but never removes or rewrites them. Rolling an
+old application back against a newer schema is only safe if the change was
+additive:
+
+```bash
+cd /opt/kadence
+docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T db \
+  sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
+  > "backup-$(date +%F-%H%M).sql"
+```
+
+## The Android APK
+
+### What the job does
+
+`android-release.yml`:
+
+1. checks out the tag or commit;
+2. installs JDK 21 and downloads the Glyph Matrix AAR from Nothing;
+3. installs the Android SDK (`platforms;android-37.0`, build tools, platform
+   tools) and restores the Gradle caches;
+4. runs `:app`'s JVM/Robolectric tests;
+5. builds `assembleRelease` with the version code from the workflow run number;
+6. verifies the signing certificate with `apksigner` and rejects a debug key;
+7. writes `update.json` (version, commit, size, SHA-256 and signer digest);
+8. uploads `kadence-release-apk` and `kadence-update-metadata`, and on a `v*`
+   tag attaches the APK to the Gitea release as `kadence-<version>.apk`.
+
+The build is signed from the keystore secrets. `KADENCE_REQUIRE_SIGNING` is set
+when a keystore was found, so a missing password or wrong alias fails the job
+rather than producing a debug-signed "release".
+
+### Version code and version name
+
+`versionCode` comes from `${{ gitea.run_number }}`, so successive CI builds can
+install over each other. That is the number Android compares when updating.
+
+`versionName` is resolved before the build:
+
+| Trigger | `versionName` |
+| --- | --- |
+| Tag `v0.2.0` | `0.2.0` (the leading `v` is dropped) |
+| Push to `master` | `0.1.0-master.<run_number>` |
+
+So a tagged release reports its tag in Settings, while a master build is
+identifiable as a CI build and still has a unique, monotonic `versionCode`.
+
+### Installing the artifact
+
+From the Actions run page, download `kadence-release-apk.zip`, unpack it, and
+install the `app-release.apk` inside:
+
+```bash
+adb install -r app-release.apk
+```
+
+Or use the release asset on a tagged build:
+
+```bash
+curl -fL -o kadence-v0.2.0.apk \
+  https://git.timka.su/timka123/kadence/releases/download/v0.2.0/kadence-v0.2.0.apk
+adb install -r kadence-v0.2.0.apk
+```
+
+The phone must already have the app installed with the same signing key. The
+debug build uses a different application id (`.debug`) and can sit beside the
+release build, but it cannot be upgraded into it.
+
+### Over-the-air updates
+
+The Android app can update itself from these artifacts, without the phone ever
+holding a Gitea credential. The repository is private, so the backend reads the
+newest successful build from `GITEA_UPDATE_BRANCH` and re-serves it behind the
+same `/api` auth as everything else.
+
+Configure the backend (compose passes these through from `.env`):
+
+| Variable | What for |
+| --- | --- |
+| `GITEA_TOKEN` | read-only token that can see this repository's Actions artifacts. Empty disables OTA: the endpoints answer `503` and the app never offers an update |
+| `GITEA_REPO` | `owner/repository`, default `timka123/kadence` |
+| `GITEA_URL` | Gitea base URL, default `https://git.timka.su` |
+| `GITEA_UPDATE_BRANCH` | branch whose successful builds are offered, default `master` |
+
+The token only needs repository read access; it is never sent to the phone. The
+two endpoints are:
+
+* `GET /api/updates/android` — `update.json` for the newest successful build,
+  with `run_number` filled in from Gitea even for older artifacts.
+* `GET /api/updates/android/apk` — the APK itself, streamed with
+  `Cache-Control: no-store`.
+
+A missing token is `503`, no usable artifact is `404`, and a Gitea failure is
+`502`. The app treats background failures quietly; Settings → Updates shows the
+reason.
+
+On the phone:
+
+* checks run on resume, at most once every six hours, in release builds only,
+  and only after onboarding;
+* only a strictly higher `version_code` is offered, and a skipped build stays
+  skipped until **Check now** in Settings;
+* the APK is downloaded only after the user asks, into the app cache, with a
+  `.part` file so an interrupted transfer cannot look installable;
+* before handing it to Android, the app checks the package name, the version
+  code and that the APK's signing certificate matches the installed app. A
+  mismatch is deleted and reported instead of producing "App not installed";
+* Android asks for "install unknown apps" permission the first time, then its
+  own installer confirms the upgrade.
+
+This is why the release keystore matters: an APK signed with a different key
+cannot upgrade an install, and the app refuses it before downloading. Debug
+builds have OTA disabled entirely.
+
+### Artifact retention
+
+Gitea's artifact retention is finite (the instance's configured default, often
+90 days). The release asset attached to a tag is not subject to the same
+retention as a workflow artifact; that is why tagged builds are the production
+path and plain `master` artifacts are only a convenience.
+
+## The desktop agent
+
+There is no desktop build in `.gitea/workflows` today. A production desktop
+"build" is still the checkout plus the installer script:
+
+```bash
+git checkout v0.2.0
+./desktop/packaging/install-desktop.sh
+```
+
+The app is plain Python with PyQt6, run by the system interpreter. The launcher
+installs a `.pth` file pointing at the checkout, so upgrading means checking out
+the new tag and re-running the installer. The privileged helper is separate and
+needs its own install:
+
+```bash
+sudo ./desktop/helper/install.sh
+```
+
+If you need a downloadable desktop artifact, build a wheel from the same tag:
+
+```bash
+cd desktop
+python3 -m venv /tmp/kadence-build
+/tmp/kadence-build/bin/pip install build
+/tmp/kadence-build/bin/python -m build
+# dist/kadence_desktop-0.2.0-py3-none-any.whl
+```
+
+That wheel contains the Python package and QML files, not the privileged helper
+or the Qt system dependency. It is not a replacement for
+`install-desktop.sh` on a machine that needs the helper and menu entry.
+
+## Versioning and rollback
+
+| Thing | Value | Where it comes from |
+| --- | --- | --- |
+| Image release tag | `v0.2.0` | Git tag, verbatim |
+| Image commit tag | first 7 characters of `HEAD` | `git rev-parse --short HEAD` |
+| Image `latest` | newest successful `master` build | `docker.yml` |
+| APK `versionCode` | workflow run number | `gitea.run_number` |
+| APK `versionName` | tag without `v`, or `0.1.0-master.<run_number>` | `Resolve the version name` step |
+| APK release asset | `kadence-v0.2.0.apk` | Git tag |
+
+To roll back the backend, redeploy the previous version tag or digest:
+
+```bash
+export KADENCE_IMAGE=git.timka.su/timka123/kadence:v0.1.3
+docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
+docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build
+```
+
+Check `/api/health` and the app log afterwards. If the newer release added a
+column the old code does not understand, restore the database backup taken before
+the upgrade; `create_all` does not undo schema changes.
+
+To roll back the phone, install the previous release APK over the current one.
+Android refuses a lower `versionCode` during a normal install; `adb install -r -d`
+allows it on a device where downgrades are permitted, but that is not a reliable
+production path. Keep the previous APK and the matching keystore, or bump the
+version code and rebuild.
+
+## Downloading artifacts without the web UI
+
+The run page in Gitea is the simplest route: **Actions → run → Artifacts**.
+For a tagged release, the attached asset is under **Releases**.
+
+With a personal access token, the API works too. Listing the artifacts of the
+latest successful `android` run:
+
+```bash
+TOKEN=...  # a Gitea token with repository read access
+REPO=https://git.timka.su/api/v1/repos/timka123/kadence
+
+curl -fsSL -H "Authorization: token $TOKEN" \
+  "$REPO/actions/artifacts?name=kadence-release-apk" \
+  | jq -r '.artifacts[] | select(.expired == false) | "\(.id) \(.created_at)"' \
+  | sort -k2 | tail -1
+```
+
+Then download and unpack the chosen artifact id:
+
+```bash
+ARTIFACT_ID=...
+curl -fsSL -H "Authorization: token $TOKEN" \
+  "$REPO/actions/artifacts/$ARTIFACT_ID/zip" -o kadence-release-apk.zip
+unzip -o kadence-release-apk.zip
+```
+
+For a tagged build, the release asset can be fetched by tag instead:
+
+```bash
+curl -fsSL -H "Authorization: token $TOKEN" \
+  "$REPO/releases/tags/v0.2.0" \
+  | jq -r '.assets[] | select(.name | startswith("kadence-")) | .browser_download_url'
+```
+
+Artifacts from an older run can be expired; the release asset is the durable
+copy.
+
+## Troubleshooting
+
+| Symptom | Likely cause |
+| --- | --- |
+| The run sits queued forever | No online runner claims `ubuntu-latest`, or the runner is registered at a scope that does not include this repository |
+| `docker login` step warns that there is no `REGISTRY_TOKEN` | The secret is missing, empty, or lacks `package:write`; the image is built but not pushed |
+| `Cannot connect to the Docker daemon` in `docker.yml` | The job container does not have the host socket; check `docker_host` and `valid_volumes` in `deploy/act-runner.yaml` |
+| The Docker push returns 401 | `REGISTRY_TOKEN` is a job token or a PAT without `package:write`, or the username does not match the token |
+| Android build fails before Gradle starts | The SDK install failed; check `android/ci/install-sdk.sh` and the cache key. A cold cache can take several minutes |
+| The APK is debug-signed despite the keystore secrets | At least one of `KADENCE_KEYSTORE_BASE64`, `KADENCE_KEYSTORE_PASSWORD`, `KADENCE_KEY_ALIAS`, `KADENCE_KEY_PASSWORD` is missing or wrong. With `KADENCE_REQUIRE_SIGNING` the job should have failed; fix the secret rather than shipping it |
+| `kadence-release-apk` is missing from the run | The build failed before the upload step, or the artifact has expired |
+| The tag build cannot attach the APK to a release | The tag was pushed but no release exists yet, or the built-in `GITEA_TOKEN` lacks permission. Create the release, or check the repository's Actions token permissions |
+| A tag push did not start a workflow | The tag does not match `v*`, or it was created locally and never pushed: `git push origin v0.2.0` |
+| The deployment still runs old code after a pull | `docker compose up` rebuilt the local checkout. Use `--no-build` and pin `KADENCE_IMAGE` |
+| Android installs reject the new APK | It is signed with a different key, or has a lower version code than the installed one. Keep the release keystore; do not switch it between releases |
+
+### Secrets are not in the build logs
+
+The workflows never echo a token or a keystore password. The APK job does print
+the certificate with `apksigner --print-certs`, which is the public certificate,
+not the private key. Do not add a debug step that prints the environment, the
+decoded keystore, or the registry token.