kadence/docs/production-builds.md

# Production builds in Gitea Actions

This is the practical guide to turning a commit into something that can be
deployed or installed: the backend + web image, the signed Android APK, and what
to do about the desktop agent. It assumes the runner from
[`gitea-runner.md`](gitea-runner.md) is already registered and online.

The short version:

* a **push to `master`** produces a snapshot — `latest` plus the short commit SHA;
* a **`v*` tag** produces a release — the same image tagged with the version, and
  an APK attached to the Gitea release.

A tag is the only thing that means "production". `latest` moves with every push
and should never be pinned in a production compose file.

* [What the workflows produce](#what-the-workflows-produce)
* [One-time setup](#one-time-setup)
* [Cutting a release](#cutting-a-release)
* [The backend + web image](#the-backend--web-image)
* [The Android APK](#the-android-apk)
* [Over-the-air updates](#over-the-air-updates)
* [The desktop agent](#the-desktop-agent)
* [Versioning and rollback](#versioning-and-rollback)
* [Downloading artifacts without the web UI](#downloading-artifacts-without-the-web-ui)
* [Troubleshooting](#troubleshooting)

---

## What the workflows produce

| Workflow | Trigger | Artifact |
| --- | --- | --- |
| `test.yml` | push to `master`, every pull request | No release artifact. It is the gate: backend, desktop, `:core`, and the web typecheck + build |
| `lint.yml` | push to `master`, every pull request | No release artifact. `ruff check` over `backend/` and `desktop/` |
| `docker.yml` | push to `master`, `v*` tags | Image `git.timka.su/timka123/kadence:<short-sha>` and `:latest`; on a tag also `:<version>` |
| `android-release.yml` | push to `master`, `v*` tags | APK artifact `kadence-release-apk` and metadata `kadence-update-metadata`; on a tag also the release asset `kadence-<version>.apk` |

There is no build job for the desktop agent. See
[The desktop agent](#the-desktop-agent) for what that means in practice.

## One-time setup

### The runner

Use the docker-mode runner from [`gitea-runner.md`](gitea-runner.md). The
Android job needs a real Android SDK and Gradle cache, and the Docker job needs
access to the host's Docker daemon through the mounted socket. A host-mode runner
answering `ubuntu-latest` is the most common reason a production build behaves
differently from the one in the docs.

### Secrets

All secrets live under **Repository → Settings → Actions → Secrets**.

| Secret | Used by | What it must be |
| --- | --- | --- |
| `REGISTRY_TOKEN` | `docker.yml` | A personal access token with `package:write`. Gitea's built-in job token cannot publish packages |
| `REGISTRY_USER` | `docker.yml` | Optional. The registry username paired with the token; defaults to the repository owner |
| `KADENCE_KEYSTORE_BASE64` | `android-release.yml` | `base64 -w0 your-release.keystore` |
| `KADENCE_KEYSTORE_PASSWORD` | `android-release.yml` | The store password |
| `KADENCE_KEY_ALIAS` | `android-release.yml` | The key alias, normally `kadence` |
| `KADENCE_KEY_PASSWORD` | `android-release.yml` | The key password |

All four `KADENCE_KEY*` secrets are needed for a release. Before uploading the
base64 blob, check that the alias and passwords actually open the file:

```bash
keytool -list \
  -keystore your-release.keystore \
  -storepass "$KADENCE_KEYSTORE_PASSWORD" \
  -alias "$KADENCE_KEY_ALIAS"
```

Then produce the secret with `base64 -w0 your-release.keystore`. The workflow
fails early and names any missing signing secret, instead of waiting for
`packageRelease` to report a keytool error.

The workflow uses Gitea's built-in `GITEA_TOKEN` to attach the APK to a tagged
release. That token is supplied by Actions; it is not one of the secrets above.

Without `REGISTRY_TOKEN` the Docker workflow still builds and smoke-tests the
image, then skips the push with a warning. That is useful for a test repository,
not for production.

Without the keystore secrets the Android job still builds an APK, but it is
**debug-signed**. It installs on your own phone and cannot be upgraded by a
properly signed release later. For production, set all four keystore values:
`KADENCE_REQUIRE_SIGNING` then makes a missing or misnamed secret fail the build
instead of silently falling back to the debug key.

### The keystore

If you do not already have one, create it once and keep it somewhere safe — in a
password manager or another encrypted backup, never in the repository:

```bash
keytool -genkeypair -v \
  -keystore kadence-release.keystore \
  -alias kadence \
  -keyalg RSA -keysize 4096 -validity 10000 \
  -dname "CN=Kadence, O=Kadence"

base64 -w0 kadence-release.keystore > kadence-release.keystore.b64
```

Put the contents of `kadence-release.keystore.b64` into
`KADENCE_KEYSTORE_BASE64`. Losing this file means future APKs cannot upgrade the
installed app; Android treats them as a different application.

### The tag convention

Use annotated `v` tags with a semantic version:

```bash
git tag -a v0.2.0 -m "Kadence 0.2.0"
git push origin v0.2.0
```

`v0.2.0`, not `0.2.0`. Both publishing workflows filter on `v*`, and the Docker
workflow uses the tag as the image tag verbatim.

## Cutting a release

1. **Start from a green `master`.** The release workflows do not wait for the
   test workflow, so check that the latest `master` run is green before tagging.

   ```bash
   git checkout master
   git pull --ff-only
   git status --short
   ```

2. **Tag the commit you mean to ship.** The tag is what the workflows check out;
   an unpushed commit is not part of the build.

   ```bash
   git tag -a v0.2.0 -m "Kadence 0.2.0"
   git push origin v0.2.0
   ```

3. **Watch both workflows.** In Gitea: **Actions → `docker`** and
   **Actions → `android`**. The Android job is the slower one on a cold cache
   because it installs the SDK and fetches the Glyph Matrix AAR.

4. **Verify the image.** It should exist under both the version and the commit:

   ```bash
   docker pull git.timka.su/timka123/kadence:v0.2.0
   docker image inspect git.timka.su/timka123/kadence:v0.2.0 \
     --format '{{index .RepoDigests 0}}'
   ```

5. **Verify the APK.** Download it from the release page
   (`https://git.timka.su/timka123/kadence/releases/tag/v0.2.0`) and check the
   certificate before putting it on a phone:

   ```bash
   apksigner verify --print-certs kadence-v0.2.0.apk
   ```

   The certificate must be the release key, not `CN=Android Debug`.

6. **Deploy or install.** The next two sections cover each artifact.

## The backend + web image

### What the job does

`docker.yml` builds the single image from the repository `Dockerfile`: Vite
builds the web app, then the Python 3.12 runtime serves both the API and the
built static files. The job then starts the image against SQLite with the
scheduler and Telegram bot disabled and waits for `/api/health` before it pushes
anything. A tag that cannot boot is not published.

The pushed names are:

| Tag | Meaning |
| --- | --- |
| `:<short-sha>` | The exact commit, for every build |
| `:latest` | The newest successful build from `master` or a tag; moves on every push |
| `:vX.Y.Z` | A tagged release |

### Deploying the image

The repository's `deploy/deploy.sh` currently builds on the server from the
checked-out branch. That is the simplest path, and it means the deployment does
not need registry credentials.

To deploy the image that CI actually built instead, pin it in the environment:

```bash
ssh sw_pl_1
cd /opt/kadence

# Only needed if the registry package is private.
docker login git.timka.su

export KADENCE_IMAGE=git.timka.su/timka123/kadence:v0.2.0
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build

curl -fsS https://kadence.timka.su/api/health
```

`--no-build` is deliberate: without it, `up` may build the local checkout and
quietly ignore the image you just pulled. The production overlay keeps the app on
`127.0.0.1` and leaves TLS to nginx; see `docker-compose.prod.yml` and
`deploy/nginx/kadence.timka.su.conf`.

For a fully pinned deployment, use the digest from step 4 instead of the version
tag:

```bash
export KADENCE_IMAGE=git.timka.su/timka123/kadence@sha256:<digest>
```

### Before you roll a release

Take a database backup. The backend uses `create_all` rather than a migration
tool, so it adds tables and columns but never removes or rewrites them. Rolling an
old application back against a newer schema is only safe if the change was
additive:

```bash
cd /opt/kadence
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T db \
  sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
  > "backup-$(date +%F-%H%M).sql"
```

## The Android APK

### What the job does

`android-release.yml`:

1. checks out the tag or commit;
2. installs JDK 21 and downloads the Glyph Matrix AAR from Nothing;
3. installs the Android SDK (`platforms;android-37.0`, build tools, platform
   tools) and restores the Gradle caches;
4. runs `:app`'s JVM/Robolectric tests;
5. builds `assembleRelease` with the version code from the workflow run number;
6. verifies the signing certificate with `apksigner` and rejects a debug key;
7. writes `update.json` (version, commit, size, SHA-256 and signer digest);
8. uploads `kadence-release-apk` and `kadence-update-metadata`, and on a `v*`
   tag attaches the APK to the Gitea release as `kadence-<version>.apk`.

The build is signed from the keystore secrets. `KADENCE_REQUIRE_SIGNING` is set
when a keystore was found, so a missing password or wrong alias fails the job
rather than producing a debug-signed "release".

### Version code and version name

`versionCode` comes from `${{ gitea.run_number }}`, so successive CI builds can
install over each other. That is the number Android compares when updating.

`versionName` is resolved before the build:

| Trigger | `versionName` |
| --- | --- |
| Tag `v0.2.0` | `0.2.0` (the leading `v` is dropped) |
| Push to `master` | `0.1.0-master.<run_number>` |

So a tagged release reports its tag in Settings, while a master build is
identifiable as a CI build and still has a unique, monotonic `versionCode`.

### Installing the artifact

From the Actions run page, download `kadence-release-apk.zip`, unpack it, and
install the `app-release.apk` inside:

```bash
adb install -r app-release.apk
```

Or use the release asset on a tagged build:

```bash
curl -fL -o kadence-v0.2.0.apk \
  https://git.timka.su/timka123/kadence/releases/download/v0.2.0/kadence-v0.2.0.apk
adb install -r kadence-v0.2.0.apk
```

The phone must already have the app installed with the same signing key. The
debug build uses a different application id (`.debug`) and can sit beside the
release build, but it cannot be upgraded into it.

### Over-the-air updates

The Android app can update itself from these artifacts, without the phone ever
holding a Gitea credential. The repository is private, so the backend reads the
newest successful build from `GITEA_UPDATE_BRANCH` and re-serves it behind the
same `/api` auth as everything else.

Configure the backend (compose passes these through from `.env`):

| Variable | What for |
| --- | --- |
| `GITEA_TOKEN` | read-only token that can see this repository's Actions artifacts. Empty disables OTA: the endpoints answer `503` and the app never offers an update |
| `GITEA_REPO` | `owner/repository`, default `timka123/kadence` |
| `GITEA_URL` | Gitea base URL, default `https://git.timka.su` |
| `GITEA_UPDATE_BRANCH` | branch whose successful builds are offered, default `master` |

The token only needs repository read access; it is never sent to the phone. The
two endpoints are:

* `GET /api/updates/android` — `update.json` for the newest successful build,
  with `run_number` filled in from Gitea even for older artifacts.
* `GET /api/updates/android/apk` — the APK itself, streamed with
  `Cache-Control: no-store`.

A missing token is `503`, no usable artifact is `404`, and a Gitea failure is
`502`. The app treats background failures quietly; Settings → Updates shows the
reason.

On the phone:

* checks run on resume, at most once every six hours, in release builds only,
  and only after onboarding;
* only a strictly higher `version_code` is offered, and a skipped build stays
  skipped until **Check now** in Settings;
* the APK is downloaded only after the user asks, into the app cache, with a
  `.part` file so an interrupted transfer cannot look installable;
* before handing it to Android, the app checks the package name, the version
  code and that the APK's signing certificate matches the installed app. A
  mismatch is deleted and reported instead of producing "App not installed";
* Android asks for "install unknown apps" permission the first time, then its
  own installer confirms the upgrade.

This is why the release keystore matters: an APK signed with a different key
cannot upgrade an install, and the app refuses it before downloading. Debug
builds have OTA disabled entirely.

### Artifact retention

Gitea's artifact retention is finite (the instance's configured default, often
90 days). The release asset attached to a tag is not subject to the same
retention as a workflow artifact; that is why tagged builds are the production
path and plain `master` artifacts are only a convenience.

## The desktop agent

There is no desktop build in `.gitea/workflows` today. A production desktop
"build" is still the checkout plus the installer script:

```bash
git checkout v0.2.0
./desktop/packaging/install-desktop.sh
```

The app is plain Python with PyQt6, run by the system interpreter. The launcher
installs a `.pth` file pointing at the checkout, so upgrading means checking out
the new tag and re-running the installer. The privileged helper is separate and
needs its own install:

```bash
sudo ./desktop/helper/install.sh
```

If you need a downloadable desktop artifact, build a wheel from the same tag:

```bash
cd desktop
python3 -m venv /tmp/kadence-build
/tmp/kadence-build/bin/pip install build
/tmp/kadence-build/bin/python -m build
# dist/kadence_desktop-0.2.0-py3-none-any.whl
```

That wheel contains the Python package and QML files, not the privileged helper
or the Qt system dependency. It is not a replacement for
`install-desktop.sh` on a machine that needs the helper and menu entry.

## Versioning and rollback

| Thing | Value | Where it comes from |
| --- | --- | --- |
| Image release tag | `v0.2.0` | Git tag, verbatim |
| Image commit tag | first 7 characters of `HEAD` | `git rev-parse --short HEAD` |
| Image `latest` | newest successful `master` build | `docker.yml` |
| APK `versionCode` | workflow run number | `gitea.run_number` |
| APK `versionName` | tag without `v`, or `0.1.0-master.<run_number>` | `Resolve the version name` step |
| APK release asset | `kadence-v0.2.0.apk` | Git tag |

To roll back the backend, redeploy the previous version tag or digest:

```bash
export KADENCE_IMAGE=git.timka.su/timka123/kadence:v0.1.3
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build
```

Check `/api/health` and the app log afterwards. If the newer release added a
column the old code does not understand, restore the database backup taken before
the upgrade; `create_all` does not undo schema changes.

To roll back the phone, install the previous release APK over the current one.
Android refuses a lower `versionCode` during a normal install; `adb install -r -d`
allows it on a device where downgrades are permitted, but that is not a reliable
production path. Keep the previous APK and the matching keystore, or bump the
version code and rebuild.

## Downloading artifacts without the web UI

The run page in Gitea is the simplest route: **Actions → run → Artifacts**.
For a tagged release, the attached asset is under **Releases**.

With a personal access token, the API works too. Gitea's artifacts API only
lists artifacts uploaded through the v4 artifact protocol, so the workflow uses
`https://gitea.com/actions/gitea-upload-artifact@v4`. Listing the artifacts of
the latest successful `android` run:

```bash
TOKEN=...  # a Gitea token with repository read access
REPO=https://git.timka.su/api/v1/repos/timka123/kadence

curl -fsSL -H "Authorization: token $TOKEN" \
  "$REPO/actions/artifacts?name=kadence-release-apk" \
  | jq -r '.artifacts[] | select(.expired == false) | "\(.id) \(.created_at)"' \
  | sort -k2 | tail -1
```

Then download and unpack the chosen artifact id:

```bash
ARTIFACT_ID=...
curl -fsSL -H "Authorization: token $TOKEN" \
  "$REPO/actions/artifacts/$ARTIFACT_ID/zip" -o kadence-release-apk.zip
unzip -o kadence-release-apk.zip
```

For a tagged build, the release asset can be fetched by tag instead:

```bash
curl -fsSL -H "Authorization: token $TOKEN" \
  "$REPO/releases/tags/v0.2.0" \
  | jq -r '.assets[] | select(.name | startswith("kadence-")) | .browser_download_url'
```

Artifacts from an older run can be expired; the release asset is the durable
copy.

## Troubleshooting

| Symptom | Likely cause |
| --- | --- |
| The run sits queued forever | No online runner claims `ubuntu-latest`, or the runner is registered at a scope that does not include this repository |
| `docker login` step warns that there is no `REGISTRY_TOKEN` | The secret is missing, empty, or lacks `package:write`; the image is built but not pushed |
| `Cannot connect to the Docker daemon` in `docker.yml` | The job container does not have the host socket; check `docker_host` and `valid_volumes` in `deploy/act-runner.yaml` |
| The Docker push returns 401 | `REGISTRY_TOKEN` is a job token or a PAT without `package:write`, or the username does not match the token |
| Android build fails before Gradle starts | The SDK install failed; check `android/ci/install-sdk.sh` and the cache key. A cold cache can take several minutes |
| `Failed to read key  from store ...: keystore password was incorrect` | The alias or a password secret is empty or wrong; the blank before `from` is the alias. All four `KADENCE_KEY*` secrets are required. Verify locally with `keytool -list` before re-encoding the keystore |
| The APK is debug-signed despite the keystore secrets | At least one of `KADENCE_KEYSTORE_BASE64`, `KADENCE_KEYSTORE_PASSWORD`, `KADENCE_KEY_ALIAS`, `KADENCE_KEY_PASSWORD` is missing or wrong. With `KADENCE_REQUIRE_SIGNING` the job should have failed; fix the secret rather than shipping it |
| `kadence-release-apk` is missing from the run | The build failed before the upload step, or the artifact has expired |
| The artifact API returns `total_count: 0` although the job uploaded the files | The workflow used `actions/upload-artifact@v3`; Gitea 1.24+ only exposes v4-protocol artifacts through `/actions/artifacts`. Use `https://gitea.com/actions/gitea-upload-artifact@v4` and run the build again |
| The tag build cannot attach the APK to a release | The tag was pushed but no release exists yet, or the built-in `GITEA_TOKEN` lacks permission. Create the release, or check the repository's Actions token permissions |
| A tag push did not start a workflow | The tag does not match `v*`, or it was created locally and never pushed: `git push origin v0.2.0` |
| The deployment still runs old code after a pull | `docker compose up` rebuilt the local checkout. Use `--no-build` and pin `KADENCE_IMAGE` |
| Android installs reject the new APK | It is signed with a different key, or has a lower version code than the installed one. Keep the release keystore; do not switch it between releases |

### Secrets are not in the build logs

The workflows never echo a token or a keystore password. The APK job does print
the certificate with `apksigner --print-certs`, which is the public certificate,
not the private key. Do not add a debug step that prints the environment, the
decoded keystore, or the registry token.