kadence/docs/production-builds.md

Production builds in Gitea Actions

This is the practical guide to turning a commit into something that can be deployed or installed: the backend + web image, the signed Android APK, and what to do about the desktop agent. It assumes the runner from gitea-runner.md is already registered and online.

The short version:

• a push to master produces a snapshot — latest plus the short commit SHA;
• a v* tag produces a release — the same image tagged with the version, and an APK attached to the Gitea release.

A tag is the only thing that means "production". latest moves with every push and should never be pinned in a production compose file.

• What the workflows produce
• One-time setup
• Cutting a release
• The backend + web image
• The Android APK
• Over-the-air updates
• The desktop agent
• Versioning and rollback
• Downloading artifacts without the web UI
• Troubleshooting

---

What the workflows produce

Workflow	Trigger	Artifact
test.yml	push to master, every pull request	No release artifact. It is the gate: backend, desktop, :core, and the web typecheck + build
lint.yml	push to master, every pull request	No release artifact. ruff check over backend/ and desktop/
docker.yml	push to master, v* tags	Image git.timka.su/timka123/kadence:<short-sha> and :latest; on a tag also :<version>
android-release.yml	push to master, v* tags	APK artifact kadence-release-apk and metadata kadence-update-metadata; on a tag also the release asset kadence-<version>.apk

There is no build job for the desktop agent. See The desktop agent for what that means in practice.

One-time setup

The runner

Use the docker-mode runner from gitea-runner.md. The Android job needs a real Android SDK and Gradle cache, and the Docker job needs access to the host's Docker daemon through the mounted socket. A host-mode runner answering ubuntu-latest is the most common reason a production build behaves differently from the one in the docs.

Secrets

All secrets live under Repository → Settings → Actions → Secrets.

Secret	Used by	What it must be
REGISTRY_TOKEN	docker.yml	A personal access token with package:write. Gitea's built-in job token cannot publish packages
REGISTRY_USER	docker.yml	Optional. The registry username paired with the token; defaults to the repository owner
KADENCE_KEYSTORE_BASE64	android-release.yml	base64 -w0 your-release.keystore
KADENCE_KEYSTORE_PASSWORD	android-release.yml	The store password
KADENCE_KEY_ALIAS	android-release.yml	The key alias, normally kadence
KADENCE_KEY_PASSWORD	android-release.yml	The key password

All four KADENCE_KEY* secrets are needed for a release. Before uploading the base64 blob, check that the alias and passwords actually open the file:

keytool -list \
  -keystore your-release.keystore \
  -storepass "$KADENCE_KEYSTORE_PASSWORD" \
  -alias "$KADENCE_KEY_ALIAS"

Then produce the secret with base64 -w0 your-release.keystore. The workflow fails early and names any missing signing secret, instead of waiting for packageRelease to report a keytool error.

The workflow uses Gitea's built-in GITEA_TOKEN to attach the APK to a tagged release. That token is supplied by Actions; it is not one of the secrets above.

Without REGISTRY_TOKEN the Docker workflow still builds and smoke-tests the image, then skips the push with a warning. That is useful for a test repository, not for production.

Without the keystore secrets the Android job still builds an APK, but it is debug-signed. It installs on your own phone and cannot be upgraded by a properly signed release later. For production, set all four keystore values: KADENCE_REQUIRE_SIGNING then makes a missing or misnamed secret fail the build instead of silently falling back to the debug key.

The keystore

If you do not already have one, create it once and keep it somewhere safe — in a password manager or another encrypted backup, never in the repository:

keytool -genkeypair -v \
  -keystore kadence-release.keystore \
  -alias kadence \
  -keyalg RSA -keysize 4096 -validity 10000 \
  -dname "CN=Kadence, O=Kadence"

base64 -w0 kadence-release.keystore > kadence-release.keystore.b64

Put the contents of kadence-release.keystore.b64 into KADENCE_KEYSTORE_BASE64. Losing this file means future APKs cannot upgrade the installed app; Android treats them as a different application.

The tag convention

Use annotated v tags with a semantic version:

git tag -a v0.2.0 -m "Kadence 0.2.0"
git push origin v0.2.0

v0.2.0, not 0.2.0. Both publishing workflows filter on v*, and the Docker workflow uses the tag as the image tag verbatim.

Cutting a release

1. Start from a green master. The release workflows do not wait for the test workflow, so check that the latest master run is green before tagging.

   git checkout master
   git pull --ff-only
   git status --short
   

2. Tag the commit you mean to ship. The tag is what the workflows check out; an unpushed commit is not part of the build.

   git tag -a v0.2.0 -m "Kadence 0.2.0"
   git push origin v0.2.0
   

3. Watch both workflows. In Gitea: Actions → docker and Actions → android. The Android job is the slower one on a cold cache because it installs the SDK and fetches the Glyph Matrix AAR.

4. Verify the image. It should exist under both the version and the commit:

   docker pull git.timka.su/timka123/kadence:v0.2.0
   docker image inspect git.timka.su/timka123/kadence:v0.2.0 \
     --format '{{index .RepoDigests 0}}'
   

5. Verify the APK. Download it from the release page (https://git.timka.su/timka123/kadence/releases/tag/v0.2.0) and check the certificate before putting it on a phone:

   apksigner verify --print-certs kadence-v0.2.0.apk
   

   The certificate must be the release key, not CN=Android Debug.

6. Deploy or install. The next two sections cover each artifact.

The backend + web image

What the job does

docker.yml builds the single image from the repository Dockerfile: Vite builds the web app, then the Python 3.12 runtime serves both the API and the built static files. The job then starts the image against SQLite with the scheduler and Telegram bot disabled and waits for /api/health before it pushes anything. A tag that cannot boot is not published.

The pushed names are:

Tag	Meaning
:<short-sha>	The exact commit, for every build
:latest	The newest successful build from master or a tag; moves on every push
:vX.Y.Z	A tagged release

Deploying the image

The repository's deploy/deploy.sh currently builds on the server from the checked-out branch. That is the simplest path, and it means the deployment does not need registry credentials.

To deploy the image that CI actually built instead, pin it in the environment:

ssh sw_pl_1
cd /opt/kadence

# Only needed if the registry package is private.
docker login git.timka.su

export KADENCE_IMAGE=git.timka.su/timka123/kadence:v0.2.0
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build

curl -fsS https://kadence.timka.su/api/health

--no-build is deliberate: without it, up may build the local checkout and quietly ignore the image you just pulled. The production overlay keeps the app on 127.0.0.1 and leaves TLS to nginx; see docker-compose.prod.yml and deploy/nginx/kadence.timka.su.conf.

For a fully pinned deployment, use the digest from step 4 instead of the version tag:

export KADENCE_IMAGE=git.timka.su/timka123/kadence@sha256:<digest>

Before you roll a release

Take a database backup. The backend uses create_all rather than a migration tool, so it adds tables and columns but never removes or rewrites them. Rolling an old application back against a newer schema is only safe if the change was additive:

cd /opt/kadence
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T db \
  sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
  > "backup-$(date +%F-%H%M).sql"

The Android APK

What the job does

android-release.yml:

1. checks out the tag or commit;
2. installs JDK 21 and downloads the Glyph Matrix AAR from Nothing;
3. installs the Android SDK (platforms;android-37.0, build tools, platform tools) and restores the Gradle caches;
4. runs :app's JVM/Robolectric tests;
5. builds assembleRelease with the version code from the workflow run number;
6. verifies the signing certificate with apksigner and rejects a debug key;
7. writes update.json (version, commit, size, SHA-256 and signer digest);
8. uploads kadence-release-apk and kadence-update-metadata, and on a v* tag attaches the APK to the Gitea release as kadence-<version>.apk.

The build is signed from the keystore secrets. KADENCE_REQUIRE_SIGNING is set when a keystore was found, so a missing password or wrong alias fails the job rather than producing a debug-signed "release".

Version code and version name

versionCode comes from ${{ gitea.run_number }}, so successive CI builds can install over each other. That is the number Android compares when updating.

versionName is resolved before the build:

Trigger	versionName
Tag v0.2.0	0.2.0 (the leading v is dropped)
Push to master	0.1.0-master.<run_number>

So a tagged release reports its tag in Settings, while a master build is identifiable as a CI build and still has a unique, monotonic versionCode.

Installing the artifact

From the Actions run page, download kadence-release-apk.zip, unpack it, and install the app-release.apk inside:

adb install -r app-release.apk

Or use the release asset on a tagged build:

curl -fL -o kadence-v0.2.0.apk \
  https://git.timka.su/timka123/kadence/releases/download/v0.2.0/kadence-v0.2.0.apk
adb install -r kadence-v0.2.0.apk

The phone must already have the app installed with the same signing key. The debug build uses a different application id (.debug) and can sit beside the release build, but it cannot be upgraded into it.

Over-the-air updates

The Android app can update itself from these artifacts, without the phone ever holding a Gitea credential. The repository is private, so the backend reads the newest successful build from GITEA_UPDATE_BRANCH and re-serves it behind the same /api auth as everything else.

Configure the backend (compose passes these through from .env):

Variable	What for
GITEA_TOKEN	read-only token that can see this repository's Actions artifacts. Empty disables OTA: the endpoints answer 503 and the app never offers an update
GITEA_REPO	owner/repository, default timka123/kadence
GITEA_URL	Gitea base URL, default https://git.timka.su
GITEA_UPDATE_BRANCH	branch whose successful builds are offered, default master

The token only needs repository read access; it is never sent to the phone. The two endpoints are:

• GET /api/updates/android — update.json for the newest successful build, with run_number filled in from Gitea even for older artifacts.
• GET /api/updates/android/apk — the APK itself, streamed with Cache-Control: no-store.

A missing token is 503, no usable artifact is 404, and a Gitea failure is 502. The app treats background failures quietly; Settings → Updates shows the reason.

On the phone:

• checks run on resume, at most once every six hours, in release builds only, and only after onboarding;
• only a strictly higher version_code is offered, and a skipped build stays skipped until Check now in Settings;
• the APK is downloaded only after the user asks, into the app cache, with a .part file so an interrupted transfer cannot look installable;
• before handing it to Android, the app checks the package name, the version code and that the APK's signing certificate matches the installed app. A mismatch is deleted and reported instead of producing "App not installed";
• Android asks for "install unknown apps" permission the first time, then its own installer confirms the upgrade.

This is why the release keystore matters: an APK signed with a different key cannot upgrade an install, and the app refuses it before downloading. Debug builds have OTA disabled entirely.

Artifact retention

Gitea's artifact retention is finite (the instance's configured default, often 90 days). The release asset attached to a tag is not subject to the same retention as a workflow artifact; that is why tagged builds are the production path and plain master artifacts are only a convenience.

The desktop agent

There is no desktop build in .gitea/workflows today. A production desktop "build" is still the checkout plus the installer script:

git checkout v0.2.0
./desktop/packaging/install-desktop.sh

The app is plain Python with PyQt6, run by the system interpreter. The launcher installs a .pth file pointing at the checkout, so upgrading means checking out the new tag and re-running the installer. The privileged helper is separate and needs its own install:

sudo ./desktop/helper/install.sh

If you need a downloadable desktop artifact, build a wheel from the same tag:

cd desktop
python3 -m venv /tmp/kadence-build
/tmp/kadence-build/bin/pip install build
/tmp/kadence-build/bin/python -m build
# dist/kadence_desktop-0.2.0-py3-none-any.whl

That wheel contains the Python package and QML files, not the privileged helper or the Qt system dependency. It is not a replacement for install-desktop.sh on a machine that needs the helper and menu entry.

Versioning and rollback

Thing	Value	Where it comes from
Image release tag	v0.2.0	Git tag, verbatim
Image commit tag	first 7 characters of HEAD	git rev-parse --short HEAD
Image latest	newest successful master build	docker.yml
APK versionCode	workflow run number	gitea.run_number
APK versionName	tag without v, or 0.1.0-master.<run_number>	Resolve the version name step
APK release asset	kadence-v0.2.0.apk	Git tag

To roll back the backend, redeploy the previous version tag or digest:

export KADENCE_IMAGE=git.timka.su/timka123/kadence:v0.1.3
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build

Check /api/health and the app log afterwards. If the newer release added a column the old code does not understand, restore the database backup taken before the upgrade; create_all does not undo schema changes.

To roll back the phone, install the previous release APK over the current one. Android refuses a lower versionCode during a normal install; adb install -r -d allows it on a device where downgrades are permitted, but that is not a reliable production path. Keep the previous APK and the matching keystore, or bump the version code and rebuild.

Downloading artifacts without the web UI

The run page in Gitea is the simplest route: Actions → run → Artifacts. For a tagged release, the attached asset is under Releases.

With a personal access token, the API works too. Gitea's artifacts API only lists artifacts uploaded through the v4 artifact protocol, so the workflow uses https://gitea.com/actions/gitea-upload-artifact@v4. Listing the artifacts of the latest successful android run:

TOKEN=...  # a Gitea token with repository read access
REPO=https://git.timka.su/api/v1/repos/timka123/kadence

curl -fsSL -H "Authorization: token $TOKEN" \
  "$REPO/actions/artifacts?name=kadence-release-apk" \
  | jq -r '.artifacts[] | select(.expired == false) | "\(.id) \(.created_at)"' \
  | sort -k2 | tail -1

Then download and unpack the chosen artifact id:

ARTIFACT_ID=...
curl -fsSL -H "Authorization: token $TOKEN" \
  "$REPO/actions/artifacts/$ARTIFACT_ID/zip" -o kadence-release-apk.zip
unzip -o kadence-release-apk.zip

For a tagged build, the release asset can be fetched by tag instead:

curl -fsSL -H "Authorization: token $TOKEN" \
  "$REPO/releases/tags/v0.2.0" \
  | jq -r '.assets[] | select(.name | startswith("kadence-")) | .browser_download_url'

Artifacts from an older run can be expired; the release asset is the durable copy.

Troubleshooting

Symptom	Likely cause
The run sits queued forever	No online runner claims ubuntu-latest, or the runner is registered at a scope that does not include this repository
docker login step warns that there is no REGISTRY_TOKEN	The secret is missing, empty, or lacks package:write; the image is built but not pushed
Cannot connect to the Docker daemon in docker.yml	The job container does not have the host socket; check docker_host and valid_volumes in deploy/act-runner.yaml
The Docker push returns 401	REGISTRY_TOKEN is a job token or a PAT without package:write, or the username does not match the token
Android build fails before Gradle starts	The SDK install failed; check android/ci/install-sdk.sh and the cache key. A cold cache can take several minutes
Failed to read key from store ...: keystore password was incorrect	The alias or a password secret is empty or wrong; the blank before from is the alias. All four KADENCE_KEY* secrets are required. Verify locally with keytool -list before re-encoding the keystore
The APK is debug-signed despite the keystore secrets	At least one of KADENCE_KEYSTORE_BASE64, KADENCE_KEYSTORE_PASSWORD, KADENCE_KEY_ALIAS, KADENCE_KEY_PASSWORD is missing or wrong. With KADENCE_REQUIRE_SIGNING the job should have failed; fix the secret rather than shipping it
kadence-release-apk is missing from the run	The build failed before the upload step, or the artifact has expired
The artifact API returns total_count: 0 although the job uploaded the files	The workflow used actions/upload-artifact@v3; Gitea 1.24+ only exposes v4-protocol artifacts through /actions/artifacts. Use https://gitea.com/actions/gitea-upload-artifact@v4 and run the build again
The tag build cannot attach the APK to a release	The tag was pushed but no release exists yet, or the built-in GITEA_TOKEN lacks permission. Create the release, or check the repository's Actions token permissions
A tag push did not start a workflow	The tag does not match v*, or it was created locally and never pushed: git push origin v0.2.0
The deployment still runs old code after a pull	docker compose up rebuilt the local checkout. Use --no-build and pin KADENCE_IMAGE
Android installs reject the new APK	It is signed with a different key, or has a lower version code than the installed one. Keep the release keystore; do not switch it between releases

Secrets are not in the build logs

The workflows never echo a token or a keystore password. The APK job does print the certificate with apksigner --print-certs, which is the public certificate, not the private key. Do not add a debug step that prints the environment, the decoded keystore, or the registry token.