ci: the runner is docker-mode, and a guide for setting one up

`runs-on: ubuntu-latest` stays what it is everywhere else — a disposable
container — and `docker-compose.ci.yml` is now what makes that true: the label
maps to `catthehacker/ubuntu:act-24.04`, job containers share the runner's
network so the cache server is reachable at all, and the host's docker socket is
passed through (`docker_host: automount`) so the image workflow can build.

The failure this is replacing is worth naming, because the symptom does not
point at it: with a **host**-mode runner answering the same label, every job runs
directly on the server as the runner's own user, so the workflows' `apt-get
install` steps fail on permissions — and had they succeeded, CI would have been
installing packages on a host. A workflow cannot tell which kind of runner it
got; only the label map can say.

`docs/gitea-runner.md` is the separate guide: why docker mode, the prerequisites
including Gitea's two `app.ini` settings, the install, how to remove a host-mode
runner (and what to check before you do, since a runner is shared), how to
verify a job really is in a container, why labels cannot be changed without
re-registering, and a table of the failures with their causes.

Also `workflow_dispatch` on the test workflow, so a fresh runner can be proved
from the UI without inventing a commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitea/workflows/test.yml

@@ -13,6 +13,8 @@ on:
   push:
     branches: [master]
   pull_request:
+  # So a new runner can be proved out from the UI without inventing a commit.
+  workflow_dispatch:
 
 env:
   DEBIAN_FRONTEND: noninteractive

Makefile

@@ -73,7 +73,7 @@ android-sdk: ## Install the Android SDK packages this app builds against
 android-release: ## Build the release APK (set KADENCE_KEYSTORE* to sign it)
 	cd android && ./gradlew :app:assembleRelease
 
-ci-runner: ## Start the Gitea Actions runner (needs GITEA_RUNNER_REGISTRATION_TOKEN)
+ci-runner: ## Start the Gitea Actions runner here (see docs/gitea-runner.md)
 	docker compose -f docker-compose.ci.yml up -d
 
 report: ## Build and send today's report right now

README.md

@@ -275,20 +275,25 @@ Gitea Actions, in `.gitea/workflows/`:
 | `docker.yml` | push to master, `v*` tags | builds the image, starts it against SQLite and waits for `/api/health`, then pushes `git.timka.su/timka123/kadence:latest` and `:<short-sha>` |
 | `android-release.yml` | push to master, `v*` tags | `:app`'s suite (81, on the JVM), then the release APK — signed from secrets, with `apksigner verify` printing the certificate it actually used |
 
-**Nothing runs until a runner is registered.** `docker-compose.ci.yml` is the whole
+**Nothing runs until a docker-mode runner is registered** —
-of that, and its header is the setup checklist: a registration token from
+[`docs/gitea-runner.md`](docs/gitea-runner.md) is the guide, and
-*Settings → Actions → Runners*, then `make ci-runner`. Gitea itself needs
+`docker-compose.ci.yml` + `deploy/act-runner.yaml` are the two files it installs.
-`[actions] ENABLED = true` and `DEFAULT_ACTIONS_URL = github` in `app.ini`, the
+Short version, on the runner host:
-second so `actions/checkout@v4` can be resolved.
 
-Two details decide whether the jobs work at all:
+```bash
+mkdir -p /opt/gitea-runner && cd /opt/gitea-runner
+cp …/docker-compose.ci.yml docker-compose.yml && cp …/deploy/act-runner.yaml config.yaml
+printf 'GITEA_RUNNER_REGISTRATION_TOKEN=%s\n' "<token>" > .env && chmod 600 .env
+docker compose up -d
+```
 
-* **The label map.** `runs-on: ubuntu-latest` is pinned to
+`runs-on: ubuntu-latest` means what it means on GitHub: a disposable container,
-  `catthehacker/ubuntu:act-24.04` by the runner's own config, because the jobs need
+here `catthehacker/ubuntu:act-24.04` — chosen because the jobs need a `node` in the
-  two things from the image: a `node` (Gitea's javascript actions run *inside* the
+image (Gitea's javascript actions run *inside* the job container) and Python 3.12,
-  job container) and Python 3.12, which the backend's `requires-python` insists on.
+which the backend's `requires-python` insists on. A **host-mode** runner answering
-* **The docker socket**, mounted into job containers so the image can be built. It
+the same label is the one thing that breaks all of this: the job then runs on the
-  needs both the mount and `valid_volumes` naming it — see `deploy/act-runner.yaml`.
+machine as the runner's user, where `apt-get install` either fails or installs
+packages on your server. The guide covers removing one.
 
 Secrets, all under *Repository → Settings → Actions → Secrets*:
 

deploy/act-runner.yaml

@@ -1,26 +1,48 @@
-# act_runner's own configuration. See docker-compose.ci.yml for what it is for.
+# act_runner's own configuration, mounted into the container as
+# /config/config.yaml. See docker-compose.ci.yml and docs/gitea-runner.md.
 log:
   level: info
 
 runner:
+  # Two jobs at a time. The test workflow has four, so a push runs them in two
+  # waves on an 8-core box — raise it if the box is idle anyway.
   capacity: 2
   timeout: 1h
-  # Jobs pull their own tooling; nothing here needs to be fetched per job.
+  shutdown_timeout: 3h
-  fetch_timeout: 10s
+  insecure: false
-  fetch_interval: 5s
+  fetch_timeout: 5s
+  fetch_interval: 2s
+  # Deliberately empty: the labels come from GITEA_RUNNER_LABELS at registration
+  # and live in .runner afterwards. Setting them in both places is how they end
+  # up disagreeing.
+  labels: []
 
 cache:
-  # Gitea's built-in cache server is what makes actions/cache@v4 work, and the
+  # What makes `actions/cache@v4` work — the Android SDK download and the gradle
-  # Android SDK download is worth caching: it is most of that job's wall clock.
+  # caches are most of the APK job's wall clock. The server is run by the runner
+  # itself, inside the container, and reached by job containers over the shared
+  # `gitea-runner` network, so there is no port to publish or firewall.
   enabled: true
+  dir: ""
+  host: ""
+  port: 0
 
 container:
-  network: bridge
+  # The same network the runner is on. Not `bridge`: on the default bridge the
+  # job containers cannot reach the runner's cache server.
+  network: gitea-runner
   privileged: false
-  # The host's docker daemon, for the image build. Both lines are needed: the
+  options:
-  # mount itself, and the allow-list that permits it.
+  workdir_parent: workspace
-  options: -v /var/run/docker.sock:/var/run/docker.sock
+  # Mount the host's docker socket into job containers that ask for docker, so
+  # `docker.yml` can build and push. `valid_volumes` is the allow-list that
+  # makes any such mount legal at all.
+  docker_host: automount
   valid_volumes:
     - /var/run/docker.sock
-  # Let the job's own `docker` CLI find it.
+  force_pull: false
-  docker_host: unix:///var/run/docker.sock
+  force_rebuild: false
+
+host:
+  # Only used by `:host` labels, which this runner deliberately does not offer.
+  workdir_parent: /tmp/act_runner

docker-compose.ci.yml

@@ -1,48 +1,72 @@
-# The Gitea Actions runner, for a single-machine Gitea.
+# The Gitea Actions runner, in Docker.
 #
-# Nothing in .gitea/workflows runs until a runner is registered, and this is the
+# This is the runner that executes everything in `.gitea/workflows`. It claims
-# whole of it. Two settings do the work:
+# the label **`ubuntu-latest`** and maps it to a container image, which is what
+# the workflows expect: GitHub's own name, GitHub's own semantics — the job runs
+# in a disposable container, not on the machine.
 #
-#   * the **label map** decides what `runs-on: ubuntu-latest` means. It is pinned
+# It replaced a *host*-mode runner on the same box (`ubuntu-latest:host`), where
-#     to catthehacker/ubuntu:act-24.04 because the workflows need two things from
+# every job ran directly on the server as the `act_runner` user. That is why the
-#     the image: a node (Gitea's javascript actions — checkout, cache,
+# workflows' `apt-get install` steps failed: nothing could be installed, and had
-#     upload-artifact — run *inside* the job container) and Python 3.12, which
+# it succeeded it would have been installing packages on the host.
-#     the backend's `requires-python` insists on. Ubuntu 24.04 ships 3.12.
-#   * the **docker socket**, mounted into job containers so the `docker` workflow
-#     can build and push an image. That is a real privilege handed to CI; it is
-#     also the normal arrangement when Gitea and its runner are the same box as
-#     the deployment.
 #
-# Setting it up:
+# Setting it up, and what each piece is for, is in
+# [`docs/gitea-runner.md`](docs/gitea-runner.md). The short version:
 #
-#   1. Gitea → Settings → Actions → Runners → "Create new runner", copy the token.
+#     mkdir -p /opt/gitea-runner && cd /opt/gitea-runner
-#   2. GITEA_RUNNER_REGISTRATION_TOKEN=<token> docker compose -f docker-compose.ci.yml up -d
+#     # copy this file as docker-compose.yml and deploy/act-runner.yaml as config.yaml
-#   3. Repository → Settings → Actions → Secrets, add:
+#     printf 'GITEA_RUNNER_REGISTRATION_TOKEN=%s\n' "<token from Gitea>" > .env
-#        REGISTRY_USER, REGISTRY_TOKEN            — a Gitea user and token with
+#     chmod 600 .env
-#                                                   package:write, for the registry
+#     docker compose up -d
-#        KADENCE_KEYSTORE_BASE64                  — base64 -w0 of your keystore
-#        KADENCE_KEYSTORE_PASSWORD, KADENCE_KEY_ALIAS, KADENCE_KEY_PASSWORD
-#      Without the keystore secrets the APK job still runs and produces an
-#      unsigned-for-release (debug-signed) artifact rather than failing.
 #
-# Gitea itself needs `[actions] ENABLED = true` in app.ini, and
+# The registration token is needed **once**: after the first start the runner has
-# `DEFAULT_ACTIONS_URL = github` so `actions/checkout@v4` can be resolved.
+# its own permanent token in `./data/.runner` and the one in `.env` is dead.
 services:
   runner:
-    image: gitea/act_runner:0.2.11
+    image: gitea/act_runner:0.3.1
+    container_name: gitea-runner
     restart: unless-stopped
+
     environment:
       GITEA_INSTANCE_URL: ${GITEA_INSTANCE_URL:-https://git.timka.su}
-      GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN:?set this to the token from Gitea}
+      # Only read at first registration; harmless (and ignored) afterwards.
-      GITEA_RUNNER_NAME: ${GITEA_RUNNER_NAME:-kadence-runner}
+      GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN:-}
-      GITEA_RUNNER_LABELS: ubuntu-latest:docker://catthehacker/ubuntu:act-24.04
+      GITEA_RUNNER_NAME: ${GITEA_RUNNER_NAME:-dockerized}
-      # Mounted into every job container, which is what lets the docker workflow
+      # The label map: what `runs-on` may ask for, and what it gets.
-      # build an image. `valid_volumes` has to allow it explicitly.
+      #
+      # `act-24.04` and not something slimmer because the jobs need two things
+      # from the image: a **node**, since Gitea's javascript actions (checkout,
+      # cache, upload-artifact) run *inside* the job container, and **Python
+      # 3.12**, which the backend's `requires-python` insists on and Ubuntu
+      # 24.04 ships. A plain `ubuntu:24.04` has neither and every job would
+      # fail at the first `uses:`.
+      #
+      # Labels are baked into `./data/.runner` at registration — changing this
+      # line later does nothing on its own. See the guide.
+      GITEA_RUNNER_LABELS: >-
+        ubuntu-latest:docker://catthehacker/ubuntu:act-24.04,
+        ubuntu-24.04:docker://catthehacker/ubuntu:act-24.04,
+        ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04
       CONFIG_FILE: /config/config.yaml
-    volumes:
-      - runner-data:/data
-      - ./deploy/act-runner.yaml:/config/config.yaml:ro
-      - /var/run/docker.sock:/var/run/docker.sock
 
     volumes:
-  runner-data:
+      - ./config.yaml:/config/config.yaml:ro
+      # The registration lives here. Delete `./data/.runner` and the runner
+      # re-registers as a *new* one, leaving a stale entry in Gitea.
+      - ./data:/data
+      # The host's docker daemon, so a job can build and push an image. This is
+      # a real privilege — root on the host, effectively — and the reason the
+      # runner should not be registered at instance level on a box you do not
+      # want CI to own. `config.yaml` passes it through to job containers.
+      - /var/run/docker.sock:/var/run/docker.sock
+
+    networks:
+      - runner
+
+networks:
+  # Job containers join this same network (`container.network` in config.yaml),
+  # which is what makes the runner's built-in cache server reachable from them:
+  # it advertises its own address, and on a separate network that address would
+  # route nowhere. It is also why nothing here needs a published port.
+  runner:
+    name: gitea-runner

docs/gitea-runner.md

@@ -0,0 +1,187 @@
+# A dockerized Gitea Actions runner
+
+Everything in `.gitea/workflows` needs a runner, and it has to be a **docker-mode**
+one. This is how to set one up, what each setting is for, and what goes wrong if
+it is set up the other way.
+
+* [Why docker mode and not host mode](#why-docker-mode-and-not-host-mode)
+* [Prerequisites](#prerequisites)
+* [Install](#install)
+* [Removing a host-mode runner](#removing-a-host-mode-runner)
+* [Verifying it](#verifying-it)
+* [Changing the labels later](#changing-the-labels-later)
+* [What can go wrong](#what-can-go-wrong)
+
+---
+
+## Why docker mode and not host mode
+
+A Gitea runner advertises **labels**, and each label says *where* a job runs:
+
+| Label | Means |
+| --- | --- |
+| `ubuntu-latest:docker://catthehacker/ubuntu:act-24.04` | run the job in a fresh container from that image |
+| `ubuntu-latest:host` | run the job **directly on the machine**, as the runner's own user |
+
+A workflow only says `runs-on: ubuntu-latest`. It cannot tell which of those it
+got, and the two behave nothing alike:
+
+* **Host mode has no job isolation and no job toolchain.** `apt-get install` either
+  fails (the runner user is not root — the common case) or succeeds, and then you
+  have CI installing packages on a server. `actions/checkout` clones into a temp
+  directory that persists between runs, so one job's leftovers are the next job's
+  surprise. Every workflow written against GitHub's runners assumes a clean
+  container and does not get one.
+* **Docker mode is what `ubuntu-latest` means everywhere else.** A disposable
+  container per job, the tools the image ships, and nothing left behind.
+
+Host mode is for the narrow case where a job *must* touch the machine — a deploy
+script using that host's ssh keys, say. Everything else wants docker mode.
+
+**One label, one meaning.** If two runners both claim `ubuntu-latest`, jobs land on
+whichever answers first, and a workflow that passes on one fails on the other for
+reasons nobody can see from the logs. Pick one, or give the special-purpose runner
+a label of its own (`deploy:host`) and leave `ubuntu-latest` to the docker one.
+
+## Prerequisites
+
+* Docker and the compose plugin on the host (`docker --version`, `docker compose version`).
+* Gitea with Actions on. In `app.ini`:
+
+  ```ini
+  [actions]
+  ENABLED = true
+  ; so `uses: actions/checkout@v4` can be resolved from GitHub
+  DEFAULT_ACTIONS_URL = github
+  ```
+
+* Disk. The job image is ~1–2 GB; the Android SDK cache another ~3 GB, and gradle
+  caches grow. Budget 20 GB and it will not surprise you.
+* A **registration token**, from whichever scope the runner should serve:
+
+  | Scope | Where | Serves |
+  | --- | --- | --- |
+  | Repository | repo → *Settings → Actions → Runners → Create new runner* | that repo only |
+  | Organisation | org → *Settings → Actions → Runners* | every repo in the org |
+  | Instance | *Site Administration → Actions → Runners* | everything on the instance |
+
+  Prefer the narrowest scope that covers what you need — the runner's docker socket
+  is effectively root on its host, and an instance-level runner hands that to every
+  repository on the server.
+
+## Install
+
+Two files, both in this repo: `docker-compose.ci.yml` (the service) and
+`deploy/act-runner.yaml` (act_runner's own config). On the runner host:
+
+```bash
+mkdir -p /opt/gitea-runner && cd /opt/gitea-runner
+
+# From a checkout, or scp them across:
+cp /path/to/kadence/docker-compose.ci.yml docker-compose.yml
+cp /path/to/kadence/deploy/act-runner.yaml config.yaml
+
+# The token is needed once, at registration. Keep it out of the compose file.
+printf 'GITEA_INSTANCE_URL=https://git.timka.su\n' > .env
+printf 'GITEA_RUNNER_REGISTRATION_TOKEN=%s\n' "<paste the token>" >> .env
+printf 'GITEA_RUNNER_NAME=%s\n' "$(hostname)-docker" >> .env
+chmod 600 .env
+
+docker compose up -d
+docker compose logs -f           # expect: "runner registered successfully"
+```
+
+That is it. The runner appears in Gitea under the scope you took the token from,
+with its labels listed beside it.
+
+### What the pieces do
+
+* **`GITEA_RUNNER_LABELS`** is the label map, and it is read **only at first
+  registration** — see [Changing the labels later](#changing-the-labels-later).
+* **`./data`** holds `.runner`: the runner's identity and its permanent token.
+  Back it up or do not delete it; removing it makes the runner register again as a
+  *new* one and leaves a dead entry behind in Gitea.
+* **`/var/run/docker.sock`** is mounted so a job can build images. With
+  `docker_host: automount` in `config.yaml`, act_runner passes it into the job
+  container too, which is what lets `docker build` work inside a workflow. Treat it
+  as what it is: any workflow on this runner can do anything to this host's docker.
+* **The shared `gitea-runner` network.** Job containers join the runner's own
+  network (`container.network`), because act_runner's built-in cache server — the
+  thing that makes `actions/cache@v4` work — advertises its own address, and from
+  the default bridge that address routes nowhere. Symptom if you get it wrong:
+  cache steps hang, then time out.
+* **`catthehacker/ubuntu:act-24.04`** rather than `ubuntu:24.04`, because Gitea's
+  javascript actions (`checkout`, `cache`, `upload-artifact`) execute **inside the
+  job container** and need a `node` there. This image also ships Python 3.12, which
+  Kadence's backend requires.
+
+## Removing a host-mode runner
+
+If the box already has one (the usual sign: jobs fail at `apt-get` with permission
+errors, or succeed and change the host):
+
+```bash
+systemctl disable --now act_runner.service
+mv /var/lib/act_runner/.runner /var/lib/act_runner/.runner.disabled   # reversible
+rm -f /etc/systemd/system/act_runner.service && systemctl daemon-reload
+rm -f /usr/local/bin/act_runner
+```
+
+Then delete its (now offline) entry in Gitea, or it sits there looking available.
+
+**Check what else used it first.** A runner is shared: look at its recent tasks
+(`journalctl -u act_runner | grep "task .* repo is"`) and at the other repos'
+workflows. Jobs that ran on the host because they *needed* the host — deploying
+with that machine's ssh keys, writing to a host path — will not work in a
+container as they are. The docker socket is mounted, so `docker compose` steps
+generally still do.
+
+## Verifying it
+
+```bash
+docker compose logs --tail 20          # "runner registered" / "daemon started"
+```
+
+In Gitea: the runner is listed, online, with its labels. Then run something — a
+push, or the *Run workflow* button if the workflow has `workflow_dispatch`. The
+first job pulls the image, so give it a few minutes; later ones start in seconds.
+
+Job logs should show the job running *as root inside a container*:
+
+```
+whoami        -> root
+cat /etc/os-release -> Ubuntu 24.04
+```
+
+If it says your runner's user and the host's distribution, the job landed on a
+host-mode runner and the label map is wrong.
+
+## Changing the labels later
+
+Labels live in `./data/.runner` after registration; the environment variable is
+only read when that file does not exist. To change them, re-register:
+
+```bash
+cd /opt/gitea-runner
+docker compose down
+rm data/.runner                        # the old registration goes stale in Gitea
+# get a fresh token, put it in .env, adjust GITEA_RUNNER_LABELS
+docker compose up -d
+```
+
+Then delete the stale runner in Gitea. (Editing `.runner`'s `labels` array by hand
+works too, but Gitea keeps its own copy, so the UI and the runner can disagree —
+re-registering is the honest route.)
+
+## What can go wrong
+
+| Symptom | Cause |
+| --- | --- |
+| Jobs stay queued forever | No runner claims that `runs-on` label, or the runner is registered at a scope that does not include this repo |
+| `Cannot find: node in PATH` | A job image without node — use a `catthehacker/*` image, not bare `ubuntu` |
+| `apt-get: Permission denied` | The job is on a **host**-mode runner |
+| `uses: actions/checkout@v4` fails to resolve | `DEFAULT_ACTIONS_URL` is not `github` |
+| Cache steps hang | Job containers are not on the runner's network — see `container.network` |
+| `docker: command not found` in a job | That image has no docker CLI; use the `catthehacker` one, which does |
+| `Cannot connect to the Docker daemon` in a job | `docker_host: automount` missing from `config.yaml`, or the socket is not mounted into the runner |
+| Everything is slow on the first run | It is pulling the job image. It is cached afterwards |