kadence/docs/gitea-runner.md

# A dockerized Gitea Actions runner

Everything in `.gitea/workflows` needs a runner, and it has to be a **docker-mode**
one. This is how to set one up, what each setting is for, and what goes wrong if
it is set up the other way.

* [Why docker mode and not host mode](#why-docker-mode-and-not-host-mode)
* [Prerequisites](#prerequisites)
* [Install](#install)
* [Removing a host-mode runner](#removing-a-host-mode-runner)
* [Verifying it](#verifying-it)
* [Changing the labels later](#changing-the-labels-later)
* [What can go wrong](#what-can-go-wrong)

---

## Why docker mode and not host mode

A Gitea runner advertises **labels**, and each label says *where* a job runs:

| Label | Means |
| --- | --- |
| `ubuntu-latest:docker://catthehacker/ubuntu:act-24.04` | run the job in a fresh container from that image |
| `ubuntu-latest:host` | run the job **directly on the machine**, as the runner's own user |

A workflow only says `runs-on: ubuntu-latest`. It cannot tell which of those it
got, and the two behave nothing alike:

* **Host mode has no job isolation and no job toolchain.** `apt-get install` either
  fails (the runner user is not root — the common case) or succeeds, and then you
  have CI installing packages on a server. `actions/checkout` clones into a temp
  directory that persists between runs, so one job's leftovers are the next job's
  surprise. Every workflow written against GitHub's runners assumes a clean
  container and does not get one.
* **Docker mode is what `ubuntu-latest` means everywhere else.** A disposable
  container per job, the tools the image ships, and nothing left behind.

Host mode is for the narrow case where a job *must* touch the machine — a deploy
script using that host's ssh keys, say. Everything else wants docker mode.

**One label, one meaning.** If two runners both claim `ubuntu-latest`, jobs land on
whichever answers first, and a workflow that passes on one fails on the other for
reasons nobody can see from the logs. Pick one, or give the special-purpose runner
a label of its own (`deploy:host`) and leave `ubuntu-latest` to the docker one.

## Prerequisites

* Docker and the compose plugin on the host (`docker --version`, `docker compose version`).
* Gitea with Actions on. In `app.ini`:

  ```ini
  [actions]
  ENABLED = true
  ; so `uses: actions/checkout@v4` can be resolved from GitHub
  DEFAULT_ACTIONS_URL = github
  ```

* Disk. The job image is ~1–2 GB; the Android SDK cache another ~3 GB, and gradle
  caches grow. Budget 20 GB and it will not surprise you.
* A **registration token**, from whichever scope the runner should serve:

  | Scope | Where | Serves |
  | --- | --- | --- |
  | Repository | repo → *Settings → Actions → Runners → Create new runner* | that repo only |
  | Organisation | org → *Settings → Actions → Runners* | every repo in the org |
  | Instance | *Site Administration → Actions → Runners* | everything on the instance |

  Prefer the narrowest scope that covers what you need — the runner's docker socket
  is effectively root on its host, and an instance-level runner hands that to every
  repository on the server.

## Install

Two files, both in this repo: `docker-compose.ci.yml` (the service) and
`deploy/act-runner.yaml` (act_runner's own config). On the runner host:

```bash
mkdir -p /opt/gitea-runner && cd /opt/gitea-runner

# From a checkout, or scp them across:
cp /path/to/kadence/docker-compose.ci.yml docker-compose.yml
cp /path/to/kadence/deploy/act-runner.yaml config.yaml

# The token is needed once, at registration. Keep it out of the compose file.
printf 'GITEA_INSTANCE_URL=https://git.timka.su\n' > .env
printf 'GITEA_RUNNER_REGISTRATION_TOKEN=%s\n' "<paste the token>" >> .env
printf 'GITEA_RUNNER_NAME=%s\n' "$(hostname)-docker" >> .env
chmod 600 .env

docker compose up -d
docker compose logs -f           # expect: "runner registered successfully"
```

That is it. The runner appears in Gitea under the scope you took the token from,
with its labels listed beside it. For the release process that uses this runner —
tags, secrets, artifacts, deployment and rollback — see
[`production-builds.md`](production-builds.md).

### What the pieces do

* **`GITEA_RUNNER_LABELS`** is the label map, and it is read **only at first
  registration** — see [Changing the labels later](#changing-the-labels-later).
* **`./data`** holds `.runner`: the runner's identity and its permanent token.
  Back it up or do not delete it; removing it makes the runner register again as a
  *new* one and leaves a dead entry behind in Gitea.
* **`/var/run/docker.sock`** is mounted so a job can build images. act_runner
  passes it into the job container too — on 0.3.x that is what an **empty**
  `docker_host` in `config.yaml` means, not "disabled". Treat the result as what it
  is: any workflow on this runner can do anything to this host's docker.

  Watch the spelling here. `docker_host: "-"` finds a daemon for the runner but
  does *not* give it to jobs, and `docker_host: automount` — the same default
  spelled out, from a later act_runner — makes 0.3.1 exit with `unable to parse
  docker host automount` and restart-loop, registered but never online.
* **The shared `gitea-runner` network.** Job containers join the runner's own
  network (`container.network`), because act_runner's built-in cache server — the
  thing that makes `actions/cache@v4` work — advertises its own address, and from
  the default bridge that address routes nowhere. Symptom if you get it wrong:
  cache steps hang, then time out.
* **`catthehacker/ubuntu:act-24.04`** rather than `ubuntu:24.04`, because Gitea's
  javascript actions (`checkout`, `cache`, `upload-artifact`) execute **inside the
  job container** and need a `node` there. This image also ships Python 3.12, which
  Kadence's backend requires.

## Removing a host-mode runner

If the box already has one (the usual sign: jobs fail at `apt-get` with permission
errors, or succeed and change the host):

```bash
systemctl disable --now act_runner.service
mv /var/lib/act_runner/.runner /var/lib/act_runner/.runner.disabled   # reversible
rm -f /etc/systemd/system/act_runner.service && systemctl daemon-reload
rm -f /usr/local/bin/act_runner
```

Then delete its (now offline) entry in Gitea, or it sits there looking available.

**Check what else used it first.** A runner is shared: look at its recent tasks
(`journalctl -u act_runner | grep "task .* repo is"`) and at the other repos'
workflows. Jobs that ran on the host because they *needed* the host — deploying
with that machine's ssh keys, writing to a host path — will not work in a
container as they are. The docker socket is mounted, so `docker compose` steps
generally still do.

## Verifying it

```bash
docker compose logs --tail 20          # "runner registered" / "daemon started"
```

In Gitea: the runner is listed, online, with its labels. Then run something — a
push, or the *Run workflow* button if the workflow has `workflow_dispatch`. The
first job pulls the image, so give it a few minutes; later ones start in seconds.

Job logs should show the job running *as root inside a container*:

```
whoami        -> root
cat /etc/os-release -> Ubuntu 24.04
```

If it says your runner's user and the host's distribution, the job landed on a
host-mode runner and the label map is wrong.

## Changing the labels later

Labels live in `./data/.runner` after registration; the environment variable is
only read when that file does not exist. To change them, re-register:

```bash
cd /opt/gitea-runner
docker compose down
rm data/.runner                        # the old registration goes stale in Gitea
# get a fresh token, put it in .env, adjust GITEA_RUNNER_LABELS
docker compose up -d
```

Then delete the stale runner in Gitea. (Editing `.runner`'s `labels` array by hand
works too, but Gitea keeps its own copy, so the UI and the runner can disagree —
re-registering is the honest route.)

## What can go wrong

| Symptom | Cause |
| --- | --- |
| Jobs stay queued forever | No runner claims that `runs-on` label, or the runner is registered at a scope that does not include this repo |
| `Cannot find: node in PATH` | A job image without node — use a `catthehacker/*` image, not bare `ubuntu` |
| `apt-get: Permission denied` | The job is on a **host**-mode runner |
| `uses: actions/checkout@v4` fails to resolve | `DEFAULT_ACTIONS_URL` is not `github` |
| Cache steps hang | Job containers are not on the runner's network — see `container.network` |
| `docker: command not found` in a job | That image has no docker CLI; use the `catthehacker` one, which does |
| `Cannot connect to the Docker daemon` in a job | The socket is not mounted into the runner, or `docker_host` is `"-"` (found for the runner, withheld from jobs) |
| `unable to parse docker host …`, daemon restart-loops | `docker_host` is set to a word this act_runner does not know — `automount` on 0.3.x. Leave it empty |
| Everything is slow on the first run | It is pulling the job image. It is cached afterwards |