Runners

A runner is the thing that makes a workload exist. Two implementations, one interface.

ProcessRunner

The default, and the reason Keystone is viable on small hardware. A component is just a child process.

Process groups. Every component is started with Setpgid, so signals reach the whole tree. A shell script that spawns children is stopped properly rather than leaving orphans.

Stopping. SIGTERM to the process group, wait for the timeout, then SIGKILL, then a final 3 s grace. A component that ignores SIGTERM still dies.

Logs. Where journald runs, a component’s stdout and stderr are journald streams, opened by the agent and handed to the process. Entries carry the identifier keystone/<component>, stdout at priority info and stderr at err, and belong to the agent’s unit:

journalctl -t keystone/api          # one component
journalctl -u keystone              # the agent and everything it runs

The stream belongs to the component, not to the agent. That is the point: when the agent dies — a crash, or exiting to replace its own binary — the component goes on logging, and an adopted process has no gap in its output. Journald’s own rate limits and disk caps apply.

Without journald (no /run/systemd/journal/stdout), output falls back to pipes the agent reads, tagged into its log ([runner] component=api stream=stdout …). The agent logs a warning per component, because then a component that writes dies with the agent: its next write gets SIGPIPE. Re-adoption only holds for silent processes in that mode.

Resource limits. RLIMIT_NOFILE from [resources].open_files is applied. CPU and memory limits for processes are placeholders today — use a container, or a systemd slice, if you need them enforced.

Privilege dropping. See Process privileges.

ContainerRunner

Two paths, chosen by [lifecycle.run.container].runtime:

The containerd path talks directly to the socket (KEYSTONE_CONTAINERD_SOCKET, namespace KEYSTONE_CONTAINERD_NAMESPACE) and manages images, snapshots and tasks itself. The CLI path is the pragmatic fallback for devices where Docker or Podman is already the way things are done.

Container components report pid = 0 in the API — there is no host PID that means anything — so their liveness signal is the supervision loop. See the caveat in Component state.

Health probes

sequenceDiagram
    autonumber
    participant AG as Agent
    participant RU as RunManaged
    participant P as Process

    AG->>RU: start supervising
    RU->>P: spawn
    RU->>AG: onStart(pid)
    loop every interval
        RU->>P: health probe
        RU->>AG: onHealth(ok) when it changes
    end
    P--xRU: exits
    RU->>RU: restart policy applies?
    RU->>AG: onExit(err) when it will not restart

Three forms, all with interval, timeout and failure_threshold:

check = "http://127.0.0.1:8080/healthz"   # 2xx is healthy — a redirect is not
check = "https://127.0.0.1:8443/healthz"  # same, over TLS
check = "tcp://127.0.0.1:5432"            # a successful connect is healthy
check = "cmd:/usr/local/bin/check.sh"     # exit 0 is healthy

The HTTP probe accepts 200–299 only. A health endpoint that redirects reads as unhealthy, which catches a surprising number of misconfigured reverse proxies.

Health interacts with the restart policy:

• restart_policy = "always" — after failure_threshold consecutive failures the runner stops the component, which makes the exit path restart it.
• on-failure / never — a failing probe is reported (last_health: unhealthy) but does not by itself trigger a restart. The state is honest; the decision is yours.

Restart policy and backoff

flowchart TB
    E["the workload exits"] --> P{"policy?"}
    P -- "never" --> STOP["report and stop"]
    P -- "on-failure" --> Q{"exit code 0?"}
    Q -- "yes" --> STOP
    Q -- "no" --> R{"retries left?"}
    P -- "always" --> R
    R -- "yes" --> BACK["wait, backoff, restart"]
    R -- "no" --> FAIL["failed"]

attempt 1 → 1s   attempt 2 → 2s   attempt 3 → 4s   …   capped at 60s

±25 % jitter, so a device that reboots with several crash-looping components does not produce a synchronised thundering herd. max_retries defaults to 5 for always and on-failure; exhausting it is terminal (failed).

The exit contract

When a workload exits and will not be restarted, the runner calls back once with the exit error — nil for a clean exit. The agent treats both as terminal:

• error → failed
• clean → stopped

and in both cases clears the PID, resets the health verdict, deregisters the handle and releases the runner. Treating a clean exit as “nothing to report” is what used to freeze a dead component at running/healthy.

An exit the agent asked for is not reported. A stop, a restart, a plan stop and the agent’s own shutdown all cancel the runner’s loop before they stop the workload, and an exit that arrives after that cancellation belongs to whoever cancelled it. That matters most for a restart: the agent starts the next instance as soon as the old one is gone, and the old one’s exit, reported as terminal, used to land on the new instance — under never, keystonectl restart stopped a component and left it down, with the restart timing out waiting for a PID. The restart policy only ever decides about exits nobody asked for.