Plans

A plan is the list of components a device should be running. It is deliberately almost empty of logic — all the detail lives in the recipes.

[[components]]
name = "database"
recipe = "recipes/com.acme.database.toml"

[[components]]
name = "api"
recipe = "recipes/com.acme.api.toml"

[[components]]
name = "metrics"
recipe = "com.acme.telegraf:1.2.0"     # from the agent's recipe store

• name is the component name — how you will refer to it in the API and the logs.
• recipe is either a path to a TOML file or a name:version reference to a recipe already in the agent’s store (uploaded via POST /v1/recipes).

Applying a plan

curl -X POST --data-binary @plan.toml http://127.0.0.1:8080/v1/plan/apply
keystonectl apply plan.toml                       # same thing

You upload the plan’s content. Asking the agent to read a path of its own ({"planPath": …}) is rejected: it would let anyone who can reach the API read arbitrary files from the device.

To preview without touching anything:

curl -X POST --data-binary @plan.toml "http://127.0.0.1:8080/v1/plan/apply?dry=true"

A dry run logs the stop, start and no-touch sets in the agent’s log and changes nothing: it installs and starts nothing, and the plan in force, its status and the persisted state stay exactly as they were. A dry run on a running device is safe — previewing the next plan, or the one already in force, has no effect on the next apply, on a restart of the agent or of a component, or on a reconcile.

It used to record the previewed plan as the one in force, under the status dry-run. The next apply then found nothing to do, and an agent restarted after a preview — an upgrade, a power cut — resumed nothing, because dry-run means nothing was ever installed.

It is also stricter than a real apply on one point: it refuses any key the agent does not recognise, where an apply accepts it and reports it. That is deliberate — see the TOML cheat sheet for why the two differ and which one you want when.

What an apply actually does

flowchart TB
    A["plan arrives"] --> B["load and validate recipes"]
    B --> C["verify recipe signatures"]
    C --> D["build the dependency graph"]
    D --> E["reconcile with the running plan"]
    E --> F["stop what must go"]
    F --> G["download and verify artifacts"]
    G --> H["run install hooks"]
    H --> I["start layer by layer"]
    I --> J{"every layer up?"}
    J -- "yes" --> K["plan status: running"]
    J -- "no" --> L["unwind, then roll back"]

The reconcile step is what makes re-applying cheap: unchanged components that are alive and supervised are left completely alone. See Reconcile and reuse.

The same thing, as a conversation

Who calls whom during a successful apply of a two-component plan:

sequenceDiagram
    actor OP as Operator
    participant AG as Agent
    participant AR as Artifacts

    OP->>AG: POST /v1/plan/apply
    AG->>AG: parse plan, load recipes
    AG->>AG: verify recipe signatures
    Note over AG: before any hook runs
    AG->>AG: reconcile against the running plan
    AG->>AR: download and verify artifacts
    AR-->>AG: ok
    AG-->>OP: 202 Accepted

Signature verification happens before any hook, because an install hook is arbitrary shell — verifying afterwards would be theatre.

Then the supervisor brings the components up, layer by layer:

sequenceDiagram
    participant SU as Supervisor
    participant AG as Agent
    participant RU as Runner

    SU->>AG: Install(database)
    SU->>AG: Start(database)
    AG->>RU: RunManaged(database)
    RU-->>SU: ready
    SU->>AG: Install + Start(api)
    AG->>RU: RunManaged(api)
    RU->>RU: health probe loop
    RU-->>SU: ready when first healthy
    SU-->>AG: all components running

api reports ready on its first healthy probe, not when its process appears, which is what makes layered ordering mean something.

Two details the diagram makes obvious. Signature verification happens before any hook, because an install hook is arbitrary shell — verifying afterwards would be theatre. And api only reports ready on its first healthy probe, not when its process appears, which is what makes layered ordering mean something.

When it goes wrong

sequenceDiagram
    participant SU as Supervisor
    participant AG as Agent
    participant RU as Runner

    SU->>RU: start database
    RU-->>SU: ready
    SU->>RU: start api
    RU--xSU: readiness timeout
    Note over SU: layer failed
    SU->>SU: cancel the shared context
    SU->>AG: stop what started
    AG->>RU: stop database
    Note over AG: handle dropped, state recorded
    AG->>AG: re-apply the previous plan

Step 8 is the one that used to be missing. Signalling the process without dropping its handle left a component that still read as running, which the rollback then adopted — the bug behind #10.

Plan status

curl -s http://127.0.0.1:8080/v1/plan/status | jq

The response carries planPath, status, the component list, and error when the last operation failed.

flowchart TB
    I["idle"] --> A["applying"]
    A -- "every layer up" --> R["running"]
    A -- "a layer failed" --> F["failed"]
    R -- "apply again" --> A
    F -- "apply again" --> A
    R -- "stop" --> S["stopped"]
    S -- "apply again" --> A

Only one apply runs at a time; a second concurrent request is rejected rather than queued.

Rollback

If any layer fails to come up, the agent stops what it started and re-applies the previous plan. You end up back where you were, with an error explaining why:

apply failed and rollback was completed: api start: start readiness timeout

If the rollback itself fails you get both errors, and the plan status is failed.

There has to be a different plan to go back to. Re-applying the plan already in effect — the boot resume, or the same plan applied again — has no predecessor: the rollback would re-read the same file and re-apply the same failure, after stopping every component in the plan to get there. The agent detects this and says so instead:

apply failed and there is nothing to roll back to: runtime/plans/applied.toml
was already the plan in effect, so the plan is not stopped and re-applied

The failed apply still unwinds whatever it started in the layer that failed. What it no longer does is stop the components it reused and never touched.

What rollback does not undo

Two limits, both worth knowing before you rely on a rollback in production.

A moving tag makes rollback a no-op. The previous plan names its images by reference. If that reference is :latest — or no tag at all, or main, stable, edge — then re-applying it pulls whatever that tag points at now, which may well be the image that just failed. The rollback completes and reports success, having changed nothing. The agent warns at apply time when a container recipe uses one:

component=api WARNING image "registry.example:5000/api" uses the moving tag
"latest": a rollback re-applies the previous plan, which would pull whatever
that tag points at then and report success. Pin a digest or an immutable tag

Rollback is atomic over files and processes, never over data. It reverts the plan — binaries, images, which components run — and it does not touch anything a component wrote: databases, volumes, caches, state directories. If a component migrates its data on startup and has no down-migration, rolling the binary back leaves it facing state it cannot read, which fails in ways that look nothing like a failed deployment.

Keystone cannot see that by itself, so a recipe declares it:

[lifecycle.run.state]
version = 21        # the state version this build requires

Before rolling back, the agent compares that number for each component against the one the previous plan declares for the same component. If the previous plan is behind, the rollback would move the binaries backwards across a migration the data has already gone through, and it is refused:

apply failed and the rollback was REFUSED: it would cross a state migration
(component api: state version 21 -> 20). Rolling back reverts binaries and
images, never the data a component wrote, so the previous build would face
state it cannot read. Resolve the data side first, then apply the previous
plan explicitly

The way out is to fix the data side and then apply the previous plan as a normal apply — an explicit decision by an operator who knows what the data looks like, which is the only place that decision can be made.

Four things worth knowing about the comparison:

• A component is only ever compared with itself. Two components’ numbers never meet: they describe different data with different histories, so one project numbering migrations by timestamp and another by a short sequence never collide.
• Absent is not zero. Zero is a legitimate version — state before the first migration. A recipe that omits the block has no opinion.
• A version on only one side is reported, not enforced. It cannot be compared, and refusing there would make the field impossible to adopt: the first plan to declare one always faces a predecessor that does not.
• The number must stay monotonic within a component across its own versions. Changing numbering scheme midway — a short sequence to a timestamp — breaks the guardrail while every individual number stays correct.

It is a declaration, not an observation, on purpose. The build that migrates the data is the new one, and by the time a failed apply decides to roll back the agent has already stopped it: there is nobody left to ask. Two recipes are both on disk at that moment.

Stopping

curl -X POST http://127.0.0.1:8080/v1/plan/stop

Components are stopped in reverse dependency order, and the plan status becomes stopped — which the agent takes as an explicit instruction, so it will not resume that plan on the next boot.

The graph endpoint

curl -s http://127.0.0.1:8080/v1/plan/graph | jq

Returns the nodes, the edges and the topological start order — useful for rendering, and for checking that your [[dependencies]] blocks say what you meant.