Signing

Everything the agent installs must be traceable to a key you control. That is done with detached signatures verified against a trust bundle on the device.

What gets verified

Thing	Signature	When it is checked
Artifact	`sig_uri` (or `<uri>.sig`)	Before it is unpacked or used, on apply and restart
Recipe file	`<recipe>.toml.sig`	Before any lifecycle hook runs
Recipe via API	—	Trusted by the API authentication instead

A staged artifact is the verified file itself — the workdir holds a hard link to the cached copy — so tampering with it in place is caught at the next start. See Staging into the workdir.

The order matters for recipes: the install hook is arbitrary shell, so verifying after running it would be pointless.

sequenceDiagram
    autonumber
    participant CI as Your CI
    participant ST as Artifact store
    participant AG as Agent

    CI->>CI: build, then sign with the release key
    CI->>ST: publish artifact + .sig
    AG->>ST: download both
    AG->>AG: sha256 matches?
    AG->>AG: signature chains to the trust bundle?
    AG->>AG: only now: unpack and run hooks

Trust bundle

Point the agent at a PEM file of CA certificates:

export KEYSTONE_TRUST_BUNDLE=/etc/keystone/trust/ca.pem
keystone --http 127.0.0.1:8080

Signatures are Ed25519 (the default), RSA or ECDSA, verified against a leaf certificate that must chain to the bundle. Ed25519 is what the tooling now generates: 32-byte keys, and verification is far cheaper on an ARMv7 gateway than RSA-3072. The other two keep working, so material generated earlier is unaffected. The leaf can be provisioned on the device or fetched per artifact with cert_uri.

The signer must be issued for codeSigning

The leaf that signs has to list codeSigning in its extended key usage, and every CA between it and the bundle has to allow it too. Anything else is refused: a TLS server certificate, a device’s client certificate, or a certificate with no extended key usage at all.

Why it is strict: the verifier used to ask for no usage, and Go then checks for server authentication. So any certificate from the bundle with no EKU or with serverAuth — a broker’s TLS certificate, for one — could sign a recipe, while a correctly issued codeSigning-only certificate was rejected. A CA that issues identities for connections must never be able to authorise code; restricting its EKU is what makes that hold.

Certificates after the leaf in the certificate file are used as intermediates, so a signer can chain through a signing CA without that CA being in the bundle.

Certificates with no EKU, during a transition

Signers issued before this was required usually carry no extended key usage and are now refused. --allow-no-eku-signers (KEYSTONE_ALLOW_NO_EKU_SIGNERS=true) admits them, and only them: never a certificate issued for something else. The agent says so in its log every time it uses the allowance. Reissue the signer for codeSigning and turn it off; the flag will be removed. keystonectl verify honours the same variable, and keystonectl sign refuses a certificate the agents will refuse, so the mistake shows at signing rather than on every device. The self-update gate (keystone --verify-update) honours the variable too, but not the flag, which it never sees: set it in /etc/keystone/keystone.env, or the gate refuses what the agent proposed and the update rolls back.

Upgrading a device from v0.12.7 or earlier

Versions before v0.12.8 verify signers for serverAuth, so they refuse a certificate issued only for codeSigning. The update itself is verified by the version already installed, both by the agent and by the self-update gate, so re-signing the new binary with a new signer makes the update fail. Move a device in this order:

Set KEYSTONE_ALLOW_NO_EKU_SIGNERS=true in /etc/keystone/keystone.env. Older versions ignore it.
Update to v0.12.8 or later, with the binary and recipes still signed by the existing no-EKU signer.
Once the update is confirmed, reissue the signer for codeSigning, re-sign recipes and artifacts, and remove the variable.

The trust bundle is for code only

The agent refuses to start when a certificate in the trust bundle shares a key with one used for transport: --mqtt-tls-ca or the --mqtt-tls-cert chain. It is compared by key, so a CA reissued under a new serial still counts. Keep one CA, or a set of them, for signing code, and different ones for broker and device identities.

To issue a signer with openssl:

printf 'extendedKeyUsage=codeSigning\nkeyUsage=critical,digitalSignature\n' > leaf.ext
openssl x509 -req -in leaf.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
  -extfile leaf.ext -out leaf.pem -days 365

The scheme, exactly

The signed message is the file’s 32-byte SHA-256 digest, for every algorithm. For Ed25519 that means the digest is the message and the scheme is not Ed25519ph — Ed25519 hashes what it is given with SHA-512 internally, so it hashes the digest. Signer and verifier must agree on this or nothing validates, which is why it is written here rather than left to be read out of the code.

Ed25519 and older agents

Support for Ed25519 was added after RSA and ECDSA. An agent from before it rejects an Ed25519 signature as an unsupported key type — correctly, failing closed. Do not start signing with Ed25519 until the fleet runs a build that understands it. Nothing forces the move: sign with RSA or ECDSA until it has.

openssl cannot sign with an Ed25519 key the usual way

openssl dgst -sha256 -sign does not work with an Ed25519 key — that needs pkeyutl -rawin over the digest, and getting it subtly wrong produces a signature that only fails on a device. Use keystonectl sign, which is the code the agent verifies with.

Signing

keystonectl signs; the agent never does:

keystonectl sign --key signer.key --cert signer.pem com.example.api.recipe.toml
# → com.example.api.recipe.toml.sig

keystonectl verify --trust-bundle ca.pem com.example.api.recipe.toml
# → OK: ... verifies against ca.pem

sign is local and contacts no agent. When --cert is given the signature is checked against that certificate before anything is written, so a mismatched key and certificate fail at your desk instead of on every device.

Run keystonectl verify in CI on every publication. A signature that fails there fails on the whole fleet, and finding out now costs nothing.

The agent cannot sign

internal/signing is linked into keystonectl only, and a test fails the build if cmd/keystone ever links it. A gateway in a plant is the most exposed thing in the system; one that carried signing machinery would hand whoever took it a head start on forging updates for everything else.

Setting it up for development

The repository ships a helper that creates a throwaway CA on first use and signs each file you pass it:

scripts/dev-sign.sh configs/examples/com.keystone.server.recipe.toml
# creates configs/trust/{ca,leaf}.{key,pem} once, then writes <file>.sig

export KEYSTONE_TRUST_BUNDLE=configs/trust/ca.pem
export KEYSTONE_LEAF_CERT=configs/trust/leaf.pem

See configs/trust/README.md for the long form, including how to lay this out for production with a real CA.

Warning

A dev CA is a dev CA. For real fleets the signing key belongs in an HSM or a CI-managed KMS, and the private key should never touch a developer laptop.

Devices whose clock cannot be trusted

A gateway with no RTC boots with its clock at 1970. Every valid certificate is then “not yet valid”, and the device rejects the recipes, artifacts and manifests it needs — often before it can reach NTP, because the component that gives it a network is in the plan it just refused.

Turning off the validity check is not the answer: expiry is what makes a compromised signer stop working on its own, and switching it off hands an attacker a way to revive a certificate you revoked. So the agent keeps its own lower bound on the real time, built from evidence:

the build timestamp of the binary, since it cannot be running before it was built;
a high-water mark persisted across restarts, so time never goes backwards between boots.

Certificates are checked against the later of the system clock and that evidence. Setting a device’s clock back therefore achieves nothing, which closes the attack that matters. Setting it forward only expires certificates early — noisy, and not a bypass.

keystone --clock-policy high-water   # default
keystone --clock-policy strict       # refuse to verify while the clock is behind

Under strict a verification failure is reported as retryable (HTTP 503), since it clears by itself once NTP runs.

Either way the agent says so, and it is worth alerting on — a device checking expiry against an approximate time looks perfectly healthy otherwise:

curl -s localhost:8080/healthz | jq '{clock_trusted, clock_source}'
# { "clock_trusted": false, "clock_source": "high-water" }

Note

If you specify the hardware, fit an RTC with a battery. All of the above is a good mitigation for saving a euro on the bill of materials, and it is still needed for hardware you do not control.

Rotating

The trust bundle is a file of CA certificates, so rotation is: add the new CA to the bundle, ship it, start signing with the new key, then remove the old CA once nothing signed by it remains in any plan you might roll back to.

That last clause is the one people get wrong. A rollback re-applies an older plan with older artifacts — if you have already dropped the CA that signed them, the rollback fails closed. Keep the old CA for at least as long as your rollback horizon.