Skip to content

Run a host that launches your team's workers

CLI onlyAdvanced~30 min

Once a machine is set up as a host, your team App can start a worker's agent on it, ask that agent to end its session, and create or remove its git worktrees, without anyone logging in to the machine. You will need the CLI and a shell on that machine for this one. It is a one-time install, like setting up a server; no agent or portal page can do it for you.

A host runs one long-lived process, the host poller. It checks in with the platform, collects the operations issued to it, carries out the ones it recognises, and reports how each one went. What a host is, and what happens when one stops checking in, is in Team tasks: Changing the windows.

Before you start

  • A CLI release that has the host commands. They arrive in the first release after v0.23.1. Until that is out, build the CLI from main at hadron-cli#882 or later. Check with hadron team host --help: it lists run and status.
  • You manage the team App. Only whoever manages the App can give a key the host scope.
  • Git, a local clone of each repo the host should make worktrees from, and the command that starts each worker's agent on this machine.
  • The agent's own login is already on the machine, under the OS user the agent will run as. The poller never hands the agent a Hadron credential: the agent binds its worker session with its holder's own stored token.

Give the host its key

A host authenticates with an App key that holds the host:operate scope for one host id. The host id is the name the platform knows the host by, and the key fixes it: a key granted for build-box-1 registers, collects and reports as build-box-1 and nothing else. The label in the configuration is only a display name.

  1. Create an App key for the team App on the App's settings page in the portal, and copy the key and its id.

  2. Grant it the scope, naming the host id:

    hadron app key scope grant <key-id> host:operate --host-id build-box-1

    A key that already operates as another host is refused APP_KEY_SCOPE_HOST_CONFLICT. Revoke host:operate from it first, or use another key.

  3. Put the key in a file that only the poller's OS user can read:

    sudo install -d -o hadron-host -m 0700 /etc/hadron
sudo install -o hadron-host -m 0600 /dev/null /etc/hadron/host.key
sudoedit /etc/hadron/host.key        # paste the key, save

The poller reads the key only from this file. It never reads it from HADRON_TOKEN, the CLI's token store or any environment variable, and it refuses a file that group or others can read.

Run the poller as its own OS user

The poller keeps the key out of every launched agent's environment, arguments and files. It cannot keep the key from the agent's OS user: an agent running as the same user as the poller can read key_file itself. Run the poller as a dedicated user (hadron-host above), with key_file readable by that user only, and the agents as their holders' users. The poller starts each agent as a child process, so its launch argv has to switch to the holder's user itself, and whatever does the switch must keep HADRON_LAUNCH_GRANT in the environment (sudo drops it unless told to preserve it). If you don't separate the users, the gap stays open: treat the host key as readable by every agent you launch there.

Write the host configuration

The configuration stays on the host. The platform never sees it, including the launch commands.

app: hrn:app:acme.com:eng-team
label: build-box-1                  # shown to people; default: the hostname
poll_interval_seconds: 30           # 1 to 86400; default 30
key_file: /etc/hadron/host.key      # relative paths resolve beside this file
worktrees_root: /srv/worktrees      # must be absolute
state_file: /var/lib/hadron/host-state.json   # default: beside this file
repos:
  acme/api: /srv/repos/api          # repo token -> local clone (absolute);
                                    # worktree.create's repo must match it exactly
workers:
  dara:                             # the worker's name in the App, URN or id
    family: claude                  # the model family it runs as here
    launch: [claude-worker, --model, opus]   # placeholder: your own foreground
                                             # launcher, which switches to the
                                             # holder's OS user (see the warning)
    end: [notify-agent, dara, "please end your session with a handoff"]
    end_wait: 15m                   # default 15m
    worktrees: [api/dara]           # existing trees, relative to worktrees_root

Each worker entry says how this host runs that worker:

Key Required What it is
family yes Lowercase family token, such as claude or codex. The platform only starts the worker here as this family.
launch yes The argv that starts the agent in a worktree. It runs as given, with nothing appended.
end no An argv that asks the running agent to end its own session. It is a prompt, never a kill.
end_wait no How long to wait for the session to end after end. A Go duration, such as 15m.
worktrees no Worktrees the worker already has, up to 32. Trees the poller creates are added on its own.

The launch command must stay in the foreground for the agent's whole life. The poller reads its exit as the agent's. A launcher that starts a detached session and returns at once, such as a tmux new -d wrapper, is reported failed if no session has bound by then, and its launch grant is revoked.

A worktree belongs to one worker. The configuration is refused if the same tree is listed under two workers.

Check it before you run it

hadron team host status --config /etc/hadron/host.yaml

status loads the configuration and the key, prints what the poller would register, lists any worktree it is keeping, and shows the host's registration as the platform holds it. It changes nothing.

host build-box-1 for hrn:app:acme.com:eng-team, checking in every 30s
WORKER  FAMILY  LAUNCH                          WORKTREES
dara    claude  claude-worker --model opus      api/dara
registration: none found for this poller's host id (it has not registered yet, or the registration is gone)

Before the first run there is no registration yet. After it, the last line reads registration: host build-box-1, live, refreshed …. status finds the registration by the host id saved at the last check-in, never by the label, because two hosts may share a label.

Run the poller

Try one poll first:

hadron team host run --config /etc/hadron/host.yaml --once

--once checks in, collects, waits for everything it collected to finish, and exits. Then run it for real, under a service manager that restarts it:

hadron team host run --config /etc/hadron/host.yaml

Run one poller per host or container. It prints one event per line; --json prints each as an object with time, kind, operation, type, outcome and detail, every key always present.

Every poll does three things:

  1. Registers every configured worker, with its family and its worktrees, including the ones this poller created.
  2. Collects the operations issued to this host, and starts each one once. An operation that is still running when the next collect re-delivers it is skipped, not run twice.
  3. Reports each outcome. A report that does not reach the platform leaves the operation collected, so it comes back on the next collect and runs again. Every operation is safe to repeat.

One worker's operations run one at a time, in the order collected, so a remove queued behind a start sees the agent that start launched. Different workers' operations run side by side.

What each operation does

A host carries out exactly five operation types and nothing else. There is no shell passthrough: every command is git's own or an argv from the configuration.

Operation What the host does Reported
worktree.create Fetches origin/<base> and adds a worktree on a new branch, at <worktrees_root>/<repo basename>/<label>. Its repo argument must match a key under repos exactly (acme/api, not api); anything else is reported failed, repo "api" is not configured on this host. Succeeded, with the new tree's name. A tree already there on that branch also counts as succeeded.
worker.start Runs the worker's launch argv in a worktree registered for it, with a single-use launch grant in HADRON_LAUNCH_GRANT. The agent redeems it when it runs hadron team session start. Succeeded once the agent's session binds. Failed if the agent exits without binding, or the grant's 10-minute lifetime runs out.
worker.end Runs the worker's end argv, then waits up to end_wait or until the agent it launched exits. Always failed: see below.
worker.deliver Nothing yet. Failed: no delivery path on this host yet (hadron-cli#829).
worktree.remove Removes one of the operation's own worker's registered trees, only if nothing would be lost. Succeeded, or failed with the reason. A tree that is already gone counts as removed, so give the full name token, such as api/dara-2: a token naming no tree also reports succeeded.

Who issues them. The worktree operations come from the worker's holder, the worker it reports to, or whoever manages the App, with hadron team host issue. A worker.start comes from whoever manages the App. The platform issues worker.end itself when a finished task asked to close its worker's session and the session is still open after the grace period.

Why worker.end reports failed. Ending a worker means asking it to end its own session, never killing it, and a worker that ends with or without a handoff has done the right thing. The host key cannot see whether the session ended, so the poller reports failed after the wait. The platform records a failed end on a session that did end as the success it was. Reporting succeeded would make the platform end the session itself, which is the kill the rule forbids.

How it fails closed

The poller refuses rather than guesses. The messages below are the commands' own.

Before it starts. These stop run and status with an exit code and nothing sent:

What is wrong What you see Exit
No --config --config is required: the host configuration file 2
An unknown key in the YAML field colour not found in type hostpoller.Config 2
A bad value Every problem at once, separated by ;: worktrees_root must be an absolute path; worker "a": family "Claude" must match …; worktree "shared" is listed for both "a" and "b" … 2
No key file reading host key file: stat …: no such file or directory 3
Key file too open host key file … is readable by group or others (mode 644): chmod 600 it 3
Empty key file host key file … is empty 3

When the platform refuses the key. A revoked key, one without host:operate, or a malformed registration stops run with that code instead of retrying forever, because retrying does not fix it. For example, a key without the scope:

hadron: registering: This App key does not hold the host:operate scope. Whoever manages the App can grant it, for one host.

exits 8. A key_file that holds something other than an App key, such as a personal token, gets Only a host, with its App key's host:operate scope, does this. and also exits 8. Other check-in failures, such as a network error, are logged and the loop carries on. With --once, exit 7 means at least one report did not reach the platform; those operations come back at the next collect.

When removing a worktree. worktree.remove refuses, and reports the reason, for a tree that:

  • isn't registered for the operation's own worker;
  • has uncommitted or untracked changes;
  • has no upstream, or commits its upstream doesn't have;
  • has an agent this poller launched still running in it, remembered across restarts in the state file;
  • has any process working inside it, found through /proc on Linux. On macOS and Windows there is no such scan, so only agents the poller launched protect a tree;
  • belongs to a worker whose worker.end failed.

After a failed end, the tree is kept. The worktree the agent ran in, or every tree the worker has on this host if the poller didn't launch it, is recorded as kept in the state file and never removed. status lists it as kept:. No command clears a keep yet: remove its entry from the state file by hand once you have checked the tree.

What a launched agent gets. The poller's environment minus HADRON_TOKEN, HADRON_HOST_KEY, any inherited HADRON_LAUNCH_GRANT and any variable whose value contains the host key, plus its own HADRON_LAUNCH_GRANT. A later hadron team session start --as … in that agent's shell ignores the leftover grant, so a rotation does not try to reuse it.

What is not here yet

  • No real worker.deliver. It waits on hadron-cli#829.
  • The poller sets no availability marks. It doesn't mark a worker out of tokens or its host down.
  • Existing worktrees are declared, not discovered. List them under the worker's worktrees.
  • After a restart, worker.end waits the full end_wait, because the poller no longer has the agent's process to watch.
  • Name collisions. A created tree is named <repo basename>/<label>, so two repos with the same basename can collide. Re-creating a removed tree on the same branch fails, because removing a tree keeps its branch.
  • The poller needs a platform that never re-mints a launch grant while its agent is still booting (hadron-server#1928).