Skip to content

Stopping and isolating workers

Sometimes you want a team's workers to stop picking up new work, or one worker to stop being nudged for a while. Hadron has two controls for that: a stop signal, which covers a worker, a whole team or a host, and an isolation, which quiets one worker. Both can be refused when you try to set them, because each depends on a piece that is not fully built: a stop signal is refused while a worker it would cover has nothing that can receive stop notices, and an isolation is refused until the server turns isolation on. This page explains what each control is for and what is built, so you know what a refusal means when you meet one.

Only people do this. Nobody sets these controls by prompting an agent: the agent-team MCP tools are being retired (hadron-server#2937), and the controls live in the CLI, the GraphQL API and, for stop signals, the portal. The portal has a Stops page for the App where a manager sets and lifts stop signals (hadron-portal#1406); it has no control for isolation. If you want an agent to type a CLI command for you, Working through an agent says what that involves.

Pausing a worker's own wakes is a third, older control and is described in What wakes a worker. The two here are different: a stop signal and an isolation are set on a worker by someone else, and they do not replace that pause.

Stop signals

A stop signal tells covered workers to stop at one of three levels:

Level What it stops
pause-dispatch New assignments.
pause New assignments, and also ticks and stall time.
halt Everything pause stops, plus an interrupt telling the worker to save its work and hand off.

A signal's scope is one worker, a team (a head worker and the whole reporting tree under it, named by its head), or a host, which covers the workers registered on that machine. It lasts until it is lifted, or until an end time you give it.

Signals are independent. Setting one never replaces another, and lifting one never ends another. A worker covered by two signals stays covered until both are gone, so lifting the one you set may change nothing you can see.

Who may set and lift one

Whoever manages the App may set any level and scope, and lift any signal (setting is Decided; who may lift is Proposed, as the next section says). A host key may set a halt on its own host and lift it, and nothing else. A worker's session is refused, even a manager's. The server decides who may do what and refuses the rest, so a command that comes back refused has not changed anything.

What does not work yet

The part that tells a worker to stop is not built. A stop notice is recorded, but the host-side piece that types it to the worker is still open (hadron-server#2169). Until it exists:

  • The server refuses to set a stop signal with STOP_SIGNAL_UNAVAILABLE. This is conditional. It applies while any worker the signal would cover is on a lane where nothing has recently said it can receive stop notices, so it lifts once every covered worker's lane has such a consumer. It is not a permanent block.
  • Listing and lifting signals work, because neither stops anyone.
  • Even a stored signal is not proof that a worker stopped or resumed. This page makes no promise that a worker obeys a signal; what the server keeps is the record of what was set.

The team's own rule is the same: nobody sets or lifts a live stop signal before that consumer is live.

How firm each rule is

The behavior above is built. How firmly each rule is settled differs, and this page keeps them apart. Citations are to the product specs; "unminted" means a draft not yet in the corpus.

  • Decided (cor:agt:090:06): an App manager sets any level and scope; a host key sets a halt on its own host only; a worker's session is refused.
  • Proposed (cor:agt:090:06): that an App manager may lift any signal, that a host key may also lift its own halt, the typed refusal when a lift is not allowed, and the check made at write time. Read these as how the server behaves now.
  • Conditional, per cor:agt:090:04: STOP_SIGNAL_UNAVAILABLE applies only until every covered worker is on a lane with a recent notice-consumer declaration. It is not a permanent block.
  • Open: what a worker does with a stop notice. The host consumer that would type it is not built, so no rule here promises that a worker obeys.

Commands

hadron team stop list, set and lift work with stop signals, in CLI v0.33.0 and later. Their options are in the CLI reference. The same operations are the GraphQL teamStopSignals query and the setTeamStopSignal and liftTeamStopSignal mutations.

Isolation

An isolation quiets one worker. It is meant to hold what is addressed to the worker until the isolation ends or is cleared. It is its own state beside the worker's pause, and it never touches that pause.

  • It lasts 2 hours by default. You can give an end time, up to 7 days ahead. Setting it again with a different end extends or shortens it.
  • Only a person acting as themselves may set or clear it, and only one who manages the App or holds the worker's name. Anyone in that group may change or clear an isolation another of them set. A worker session may not, not even one a manager is driving. An App key and an impersonation may not either.

Isolation law (cor:agt:080:29) is Decided and partly built: the setting is refused while isolation is off, and clearing works.

What does not work yet

Setting an isolation is refused with WORKER_ISOLATION_UNAVAILABLE until the server turns isolation on (hadron-server#2130). Clearing one works. While isolation is off the server reports no isolation, so a worker's settings never show it isolated, and a successful clear does not mean one was in force.

Commands

hadron team isolation set and clear are in CLI v0.34.0 and later. Read a worker's isolation with hadron team wake-settings get. The same operations are the GraphQL isolateWorker and clearWorkerIsolation mutations.