Skip to content

Watch your team for dark or stuck workers

PortalCLIBeginner~10 min

You can tell at a glance which of your team's workers are working, which are waiting for you at a menu or question, and which have no agent running at all, without opening anyone's terminal. Open your team in the portal and look at the status next to each worker. Each status also says how long ago it was checked, so you can see when it has gone out of date.

You do not need a terminal or the CLI for this. Someone has to set up the reporting on the machine that runs your workers once (see Who fills this in), and after that it is only reading.

Where to look in the portal

Two places show it:

  • The AI Team grid on your team's App page. Each worker has a status pill, such as Locked: Waiting at a menu or question, with the age of the report beside it, such as checked 5 min. ago.
  • The worker's own page. The pill is on its own line, the reason is written out under it, and two lines below give Last turn and checked.

You see it only if you may see the worker's working state, which App participants may. Someone who may not sees no status at all, not "Not reported": a missing pill means "not shown to you", not "nothing was reported".

What each status means

The meanings here are what the platform shows today. They describe what the host saw, and they decide nothing on their own.

You see What it means What you can do
Idle The terminal is at an empty prompt with nothing running. Nothing needs doing. If you expected work, check the worker has a task or is due a wake.
Busy A turn is running. Leave it.
Locked Something is blocking the terminal. The reason says what: Waiting at a menu or question means a person has to answer it. Screen not recognized means the host could not tell, so it assumes the worst and does not guess. If it is waiting at a menu or question, answer it in that terminal.
Dark Something is broken. The reason says what (see below). Open the worker's terminal and find out.
Not reported The platform has no report for this worker. Ask whoever runs the host whether the reporting is set up and running.
Stale The last report is more than 30 minutes old. The portal then hides the old status instead of showing it. Check when the host last reported and how often it is meant to. A host that reports less often than every 30 minutes shows Stale between reports, with nothing wrong.

A Dark worker's reason is one of the following. The Locked reasons above are examples: the host can send others, and a reason the portal does not know shows as Other reason.

Reason shown Meaning
Missed its ticker wake A ticker the worker follows was due and went unanswered for at least the interval of its fastest ticker.
No terminal found The host found no terminal for this worker.
Agent not running (plain shell) There is a terminal, but only a shell, not the agent.
No agent in the terminal There is a terminal, but no agent is connected in it.

A worker with nothing due is never Dark just because it has been quiet for a long time.

How old is the report

Always read the age. A status says what the terminal looked like when it was checked, and nothing about the worker after that. A worker shown as Idle in a report from 20 minutes ago may be busy now.

The portal shows checked 5 min. ago next to the status. It marks a report Stale once it is more than 30 minutes old, and it never changes what was stored.

Reading it in a terminal

If you prefer the CLI, hadron team roster has a TERMINAL column, with the age in brackets:

WORKER    STATE      TERMINAL                       ROLE
Iris      WORKING    idle (11m)                     backend-engineer
Dara      AVAILABLE  locked (unrecognized, 9m)      qa-engineer
Tove      AVAILABLE  dark (no_terminal, 17m)        docs-engineer

The columns shown here are a subset of the real output. hadron team worker get <name> prints the same report on a terminal: line, with the host it came from and the prompt. These commands need CLI v0.44.0 or later. The reason codes in brackets are the ones above in their raw form: menu_or_question, unrecognized, no_turn_for_due_ticker, no_terminal, bare_shell, no_agent.

Two different things called dark

The word dark names two unrelated things. Do not mix them up.

  • The terminal status Dark comes from the host's report of a terminal, as described on this page. It raises no alarm.
  • The dark-worker alarm is raised by the server when a worker's check status becomes OVERDUE: it has stopped checking in for its task. It is a separate alarm that goes up the worker's reporting chain. It does not read the terminal.

A worker can be one, both or neither. When someone says a worker is "dark", ask which they mean.

Who fills this in

A host fills in the status. A host is the machine that runs Orca and starts your team's workers, and it runs a short command, hadron team health sweep, on a schedule. Without it, every worker reads Not reported. Setting that up is a job for whoever runs the host: see Run a host that launches your team's workers. The command does not alarm, restart anything or post a repair notice; it only stores what it saw.

Known defect in v0.44.0: the sweep can hold back a due wake

On the released v0.44.0, every sweep that finds an idle terminal reads due wakes the way a spending read does, not the no-spend preview. When wakes are due, that read can reserve a slot of the App's daily wake capacity, and when capacity is short it can withhold another worker's due wake; the platform may then post its daily budget notice. With nothing due, nothing is reserved. The sweep never acknowledges the wake or raises the acknowledged-wake counter, and the withheld work stays due. Until hadron-cli#1833 is fixed, do not schedule install. A sweep you run by hand makes the same read, so use it sparingly. Details are in the team health reference.