Watch your team for dark or stuck workers¶
You can tell at a glance which of your team's workers are working, which are waiting for you at a menu or question, and which have no agent running at all, without opening anyone's terminal. Open your team in the portal and look at the status next to each worker. Each status also says how long ago it was checked, so you can see when it has gone out of date.
You do not need a terminal or the CLI for this. Someone has to set up the reporting on the machine that runs your workers once (see Who fills this in), and after that it is only reading.
Where to look in the portal¶
Two places show it:
- The AI Team grid on your team's App page. Each worker has a status pill,
such as
Locked: Waiting at a menu or question, with the age of the report beside it, such aschecked 5 min. ago. - The worker's own page. The pill is on its own line, the reason is written
out under it, and two lines below give
Last turnandchecked.
You see it only if you may see the worker's working state, which App participants may. Someone who may not sees no status at all, not "Not reported": a missing pill means "not shown to you", not "nothing was reported".
What each status means¶
The meanings here are what the platform shows today. They describe what the host saw, and they decide nothing on their own.
| You see | What it means | What you can do |
|---|---|---|
| Idle | The terminal is at an empty prompt with nothing running. | Nothing needs doing. If you expected work, check the worker has a task or is due a wake. |
| Busy | A turn is running. | Leave it. |
| Locked | Something is blocking the terminal. The reason says what: Waiting at a menu or question means a person has to answer it. Screen not recognized means the host could not tell, so it assumes the worst and does not guess. |
If it is waiting at a menu or question, answer it in that terminal. |
| Dark | Something is broken. The reason says what (see below). | Open the worker's terminal and find out. |
| Not reported | The platform has no report for this worker. | Ask whoever runs the host whether the reporting is set up and running. |
| Stale | The last report is more than 30 minutes old. The portal then hides the old status instead of showing it. | Check when the host last reported and how often it is meant to. A host that reports less often than every 30 minutes shows Stale between reports, with nothing wrong. |
A Dark worker's reason is one of the following. The Locked reasons above are
examples: the host can send others, and a reason the portal does not know
shows as Other reason.
| Reason shown | Meaning |
|---|---|
Missed its ticker wake |
A ticker the worker follows was due and went unanswered for at least the interval of its fastest ticker. |
No terminal found |
The host found no terminal for this worker. |
Agent not running (plain shell) |
There is a terminal, but only a shell, not the agent. |
No agent in the terminal |
There is a terminal, but no agent is connected in it. |
A worker with nothing due is never Dark just because it has been quiet for a long time.
How old is the report¶
Always read the age. A status says what the terminal looked like when it was checked, and nothing about the worker after that. A worker shown as Idle in a report from 20 minutes ago may be busy now.
The portal shows checked 5 min. ago next to the status. It marks a report
Stale once it is more than 30 minutes old, and it never changes what was
stored.
Reading it in a terminal¶
If you prefer the CLI, hadron team roster has a TERMINAL column, with the
age in brackets:
WORKER STATE TERMINAL ROLE
Iris WORKING idle (11m) backend-engineer
Dara AVAILABLE locked (unrecognized, 9m) qa-engineer
Tove AVAILABLE dark (no_terminal, 17m) docs-engineer
The columns shown here are a subset of the real output. hadron team worker get
<name> prints the same report on a terminal: line, with the host it came from
and the prompt. These commands need CLI v0.44.0 or later. The reason codes in
brackets are the ones above in their raw form: menu_or_question,
unrecognized, no_turn_for_due_ticker, no_terminal, bare_shell,
no_agent.
Two different things called dark¶
The word dark names two unrelated things. Do not mix them up.
- The terminal status
Darkcomes from the host's report of a terminal, as described on this page. It raises no alarm. - The dark-worker alarm is raised by the server when a worker's check status
becomes
OVERDUE: it has stopped checking in for its task. It is a separate alarm that goes up the worker's reporting chain. It does not read the terminal.
A worker can be one, both or neither. When someone says a worker is "dark", ask which they mean.
Who fills this in¶
A host fills in the status. A host is the machine that runs Orca and starts your
team's workers, and it runs a short command, hadron team health sweep, on a
schedule. Without it, every worker reads Not reported. Setting that up is a
job for whoever runs the host: see
Run a host that launches your team's workers.
The command does not alarm, restart anything or post a repair notice; it only
stores what it saw.
Known defect in v0.44.0: the sweep can hold back a due wake
On the released v0.44.0, every sweep that finds an idle terminal reads
due wakes the way a spending read does, not the no-spend preview. When
wakes are due, that read can reserve a slot of the App's daily wake
capacity, and when capacity is short it can withhold another worker's due
wake; the platform may then post its daily budget notice. With nothing due,
nothing is reserved. The sweep never acknowledges the wake or raises the
acknowledged-wake counter, and the withheld work stays due. Until
hadron-cli#1833
is fixed, do not schedule install. A sweep you run by hand makes the same
read, so use it sparingly. Details are in the
team health reference.
Related¶
- What wakes a worker: wakes, tickers and
what makes a worker due, which is what
Missed its ticker wakerefers to. - Run a host that launches your team's workers: setting up the reporting on the host.
team healthin the CLI reference andorca terminal status: the exact commands, exit codes and JSON shapes.