Skip to main content
Version: Next

0032: The label is the lock; a PID and a heartbeat decide whether anyone still holds it

  • Status: Accepted (backfilled 2026-10-06: records a decision already built)
  • Date: 2026-10-06
  • Related: 0017 (state in files, not a job queue)

Context​

Two runs must never work one ticket. The lock has to be visible to every watcher, on any machine, after any restart, so it lives on the ticket: the orion-working label. Claiming a ticket swaps the queue label for it in one write.

What a label cannot say is whether anyone still holds it. orion-working on a ticket whose agent was killed looks exactly like orion-working mid-run, and the queue excludes both. Before this, an interrupted ticket was stranded until a person removed the label. The tracker cannot answer the question either: most workflows do not put a stalled ticket in the Done category.

Decision​

The label stays the lock. It is matched exactly by the queue and the in-flight query; which actor holds it is a separate orion-stage-* label that nothing reads for control (internal/actors).

A claim record decides liveness (internal/claim/claim.go): the PID of the process that took the claim and a heartbeat it refreshes. Both are needed, and neither alone is enough:

  • a PID alone is wrong across a reboot, where the number is reused by an unrelated process and the claim reads as live forever;
  • a heartbeat alone is wrong for a long run: an agent working for fifty-eight minutes is not stalled, and a watcher that stole its ticket at thirty would be the more expensive bug.

A claim is dead when its process is gone, and only then; the heartbeat is the tiebreaker for the reboot case.

The watch reconciles claims every pass (watch.InFlight): a dead holder's claim is released, a ticket closed by hand loses its lock, and a ticket whose own log shows its run finished, but whose label never moved, is moved to orion-ready.

Consequences​

  • A killed or crashed run never strands its ticket.
  • Two watchers, even on different machines, cannot work one ticket.
  • A person must never remove orion-working from a ticket a run still holds; the docs say so wherever the label is described.

Alternatives rejected​

  • A lock file instead of a label. Invisible to a second machine and to anyone reading the tracker.
  • A timeout on the label. Steals tickets from long, healthy runs.
  • Ask the tracker. It has no notion of a process.