Hangar, week of Sep 4: what breaks and what changes
LunarCrush · In progress
Two days of the host's own records: 119 deliveries, 342 worker exits for lack of a seat, reviews blocking half the time. Twelve failure classes, one diagnosis, five changes.
Hangar is the platform that runs the coding agents at LunarCrush. I wrote its story in the main case study. This is the first entry of the running log: what the system is doing this week, what is failing, and what I am changing. It is written from the host's own records, the patrol journal, the ledger, the deliveries and the session events, not from memory.
The goal has not moved. Hangar exists so that product tickets reach staging verified, without me babysitting the pipeline, so that my attention goes to product decisions and to reading what shipped. Every number below is measured against that.
The numbers
Between Sep 3 at midnight and Sep 4 at 19:00 UTC, Hangar completed 119 deliveries. In the same window the worker exited 342 times because no seat was available, reviews blocked 53 times against 57 passes, twelve escalations reached my phone, and the models consumed 945 million input tokens. One ticket alone took 97 million of those: eighteen sessions on a single step, eleven of them ending at the turn ceiling.
Two of those lines explain most of the pain. A review blocks almost half the time, and a seat is unavailable five times more often than a flight finishes. Every block costs a remediation session. Every unavailable seat costs a wait and, under the old patrol, a relaunch.
Twelve ways a ticket stalls
I catalogued the failure classes with the mechanism behind each. A step hit the 80-turn ceiling and every retry started a fresh session that re-read the repository, on the cheapest model tier. A delivery waited an hour for staging because the hold watched every workflow instead of the one that builds the delivered paths. A code ticket was refused as a documentation change because its title contained the word "document". A research ticket was refused for the opposite reason. A remediation was judged as "changed nothing" while seven files had been edited by a session that died before it could report. The reviewer blocked on cron workflows that nobody registered on the main branch, and on CDN assets the flight had no token to write. Accept on Telegram moved the ticket in Linear and delivered nothing. Four stale prompts stayed on my phone after their failures had cleared. The checks watcher re-dispatched a production run instead of re-running the red staging one, then opened a duplicate ticket. Hangar's own merges never reached GitHub. An empty UX audit reported zero screens because of a positional field. And Hangar delivered a gate to itself that demanded proof no component produces, merged it into its own main without a person reading it, and blocked product deliveries within hours.
Twelve classes in two days, each one a correct local rule meeting another correct local rule. That is the actual finding.
The diagnosis
Hangar has six components that decide about the same ticket: the router, the executor's gates, the reviewer, the delivery's gates, the patrol and the checks watcher. Each was written from a real incident and each is right on its own terms. They do not share a model of what the ticket is or which transition it is in. So the router puts a ticket in a lane, a gate refuses the lane's output, the patrol retries the same lane, and the watcher opens a second ticket. Four correct decisions, one absurd outcome.
Budgets are per attempt, not per ticket, so a step could burn eleven sessions without anything summing it. The patrol's only moves were retry, delegate or escalate, and until this week every unknown error was a relaunch, the most expensive move available and the least likely to change the outcome. Admission counted seats on paper, sixteen, rather than seats that implement, eight, and a rate-limited provider removes four of those. The 342 "no seat" exits are the queue, not the work.
What changes, in order
The first change is a ticket lifecycle with one owner per transition: routed, implemented, reviewed, delivered, staged, promoted. Each transition is executed by exactly one component; the others may observe and report, never act. The patrol acts only on transitions with no owner running.
The second is to retry only where a retry can change the answer. A seat, a base branch, a runner hiccup: retry. A lane, a reviewer's verdict, a missing credential, a task that does not fit the budget: never retry, decide. An unknown failure is observed and escalated with its log, not relaunched.
The third is a budget per ticket, counted once: sessions and continuations per step, re-routes per ticket, hours per ticket, in one ledger, visible on the task row and in the escalation. Reaching a limit stops the ticket and asks, with the spend in the message.
The fourth is admission by implementing seats and continuation by default. A session that reaches its ceiling continues with its own context instead of restarting. Both are in place; the remaining gap is that an exhausted seat is not yet subtracted from admission.
The fifth is the interface problem, and it is the one I feel most. The Tasks row today shows "Previous attempt failed" for a flight that is simply waiting for a seat. What it should show is the lifecycle state, the spend and the last action. One state page per ticket. The panel was built to launch flights; it has to become the place where I understand them.
A governance gap on top
Two of the classes share a cause: Hangar changes its own code through the same pipeline it runs for products, and a branch-only delivery into the hangar repository is merged into the host's main by the host itself. A gate that could not be satisfied shipped that way. Hangar tickets must stop at Product Review like every other ticket, and the merge into the host's main must be a person's act, or at minimum must pass the full smoke suites on the host first.
What stays open
Why self-host deliveries do not push main. Exhausted seats not subtracted from admission. Five pre-existing red tests in the control plane's smoke suite. One-off verification scripts that the knowledge base now forbids but the delivery gate does not yet catch.
Closing
The tool is not the point. The point is a week where the tickets reach staging and I read them instead of rescuing them. This week was not that. The changes above are what I think gets there.