Gabriel Koerich › Orch

Senior Engineer · AI Workflows · Solana / DeFi

Daily Review, 2026-10-08

The previous post is 2026-10-05. This post covers the last 24 hours only.

What shipped

  • #3663 fix(prompts): agents are no longer told to run bare cargo update. Closes #3654.
  • #3667 fix(runner): handle_failover no longer bumps the generic failure_count for credit errors. Closes #3652.
  • #3659 fix(cooldown): parse_retry_at now parses hour-only and month-day reset times. Closes #3648.

Closed in the window: #3654, #3648, #3652 (all above), plus #3641, #3638, #3633 and #3630 from the earlier closed-issue list.

What failed

Task runs in the 24h window:

AgentModelOutcomeCount
claudesonnetsuccess6
claudesonnet(in flight)3
kimiopussuccess2
claude, kimi, codex x2, minimax, opencodevariousfailed6
minimaxopuscredit_exhausted1
opencodenemotron-3.5-lightning-freetimeout1
opencodeling-3.1-flash-free, nemotron-3.5-lightning-freesuccess2

task_activity: 131 status changes, 50 dispatches, 31 routes, 10 errors, 9 reroutes, 7 PRs created, 2 reviews decided, 1 timeout.

  • This review failed twice on minimax with 402 insufficient balance (1008). The generic credit cooldown now holds minimax (about 1h left) and attempt 3 runs on claude.
  • Both codex models failed. codex:gpt-5.4 holds a 6d22h persistent model cooldown, gpt-5.4-mini about 3h.

/opt/homebrew/var/log/orch.error.log is 0 bytes, so there is nothing to refile from it.

Operational health

Active cooldowns: codex:gpt-5.4, codex:gpt-5.4-mini, kimi (agent_error), kimi:haiku, kimi:opus (silence_detected), minimax (credit exhaustion), minimax:haiku, opencode (agent_error), opencode/nemotron-3.5-lightning-free.

  • Most agents are cooled at the same time. The log shows degraded mode: using sequential dispatch with healthy_agents=1 and many routed agent/model entered cooldown after routing, deferring dispatch to next tick warnings. The generic mechanism works as designed, but throughput drops to what claude can carry.
  • A slow tick of 43 s was logged. This follows the sequential dispatch and the many deferred dispatches in one tick.
  • The service runs an older release than main. Fixes merged today are not live yet. This is expected under the operator-only upgrade policy.

Stuck tasks

TaskStatusNote
#3658 (copy-on-write worktrees)blockedresearch task, needs a human decision
#3650, #3651blockedblocked after attempt 1, check block_reason
#3649in_reviewattempt 3
#3652, #3655needs_reviewwaiting for review dispatch
#3653in_progressattempt 2
#3645, #3646, #3647, #3656, #3657routedwaiting out cooldowns

Routing accuracy

Routing follows the cooldown state. Tasks were routed to cooled kimi and opencode models and then deferred at dispatch. This is the settled behavior (re-check at dispatch, no per-task timer). No model was found failing silently beyond the kimi:opus silence cooldown, which is the detection working.

Open issues

11 open: #3645 to #3651, #3653, #3655 to #3658. They cover credit and rate-limit cooldown classification (#3647, #3649, #3650), review agent selection (#3645), test and build hygiene (#3646, #3653, #3656, #3657), a set_var race (#3655) and docs (#3651). Every problem seen in this review maps to one of them or to a settled policy, so no new issues were filed.

Priorities for tomorrow

  1. Clear the routed backlog (#3645, #3646, #3647, #3656, #3657) as cooldowns expire.
  2. Review the needs_review PRs (#3650, #3651, #3655, #3657, #3658) and unblock #3650 and #3651 after checking the block reason.
  3. Fix the cooldown classification bugs (#3647, #3649, #3650). They decide how long an agent stays out of the pool.
  4. Operator: top up minimax credit, or accept that it stays cooled.

← All updates