Cluster

What it is

Orch can run on several machines as one cluster. One machine is the leader and makes every decision: routing, recovery, review, merge, jobs and the GitHub poll. The machines agree on the task state through Raft (openraft 0.9), so when the leader goes down another machine takes over with no manual step.

The cluster is off by default. Without a cluster section in ~/.orch/config.yml, orch runs on one machine exactly as before: no Raft, no listener, no network traffic.

The design and its trade-offs are in docs/specs/2026-10-26-multi-machine-cluster-design.md.

Not built yet

• Assigning tasks to machines. In a cluster today only the leader dispatches agents. A learner or a follower tends its own sessions and nothing else. Keep the machine that runs your agents as a voter that can lead, and do not make it a learner yet.
• Sending GitHub writes from a non-leader to the leader.
• Recovery and shutdown limited to the tasks a machine owns.
• Channels on the leader only. Every machine with a channels section polls its bots, and Telegram lets one client poll a bot at a time, so two machines would split the messages between them. Keep channels in the config of one machine.

Requirements

• Tailscale on every machine, with the tailscale CLI on PATH. Nodes find each other through tailscale status, and the cluster listens only on the machine's Tailscale address.
• The same orch version on every machine.
• On a machine that can lead: the GitHub token, the agent CLIs and the checkouts of your projects, as on a single machine today.

Roles

A decision needs a majority of voters, so voters must be machines that stay on. A machine that sleeps, like a laptop, should be a learner.

How the leader is chosen

The voters elect the leader. You do not set it, and openraft 0.9 cannot hand leadership over on request. You choose who can lead through the roles:

• Only a voter without witness: true can become leader.
• A witness votes but never starts an election, so it never leads.
• A learner never votes and never leads.

So to keep the leader on two always-on machines, make them the two voters that can lead and add a witness as the third vote. The leader stays leader until it stops or loses its connection to the majority. Then the other voter takes over within about two seconds.

Sizes

Two machines alone cannot fail over safely. When the link between them drops, neither can tell a crash from a network split, so either both keep deciding or both stop. No consensus protocol avoids this: they all need a majority of votes. The witness is the cheapest third vote. Any always-on machine on the tailnet can host it.

Configure

Generate one secret and use the same value on every machine:

openssl rand -hex 32

Find each machine's node name. It is the first label of its MagicDNS name, in lowercase, as tailscale status shows it. A machine named "Node A" in the admin console is node-a:

tailscale status --json | jq -r '.Self.DNSName' | cut -d. -f1

Add the same section to ~/.orch/config.yml on every machine. Only witness differs between machines. Orch reads the secret as written and does not expand environment variables, so the file now holds a secret: keep it readable by your user only (chmod 600 ~/.orch/config.yml).

cluster:
  secret: "<the value from openssl>"
  port: 7420
  voters: [node-a, node-b, node-c]
  witness: false   # true on the witness only

You list no IP addresses. Each machine reads its own address and the voters' addresses from tailscale status, including voters that are offline.

Allow the port on the tailnet

The secret already keeps other machines out, and an ACL also stops them from reaching the port at all. In the tailnet policy, allow cluster.port between the cluster machines only:

{
  "hosts": {
    "node-a": "100.64.0.1",
    "node-b": "100.64.0.2",
    "node-c": "100.64.0.3",
    "node-d": "100.64.0.4"
  },
  "acls": [
    {
      "action": "accept",
      "src": ["node-a", "node-b", "node-c", "node-d"],
      "dst": ["node-a:7420", "node-b:7420", "node-c:7420", "node-d:7420"]
    }
  ]
}

Use your own Tailscale IPs, from tailscale status. If your policy has no other rules, keep a rule that allows the rest of your traffic.

Start it

1. Restart the voters first, the witness included: orch service restart on each. Each voter that can lead initializes the cluster with the full voter list, and they elect a leader once a majority is up.
2. Restart the learners. Every 30 seconds a learner looks for a member on the tailnet and asks to join, until one adds it.
3. Check the result on any machine:

orch nodes

NODE                 ADDRESS                ROLE     STATE
node-a               100.64.0.1:7420        voter    leader
node-b               100.64.0.2:7420        voter    this machine
node-c               100.64.0.3:7420        voter

A witness shows as a voter. orch nodes prints the members of the cluster as this machine's service sees them, so the service must be running.

When a machine goes down

Security

• Every request between machines carries the secret, and the listener binds only to the Tailscale address.
• When a machine looks for members, it proves it knows the secret with an HMAC and checks the same proof in the answer, so it never sends the secret to a machine that has not proven it. Both proofs cover the address dialed, and the answer must carry the name Tailscale gives that address, so a host that relays a real member's answer is rejected.
• Only machines of your own tailnet count. A device shared in from another tailnet is ignored, and a voter name that two machines share stops startup with an error.
• The CLI reaches its local service on 127.0.0.1 with a token from ~/.orch/state/cluster.port, a file only your user can read.

Files

Turn it off

Remove the cluster section and run orch service restart. The machine goes back to writing straight to its own database. ~/.orch/raft.db stays on disk and is not read.