Lab it, learn it

You’ve built the Simple cluster and watched what happens when the only etcd node or the single master goes down. Now it’s time to scale to 11 machines and eliminate every single point of failure — without melting your laptop. This post is the conceptual overview of the OrbStack HA cluster — what the 11 machines do, what production problems they solve, and why OrbStack is the right choice when you want full HA on a resource budget.

When you’re ready for the full technical walkthrough, the OrbStack HA deep dive covers the shared-kernel architecture, networking quirks, and the complete deployment flow. The same HA architecture is also available with UTM for maximum production realism and Vagrant for declarative infrastructure-as-code. For a side-by-side look at what each HA tool offers at the overview level, see the UTM HA overview and the Vagrant HA overview.

From 6 Machines to 11 — What Changes and Why

The Simple cluster runs 6 lightweight machines: vault, jump, one etcd node, one master, and two workers. It works, but it has three deliberate weak points. The HA cluster adds 5 machines to eliminate all of them — and OrbStack’s shared-kernel approach means the additional machines barely register on your Mac’s resource meters.

HAProxy (new). A dedicated TCP load balancer that sits in front of the two master nodes. Workers and kubectl commands connect to HAProxy on port 6443, which routes traffic to whichever master is healthy. If one master goes down, the other handles all traffic automatically. This is what makes the control plane fault-tolerant.

etcd-1, etcd-2, etcd-3 (was 1 node). A 3-node etcd cluster with Raft consensus. Any one node can fail and the cluster keeps running because 2 out of 3 still form quorum. This is the most critical HA upgrade — etcd holds every piece of cluster state, and losing it means losing everything.

master-1, master-2 (was 1 node). Two control plane nodes running kube-apiserver, kube-controller-manager, and kube-scheduler. The controller-manager and scheduler use leader election for automatic failover — only one is active at a time.

worker-1, worker-2, worker-3 (was 2 nodes). A third worker for pod distribution experiments, node drains, and scheduling constraints.

Vault and Jump remain unchanged — Vault manages the 3-tier PKI hierarchy, jump is the bastion/Ansible controller.

What You’ll Actually Learn

The HA cluster teaches concepts that a Simple cluster physically cannot demonstrate. These are the same lessons regardless of tool — the Kubernetes behavior is identical across UTM, Vagrant, and OrbStack.

How etcd quorum works in practice. Kill one etcd node and the cluster continues. Kill two and it stops — quorum lost. You’ll understand why production clusters use odd numbers of etcd nodes and why 3 is the minimum for meaningful fault tolerance.

How control plane leader election works. Find the active controller-manager, kill that master, and watch the standby take over. This is how Kubernetes survives maintenance windows and node failures without missing a beat.

How load balancers front the API server. HAProxy does L4 TCP forwarding with health checks. Understanding this pattern transfers directly to cloud load balancers in managed Kubernetes environments.

How shared-kernel environments handle HA at scale. This is OrbStack-specific. With 11 machines sharing one kernel, you encounter constraints that full VMs don’t have: zram swap that can’t be disabled (kubelet sets failSwapOn: false), shared sysctl parameters (kube-proxy uses conntrack.maxPerCore: 0), and dual IPs on a single interface where auto-detection might grab the wrong address. Learning to configure around these constraints builds the habit of explicit configuration over implicit defaults — a habit that serves you well in any environment. The UTM HA cluster avoids all of these with full kernel isolation; the Vagrant HA cluster has its own dual-NIC variant.

How certificate SANs expand for HA. The API server cert gains SANs for both master IPs, the HAProxy IP, and the service cluster IP. Certificate planning before deployment is essential across all three tools.

Why OrbStack for HA

OrbStack’s lightweight architecture becomes its biggest advantage at HA scale. Adding 5 more machines barely changes the experience on your Mac.

Minimal resource overhead. The 11-machine HA cluster uses roughly 10.6 GB of disk total. Memory is shared dynamically rather than pre-allocated per machine. Compare this to UTM’s 38 GB RAM and 300 GB disk, or Vagrant’s 42 GB RAM. If your Mac has 16 GB of memory, OrbStack is likely the only tool that can run a full 11-machine HA cluster at all.

Instant machine creation. All 11 machines are created in about 30 seconds with orb create and cloud-init configs. UTM needs several minutes for VM boot, and Vagrant’s vagrant up takes about 1 minute 42 seconds. The faster you can create and destroy, the more experiments you run.

Cool and quiet operation. No fan spin-up, no thermal throttling. You can leave the full HA cluster running while doing other work on your Mac. UTM’s 11 full VMs will noticeably warm up any MacBook.

Cloud-init native. OrbStack passes cloud-init configs directly through orb create — no ISOs to generate (like UTM) and no shell provisioner workarounds (like Vagrant). The cloud-init YAML files in the repo are the cleanest provisioning approach across all three tools.

The tradeoff is production realism. OrbStack machines share the host kernel rather than booting their own. For most Kubernetes learning — deployments, services, RBAC, monitoring, cert management — this is indistinguishable from a full VM. The edge cases (custom kernel modules, kernel-level security policies) are where differences surface. If full kernel isolation matters, the UTM HA cluster and Vagrant HA cluster both provide it.

Deployment time is about 7 minutes 26 seconds — faster than Vagrant (8m 10s) but slower than UTM (6m 13s). The slowdown is file transfer: distributing binaries and certificates across 11 machines is noticeably slower on OrbStack than on the QEMU-backed tools.

Failure Experiments to Try

The whole point of HA is fault tolerance. Break things deliberately.

Kill one etcd node. Stop the etcd service on one machine. Check cluster health — two nodes healthy, one unreachable. Deploy a pod. It succeeds.

Kill two etcd nodes. Stop a second. Pod deployments hang. Quorum is lost.

Kill the active master. Find the controller-manager leader, stop that master. The standby wins leader election within seconds. HAProxy re-routes.

Delete and recreate a machine. orb delete worker-3 followed by orb create ubuntu noble worker-3 with the cloud-init config. Re-run the worker playbook from jump. This cycle takes under a minute on OrbStack — dramatically faster than recreating a UTM or Vagrant VM — making it the best tool for rapid iteration on failure recovery.

Who Should Build This

The OrbStack HA cluster is the right choice if you’ve completed the OrbStack Simple cluster and want to experience HA without upgrading your hardware, have a Mac with 16 GB RAM where UTM and Vagrant can’t fit 11 VMs, want the fastest create-break-destroy-rebuild cycle for experimentation, and don’t need full kernel isolation for your learning goals.

If production realism and full kernel isolation matter more, see the UTM HA overview. If infrastructure-as-code and Vagrantfile-driven management are your priority, see the Vagrant HA overview.

What’s Next

Ready for the technical details? The OrbStack HA deep dive covers the shared-kernel architecture, dual-IP networking, swap handling, and deployment timing for every phase. The Vault PKI deep dive covers the certificate authority hierarchy shared across all three projects. And the Learning Path maps the full progression across all three tools.

Big tech, small lab. One reel at a time.