Lab it, learn it

The comparison post showed that OrbStack is the most resource-efficient way to run multi-node Kubernetes on a Mac. This post goes deeper — walking through every layer of the k8s-orbstack-ha-homelab project, from lightweight Linux machines to a fully automated HA cluster with Vault PKI on Apple Silicon.

The goal: run bash scripts/k8s-orbstack-ha-homelab.sh and walk away. About seven and a half minutes later, you have a production-grade cluster with a 3-node etcd cluster, 2 control plane masters behind HAProxy, 3 workers, HashiCorp Vault with a 3-tier PKI CA hierarchy, and a bastion jump server — all running on your MacBook without the fan spinning up.

New to OrbStack on Apple Silicon? Start with the OrbStack Simple cluster — the same architecture with 6 machines instead of 11, deployed in under 6 minutes. Once you’re comfortable with the fundamentals, come back here for the full HA setup.

The Architecture

The cluster runs 11 machines across 6 roles, all created by OrbStack using the orb create command with cloud-init for automated provisioning. Each machine connects on the 192.168.139.0/24 subnet, with the network prefix auto-detected from OrbStack’s configuration at deploy time.

The key difference from the UTM version and Vagrant version: OrbStack machines are not full VMs with their own kernels. They share the host kernel (6.17.8-orbstack) and run as lightweight Linux environments on top of macOS’s virtualization layer. The result is dramatically lower resource consumption — each machine uses only 1.3–3.0 GB of disk compared to the 20–40 GB that UTM and Vagrant VMs allocate.

Why OrbStack?

OrbStack is a fast, lightweight alternative to Docker Desktop for macOS that also provides Linux virtual machines. Unlike UTM (which uses QEMU to create full VMs) and Vagrant (which uses QEMU through the vagrant-qemu plugin with socket_vmnet for networking), OrbStack takes a fundamentally different approach. Its Linux machines share the host kernel rather than booting their own — they’re lightweight environments, not full VMs with separate kernels and their own network stacks.

For Kubernetes learning, this tradeoff is almost entirely in your favor. Deployments, Services, RBAC, Helm charts, monitoring, CI/CD pipelines — all of this behaves identically whether you’re running on a full VM or an OrbStack machine. The 5% of cases where differences surface — custom kernel modules, kernel-level security policies, low-level syscall behavior — are edge cases that most learners and even many production users never encounter.

The practical benefits are significant: machines are created in seconds (not minutes), memory isn’t pre-allocated per machine, the laptop stays noticeably cooler, and the total disk footprint for 11 machines is roughly 10.6 GB instead of 200+ GB. OrbStack is free for personal use.

Components and Versions

The cluster uses the exact same component versions as the UTM and Vagrant projects — the only thing that changes is the virtualization layer:

Cloud-Init on OrbStack — No ISOs Required

All three projects use cloud-init for per-machine provisioning — SSH keys, static IPs, hostnames, role-specific packages. But the delivery mechanism is different for each tool, and OrbStack’s approach is the simplest.

UTM requires generating a cloud-init ISO image for each VM — creating meta-data, user-data, and network-config files, then packaging them into a NoCloud datasource ISO that’s attached as a virtual drive. Vagrant uses its built-in shell provisioner, bypassing cloud-init entirely. OrbStack passes cloud-init configs directly through its orb create command — no ISOs to generate, no provisioner scripts to maintain.

The cloud-init YAML files live in the cloud-init/ directory of the repo, one per machine. Each config sets the hostname, creates the k8s user with passwordless sudo, injects the SSH public key, writes /etc/hosts with entries for all 11 machines, configures the static IP via Netplan, and installs role-specific packages. Workers get socat and conntrack (required by kubelet), the jump server gets Ansible and the Vault CLI, and so on.

The Deploy Script — What Happens End to End

The k8s-orbstack-ha-homelab.sh script orchestrates the entire deployment. Here’s the high-level flow:

Phase 1 — Machine Creation (runs on Mac): The script calls orb create ubuntu noble {vm-name} for each of the 11 machines, passing the corresponding cloud-init config. Machines appear in seconds. The network prefix is auto-detected from orb config show | grep network.subnet4. The script then configures the Mac’s /etc/hosts and sets up the SSH config for the jump server.

Phase 2 — Jump Server Setup (runs on Mac → jump): The SSH private key is copied to jump. The project’s ansible/ directory is synced via SCP. An SSH config is written on jump with entries for all 10 other machines. Kubernetes binaries (pre-downloaded on the Mac in the background) are cached on jump so Ansible roles don’t need to download them inside each machine.

Phase 3 — Ansible Deployment (runs on jump): The script SSHes into jump and executes each Ansible playbook in sequence: Vault bootstrap and PKI setup, certificate issuance for all components, etcd cluster deployment, HAProxy configuration, control plane deployment, worker node deployment, and Calico CNI installation.

Every step is timed individually, and a summary table is printed at the end. The script supports --ansible-only (skip machine creation, run playbooks only) and --from-step N (resume from a specific step).

The Bastion Architecture

The same bastion pattern used in the UTM and Vagrant projects applies here. The Mac host connects only to the jump server. Every other machine is accessed through jump via SSH ProxyJump. Ansible playbooks run on the jump server, not from the Mac. kubectl also runs from jump — after deployment, ssh jump gives you immediate cluster access.

This mirrors real-world bastion patterns and keeps the Mac clean — no Ansible inventory pointing to ephemeral machine IPs, no SSH config pollution with 11 different hosts.

ssh jump                    # Direct access to bastion
ssh haproxy                 # Via jump (ProxyJump configured automatically)
ssh vault
ssh etcd-1
ssh master-1
ssh worker-1
# ... etc

Networking — The Dual-IP Quirk

OrbStack’s networking has a characteristic worth understanding well. Each machine gets a single network interface (eth0), but that interface has two IP addresses assigned to it. One is the static IP on the 192.168.139.0/24 subnet (configured via cloud-init and used in the Ansible inventory), and the other is a dynamically assigned IP that OrbStack uses internally for its networking layer.

Running ip addr show eth0 on any machine shows both addresses on the same interface. This is different from Vagrant’s dual-NIC approach — Vagrant creates two separate interfaces (eth0 for NAT, eth1 for vmnet), while OrbStack stacks two IPs on one interface. And it’s different from UTM, where each VM has a single interface with a single IP — the cleanest setup.

In practice, this doesn’t cause issues for Kubernetes because all cluster components — kubelet, etcd, kube-apiserver — are explicitly configured to bind to the static 192.168.139.x address. But if you run hostname -I or let a service auto-detect its IP, it might pick up the wrong one. The Ansible playbooks handle this by always specifying the exact bind address rather than relying on auto-detection.

The Vault PKI Pipeline

The Vault setup is identical to the UTM and Vagrant versions — the same Ansible roles, same configuration. HashiCorp Vault is installed on a dedicated machine and initialized with Shamir’s Secret Sharing. A 3-tier CA hierarchy is configured:

Root CA (pki_root)
└── Intermediate CA (pki_int)
    ├── Kubernetes CA (pki_kubernetes)
    │   ├── kube-apiserver
    │   ├── kube-controller-manager
    │   ├── kube-scheduler
    │   ├── kubelet (per-node)
    │   ├── kube-proxy
    │   ├── admin
    │   └── service-accounts
    ├── etcd CA (pki_etcd)
    │   ├── etcd server (per-node)
    │   ├── etcd peer (per-node)
    │   └── etcd client
    └── Front Proxy CA (pki_front_proxy)
        └── front-proxy-client

Each CA has specific PKI roles that control what certificates can be issued — allowed domains, key usage, IP SANs, and TTLs. This separation means a compromised etcd certificate cannot be used to authenticate to the Kubernetes API, and vice versa. For a detailed walkthrough of the PKI setup, see the Vault PKI deep dive.

Shared Ansible Roles — No Code Duplication

The OrbStack project includes the exact same Ansible roles as the UTM and Vagrant projects. The roles are not forked or modified — they’re identical code. What changes between UTM, Vagrant, and OrbStack is only the machine creation layer and the inventory files.

This means improvements to any role — a better etcd health check, a new certificate rotation task, a containerd config tweak — benefit all three projects immediately. The separation between “how machines are created” and “what runs inside them” is clean and intentional.

OrbStack-Specific Gotchas

Swap and kubelet. OrbStack machines share the host kernel, and OrbStack uses zram swap that cannot be disabled at the kernel level. Historically, the kubelet would refuse to start if swap was detected on a node. However, Kubernetes has supported running with swap present since v1.22 (alpha), with the feature graduating to beta in v1.28 and reaching GA in v1.34. This project uses Kubernetes 1.32.0, where the NodeSwap feature gate is beta and enabled by default. The kubelet configuration sets failSwapOn: false, which tells the kubelet to start normally even with swap present. The default swap behavior is NoSwap, meaning Kubernetes workloads won’t actually use swap — they just won’t be blocked from starting because of it. This isn’t a workaround; it’s the intended configuration for environments like OrbStack where swap can’t be turned off at the host level.

Slow file copy. File transfer to OrbStack machines is noticeably slower than UTM or Vagrant. Distributing K8s binaries, etcd binaries, and TLS certificates across 11 machines adds up. This is the primary reason OrbStack HA (7m 26s) is over a minute slower than UTM HA (6m 13s) despite machines starting almost instantly.

kube-proxy conntrack. The shared kernel means kube-proxy can’t modify certain sysctl parameters that require host-level privileges. The configuration uses conntrack.maxPerCore: 0 to avoid “permission denied” errors on shared kernel sysctl operations.

VS Code terminal routing. VS Code’s integrated terminal may not route traffic to OrbStack’s static IPs correctly. If SSH commands to OrbStack machines fail from VS Code but work from macOS Terminal, switch to Terminal for cluster management.

Calico pods in Init state. After deployment completes, Calico pods often show as ContainerCreating or Init:2/3. This is normal across all three tools — Calico needs a minute or two to fully initialize. Running kubectl get pods -A shortly after consistently shows everything Running.

Deployment Timing

From a cold start (no pre-existing machines) to a fully working HA Kubernetes cluster with all nodes Ready and Calico CNI installed:

Machine creation is where OrbStack shines — 11 machines in about 30 seconds compared to 1m 42s for vagrant up or the multi-minute UTM boot sequence. The slower file copy phase is what brings the total time above UTM’s 6m 13s. For comparison, Vagrant completes in 8m 10s.

Project Structure

k8s-orbstack-ha-homelab/
├── scripts/
│   ├── k8s-orbstack-ha-homelab.sh   # Full end-to-end deploy script
│   └── destroy-vms.sh               # Teardown all machines + cleanup
├── cloud-init/                       # Cloud-init configs (11 machines)
│   ├── jump.yaml                     #   Bastion with Ansible/Vault CLI
│   ├── haproxy.yaml                  #   HAProxy load balancer
│   ├── vault.yaml                    #   Vault server
│   ├── etcd-{1,2,3}.yaml            #   etcd cluster nodes
│   ├── master-{1,2}.yaml            #   Control plane nodes
│   └── worker-{1,2,3}.yaml          #   Worker nodes
├── ansible/
│   ├── ansible.cfg
│   ├── inventory/
│   │   ├── homelab.yml               # Cluster inventory
│   │   └── localhost.yml             # Mac-side inventory
│   ├── playbooks/
│   │   ├── k8s-orbstack-ha-homelab.yml  # Full orchestration
│   │   ├── vault-full-setup.yml      # Vault bootstrap + PKI
│   │   ├── k8s-certs.yml             # Certificate issuance
│   │   ├── etcd-cluster.yml          # etcd deployment
│   │   ├── haproxy.yml               # HAProxy deployment
│   │   ├── control-plane.yml         # Control plane components
│   │   ├── worker.yml                # Worker node components
│   │   └── ping.yml                  # Connectivity test
│   └── roles/                        # Shared roles (identical to UTM/Vagrant)
└── README.md

OrbStack vs UTM vs Vagrant — When to Choose OrbStack

OrbStack is the right choice when resource efficiency matters most. The machines collectively use a fraction of the RAM and disk that UTM or Vagrant require. The laptop stays cool even with 11 machines running. Create and destroy cycles are nearly instant.

The tradeoff is production realism. UTM and Vagrant create full VMs with their own kernels — genuine isolation, independent kernel modules, network policies that behave like bare metal. OrbStack machines share the host kernel. For most K8s learning — deployments, services, RBAC, helm, monitoring — this is indistinguishable from a full VM. Edge cases like custom kernel modules or kernel-level security policies are where differences surface.

Choose UTM when maximum production realism and fastest deployment time matter. Choose Vagrant when declarative, version-controlled infrastructure is the priority. Choose OrbStack when you want the easiest daily driver with the lowest resource footprint. For a detailed comparison with benchmarks, see the main comparison post.

Running Standalone Playbooks

Individual playbooks can be run from the jump server for targeted operations — re-issuing certificates, redeploying workers after a config change, or testing connectivity:

ssh jump
cd ~/k8s-orbstack-ha-homelab/ansible
# Test connectivity
ansible-playbook -i inventory/homelab.yml playbooks/ping.yml
# Re-issue certificates
ansible-playbook -i inventory/homelab.yml playbooks/k8s-certs.yml
# Redeploy workers only
ansible-playbook -i inventory/homelab.yml playbooks/worker.yml

Running It Yourself

Prerequisites: macOS with Apple Silicon (ARM64), OrbStack installed, and an SSH key at ~/.ssh/k8slab.key:

ssh-keygen -t ed25519 -f ~/.ssh/k8slab.key -C "k8s-homelab" -N ""

Then clone and deploy:

git clone https://github.com/labitlearnit/k8s-orbstack-ha-homelab.git
cd k8s-orbstack-ha-homelab
bash scripts/k8s-orbstack-ha-homelab.sh

After deployment, ssh jump from your Mac, then kubectl get nodes -o wide to see your cluster. To tear everything down:

bash scripts/destroy-vms.sh

Wrapping Up

OrbStack removes the biggest barrier to running a multi-node Kubernetes homelab: resource consumption. You get the same 11-machine HA architecture, the same Vault PKI, the same Ansible automation, and the same learning experience — just without needing to pre-allocate 38 GB of RAM or 300 GB of disk. The shared kernel means you’re not getting full VM isolation, but for learning Kubernetes end to end, that tradeoff is worth it. For the full roadmap from simple to HA across all three tools, see From Simple to HA: A Learning Path for Kubernetes on Apple Silicon.

The full source code is at github.com/labitlearnit/k8s-orbstack-ha-homelab. Star the repo if you find it useful, and feel free to open issues or PRs.

Big tech, small lab. One reel at a time.