Lab it, learn it

Ever wanted to build a production-grade Kubernetes cluster from scratch — no kubeadm, no managed services, just raw binaries and real certificates? That’s exactly what this project does. A single shell script spins up 11 Ubuntu 24.04 VMs on UTM (QEMU), wires them together with HashiCorp Vault PKI, and deploys a fully functional HA Kubernetes cluster on your MacBook.

In this post, I’ll walk through every one of the 17 steps in the deployment script — what each step does, why it matters, and the design decisions behind it. The full source is on GitHub: k8s-utm-ha-homelab.sh

The Architecture at a Glance

Before diving into the steps, here’s what we’re building: 11 VMs across 5 roles, all on a single Mac using UTM’s shared networking on the 192.168.64.0/24 subnet.

Total footprint: 22 vCPUs, 38 GB RAM, 300 GB disk. The Mac host acts as the 192.168.64.1 gateway, and all inter-VM communication happens over UTM’s shared network bridge.

The Script Preamble — Variables and Helpers

The script starts by defining all the version pins and download URLs in one place. This is critical for reproducibility — every binary version is locked so you get the exact same cluster every time:

• Kubernetes 1.32.0 — kube-apiserver, controller-manager, scheduler, kubelet, kube-proxy, kubectl
• etcd 3.5.12 — the distributed key-value store backing the cluster
• containerd 1.7.24 + runc 1.2.4 — the container runtime
• Calico 3.28.0 — pod networking CNI

All binaries are ARM64 (aarch64) since we’re running on Apple Silicon. The VM definitions are stored as a simple colon-delimited array — name:ip_suffix:ram_mb:vcpu:disk_gb — making it trivial to add or remove nodes.

The script also defines helper functions for UUID generation (required for UTM’s plist config), random MAC address generation, and a header() function for pretty-printed step banners with color codes.

Step 1/17: Download the Cloud Image

The first step downloads the Ubuntu 24.04 Server Cloud Image for ARM64 — a ~600 MB qcow2 file that serves as the base disk for every VM. Cloud images are pre-built, minimal Ubuntu installations designed specifically for automated provisioning. Unlike full ISOs, they don’t require an interactive installer.

The script checks whether the image already exists and validates its size (must be over 500 MB) to guard against partial downloads. If valid, it skips the download entirely — this is the first of many idempotency checks that make the script safe to re-run.

Step 2/17: Generate SSH Key Pair

An Ed25519 SSH key pair is generated at ~/.ssh/k8slab.key. Ed25519 is chosen over RSA for its smaller key size, faster operations, and stronger security. The public key will be injected into every VM via cloud-init, giving the k8s user passwordless SSH access from the Mac host.

If the key pair already exists, this step is skipped. The private key gets chmod 600 and the public key gets chmod 644 — standard SSH security practice.

Step 2.5/17: Download K8s Binaries (Background)

This is a clever optimization. While the VMs are being created (Step 3) and booting (Steps 4-8), the script kicks off parallel background downloads of all Kubernetes binaries. The download_binaries() function spawns multiple curl processes simultaneously:

• 6 Kubernetes binaries (apiserver, controller-manager, scheduler, kubelet, kube-proxy, kubectl)
• etcd tarball
• containerd tarball
• runc binary
• Calico manifest YAML

Each download is wrapped in a background job with PID tracking. A .download-complete marker file signals completion. Already-cached binaries are skipped. This overlapping approach saves several minutes — you’re downloading binaries while cloud-init is configuring VMs.

Step 3/17: Create the 11 VMs

This is the heart of the provisioning engine. For each VM, the create_vm() function does the following:

1. Create the disk: The Ubuntu cloud image is copied and resized to the target disk size using qemu-img resize. Each VM gets its own independent qcow2 disk.

2. Generate cloud-init ISO: This is where the magic happens. The create_cloud_init_iso() function builds a NoCloud datasource ISO containing three files:

• meta-data — sets the instance ID and hostname
• user-data — the cloud-config YAML that provisions the VM
• network-config — static IP configuration using Netplan v2 syntax

The user-data is where things get interesting. Every VM gets a k8s user with passwordless sudo, the SSH public key, and a pre-populated /etc/hosts file with all 11 VM entries so they can resolve each other by name immediately. The runcmd section disables systemd-networkd-wait-online (which otherwise causes a 2-minute boot delay with static IPs) and runs growpart/resize2fs to expand the filesystem to fill the resized disk.

Role-specific packages are installed via cloud-init:

• jump — gets git, python3-pip, ansible, vault CLI, terraform, sshpass, and qemu-guest-agent
• haproxy — gets the haproxy package
• vault — gets unzip, curl, jq, gnupg
• workers — get socat, conntrack (required by kubelet)
• Others (etcd, masters) — skip package updates entirely for fastest possible boot

A boot optimization trick: the datasource config is set to [NoCloud, None], which tells cloud-init to skip probing for AWS/GCP/Azure metadata endpoints — saving 30+ seconds of boot time.

3. Build the UTM config.plist: UTM uses Apple plist files to define VM configurations. The script generates a complete config.plist for each VM with QEMU backend settings: aarch64 architecture, VirtIO disk and network interfaces, UEFI boot, shared networking mode with a unique MAC address, and the specified RAM/vCPU allocation.

The cloud-init ISO is converted from raw to qcow2 format because UTM expects disk images in that format. Both the OS disk and the cloud-init disk are attached as VirtIO drives.

Step 4/17: Restart UTM

After creating all 11 VM directories under ~/Library/Containers/com.utmapp.UTM/Data/Documents/, UTM needs to be restarted to detect the new VMs. The script kills UTM, waits 2 seconds, and re-opens it with open -a UTM. After a 5-second settling period, all VMs should appear in UTM’s sidebar.

Step 5/17: Update Mac /etc/hosts

The Mac’s /etc/hosts file is updated with entries for vault (192.168.64.11) and jump (192.168.64.12). This is the minimum needed — the Mac only talks directly to these two VMs. Vault needs a hostname for browser access to the UI, and jump is the SSH bastion. All other VMs are accessed through jump.

This step requires sudo and is idempotent — if the marker comment already exists in /etc/hosts, it’s skipped.

Step 6/17: Setup SSH Config

A Host entry is added to ~/.ssh/config for the jump server. This is what lets you simply type ssh jump from your Mac terminal. The config specifies the identity file, disables strict host key checking (since VMs are ephemeral and IPs get reused), sets UserKnownHostsFile /dev/null, and enables only publickey authentication for speed.

The design philosophy: the Mac only needs to know about jump. Once you’re on jump, you use its local SSH config (created in Step 9) to reach all other VMs.

Step 7/17: Start All VMs

Using utmctl start, the script boots all 11 VMs sequentially with a 2-second delay between each. utmctl is UTM’s command-line interface — it lets you start, stop, and manage VMs without touching the GUI. If a VM is already running, the error is caught gracefully and the script continues.

At this point, all 11 VMs are booting simultaneously. Cloud-init is running inside each VM, applying the hostname, user configuration, network setup, and role-specific packages. The VMs with package_update: false (etcd nodes, masters) boot in under 30 seconds. The jump server, which installs the most packages, takes the longest.

Step 8/17: Wait for VMs to Boot

The script enters a polling loop, checking SSH connectivity to each VM every 3 seconds. It uses the private key to attempt ssh k8s@{ip} "exit 0" with a 2-second connect timeout. As each VM becomes reachable, it’s marked ready with a timestamp.

There’s a smart early-exit condition: if jump is ready and at least 8 out of 11 VMs are up, the script proceeds rather than waiting for stragglers. The maximum timeout is 10 minutes (600 seconds) for UEFI boot, though most VMs are ready in under 2 minutes.

The output shows a real-time progress counter like [8/11 VMs ready] 45s elapsed... with each VM printed as it comes online.

Step 9/17: Configure Jump Server

With the jump server now running, the script configures it as the bastion host and Ansible controller. This involves several sub-steps:

1. Fix ownership — cloud-init sometimes creates the home directory as root; this ensures /home/k8s belongs to the k8s user
2. Copy the SSH private key — scp transfers k8slab.key to jump’s ~/.ssh/ directory
3. Create SSH config — a comprehensive SSH config is written with Host entries for all 10 other VMs, each with the correct IP, user, identity file, and relaxed host key checking
4. Copy project files — the entire ansible/ directory and execution_flow.txt are copied to ~/k8s-utm-ha-homelab/ on jump

Each sub-step has verification — after copying the key, the script checks that the file actually exists on jump. After creating the SSH config, it verifies the file was written. This defensive approach catches SCP failures early rather than letting them cascade into mysterious Ansible errors later.

Step 10/17: Connectivity Test

Now comes the validation. The script tests SSH connectivity in a two-hop pattern:

1. Mac → jump — direct SSH test
2. Mac → jump → each VM — for each of the remaining 10 VMs, the script SSHes to jump and from there SSHes to the target VM

This confirms the entire SSH fabric is working: Mac can reach jump, jump can reach every node, and the SSH keys/configs are correctly deployed. The results are printed as a scorecard: Results: 11/11 VMs reachable.

Step 10.5/17: Wait for Binary Downloads

Remember Step 2.5? Those background downloads have been running in parallel with Steps 3-10. Now the script waits for the download process to finish (if it hasn’t already). It checks the PID of the background process and polls until completion.

Once confirmed, all binaries are copied from the Mac to the jump server via SCP into /tmp/ cache directories:

• /tmp/k8s-binaries/ — all 6 Kubernetes binaries
• /tmp/etcd-cache/ — etcd tarball
• /tmp/containerd-cache/ — containerd tarball + runc
• /tmp/calico.yaml — Calico manifest

A .pre-cached marker file is created in each directory. The Ansible roles check for these markers — if binaries are pre-cached, they skip downloading them again inside the VMs, saving significant time (especially since the VMs don’t have fast internet).

Step 11/17: Setup Vault Environment

The jump server’s .bashrc is configured with Vault-related environment variables and helper functions:

• VAULT_ADDR is set to http://vault:8200
• A first-boot auto-bootstrap block checks for a ~/.vault-bootstrapped marker. On first SSH login, it automatically runs vault-full-setup.yml (which installs, initializes, and configures Vault with PKI). After success, the marker is created so it never runs again.
• VAULT_TOKEN is loaded from the Ansible-managed credentials file
• A vault-unseal helper function is defined — after a VM restart, Vault is sealed and needs the first 3 unseal keys to unlock. This function reads the keys from the credentials JSON and applies them automatically.

Step 12/17: Wait for Jump Cloud-Init

The jump server has the heaviest cloud-init workload: installing Ansible via pip, adding the HashiCorp APT repository, and installing vault CLI + terraform. This step polls until Ansible is available by running ssh jump 'which ansible-playbook' in a loop.

There’s a maximum wait of 10 minutes, with 10-second intervals between checks. If it times out, the script provides manual instructions to check cloud-init status. Once Ansible is confirmed, the required Ansible Galaxy collection (community.hashi_vault) is installed on jump.

Step 13/17: Vault Full Setup (Bootstrap + PKI)

This is the first Ansible playbook executed from the jump server. The vault-full-setup.yml playbook does two major things:

Vault Bootstrap: Installs HashiCorp Vault on the vault VM, starts the server in dev mode initially, then initializes it with Shamir’s Secret Sharing (5 key shares, 3 threshold). The unseal keys and root token are saved to .vault-credentials/vault-init.json on the jump server. The vault is then unsealed and the root token is used for all subsequent operations.

Vault PKI Configuration: Sets up a 3-tier Certificate Authority hierarchy — the crown jewel of this project’s security design:

• Root CA (365-day TTL, pathlen:2) — the trust anchor, only signs intermediate CAs
• Intermediate CA (180-day TTL, pathlen:1) — signs the leaf CAs
• Kubernetes CA (90-day TTL, pathlen:0) — issues all K8s component certificates
• etcd CA (90-day TTL, pathlen:0) — issues etcd server, peer, and client certificates
• Front Proxy CA (90-day TTL, pathlen:0) — issues the API aggregation certificate

Each CA has specific PKI roles defined in Vault that control what certificates can be issued — allowed domains, key usage, extended key usage, TTLs, and IP SANs. This separation means a compromised etcd certificate cannot be used to authenticate to the Kubernetes API, and vice versa.

Step 14/17: Issue and Deploy Certificates

With the PKI infrastructure in place, the k8s-certs.yml playbook issues certificates for every component and deploys them to the correct nodes. Running with forks=12, Ansible distributes certificates to all nodes in parallel.

The certificates issued include:

• etcd: server certs (with node-specific SANs), peer certs for inter-node communication, client certs for API server access, and healthcheck client certs
• Kubernetes: API server cert (with SANs for all master IPs, the cluster service IP, and hostnames), controller-manager cert, scheduler cert, admin cert, service account signing keypair, kube-proxy cert
• Kubelet: per-node server and client certificates for each worker and master
• Front Proxy: client cert for API aggregation

Certificates, keys, and CA bundles are placed in standardized paths on each node (e.g., /etc/etcd/pki/, /etc/kubernetes/pki/) with correct ownership and permissions.

Step 15/17: Deploy etcd Cluster + HAProxy

These two playbooks run in parallel since they’re independent:

etcd Cluster (etcd-cluster.yml): Deploys a 3-node etcd cluster with mutual TLS. The etcd binary is extracted from the pre-cached tarball, a systemd unit is created with all the TLS flags (client cert auth, peer cert auth, trusted CA files), and the cluster is bootstrapped with the initial-cluster flag pointing to all 3 members. The playbook waits for the cluster to form quorum and runs an endpoint health check.

HAProxy (haproxy.yml): Configures the HAProxy VM as a TCP load balancer for the Kubernetes API server on port 6443. It round-robins between master-1:6443 and master-2:6443 with health checks. This is what makes the control plane highly available — if one master goes down, HAProxy routes all traffic to the surviving master.

Step 16/17: Deploy Control Plane

The control-plane.yml playbook installs and configures the 3 core Kubernetes control plane components on both master nodes:

• kube-apiserver — configured with etcd endpoints (using client TLS), service account signing keys, the admission controllers, the Kubernetes CA, and authorization modes (RBAC + Node). The API server listens on 0.0.0.0:6443 and is configured to trust the front-proxy CA for aggregation.
• kube-controller-manager — connects to the local API server, manages the cluster CA, allocates pod CIDRs from 10.244.0.0/16, and handles service account token creation using the signing key pair.
• kube-scheduler — connects to the API server via kubeconfig and handles pod scheduling decisions across workers.

Each component runs as a systemd service. Kubeconfig files are generated with embedded client certificates for authentication. The playbook also creates the admin kubeconfig (used by kubectl) and deploys it to the jump server.

Step 17/17: Deploy Worker Nodes + Calico CNI

The final step deploys the worker plane and pod networking:

Worker Nodes (worker.yml): On each of the 3 worker nodes, the playbook installs:

• containerd — the container runtime, extracted from the pre-cached tarball. A config file is generated with the SystemdCgroup driver enabled (required for Kubernetes). runc is installed as the low-level OCI runtime.
• kubelet — the node agent that registers with the API server. Configured with the node’s TLS certificates, the cluster DNS address (10.96.0.10), and the pod CIDR. Runs as a systemd service with the kubelet configuration file.
• kube-proxy — handles iptables rules for Kubernetes Services. Configured in iptables mode with conntrack settings.

Calico CNI: Finally, the pre-cached calico.yaml manifest is applied via kubectl apply from the jump server. Calico provides pod-to-pod networking across nodes using the 10.244.0.0/16 CIDR. Once Calico pods are running, the nodes transition from NotReady to Ready.

The script finishes with a kubectl get nodes to show the final cluster state and prints the total execution time.

Design Decisions Worth Noting

Why a Jump Server? Instead of exposing every VM to the Mac, only the jump server is directly accessible. This mirrors real-world bastion host patterns. The Mac’s SSH config has a single entry; jump’s SSH config has entries for all 10 other nodes. This is both more secure and simpler to manage.

Why Vault for PKI? Many tutorials use openssl or cfssl to generate certificates. Using Vault provides a proper CA hierarchy with rotation capabilities, audit logging, and the ability to revoke and reissue certificates without rebuilding the cluster. It’s more work upfront but mirrors how production environments manage certificates.

Why Not kubeadm? This is deliberately “Kubernetes the Hard Way” style. Every binary is downloaded, every config file is written, every systemd unit is created from scratch. You understand exactly what’s running and why. When something breaks, you know where to look.

Why Background Downloads? On a typical home internet connection, downloading ~500 MB of binaries takes several minutes. By starting downloads before VM creation and running them in parallel, the binaries are usually ready before they’re needed — turning a sequential 20+ minute process into an overlapping one.

Why Pre-cached Binaries on Jump? The VMs use UTM’s shared networking which NATs through the Mac. Having 11 VMs all download the same binaries from the internet would be slow and wasteful. Instead, the Mac downloads once, copies to jump once, and Ansible distributes from jump to each node over the fast local network.

Running It Yourself

If you want to try this on your own Apple Silicon Mac, you’ll need UTM installed, ~38 GB of free RAM, and about 30 minutes of patience. Clone the repo and run:

git clone https://github.com/labitlearnit/k8s-utm-ha-homelab.git
cd k8s-utm-ha-homelab
./scripts/k8s-utm-ha-homelab.sh

Or if you prefer the Ansible-native approach:

cd ansible
ansible-playbook -i inventory/localhost.yml playbooks/k8s-utm-ha-homelab.yml --ask-become-pass

After deployment, ssh jump from your Mac, then kubectl get nodes to see your cluster.

Wrapping Up

Building a Kubernetes cluster the hard way teaches you things that managed services deliberately hide: how certificates flow between components, why etcd quorum matters, what kubelet actually does on each node, and how the API server ties it all together. Doing it with proper automation (cloud-init + Ansible + Vault PKI) means you can tear it all down and rebuild in under 30 minutes — the perfect feedback loop for learning.

The full source code is at github.com/labitlearnit/k8s-utm-ha-homelab. Star the repo if you find it useful, and feel free to open issues or PRs.

Big tech, small lab. One reel at a time.