Lab it, learn it

Security in Kubernetes isn’t a feature you bolt on after the cluster works. It’s baked into the architecture — every component authenticates with TLS certificates, every connection between nodes is encrypted, and the trust model is enforced by separate Certificate Authorities with distinct scopes. The problem is that most tutorials skip all of this. They use kubeadm (which hides certificates behind auto-generation) or generate a flat set of certs with openssl (which ignores CA hierarchy entirely). This project doesn’t.

Across all six repos — UTM, Vagrant, and OrbStack, each in Simple and HA variants — the security model is identical. Every cluster uses HashiCorp Vault for PKI, a three-tier CA hierarchy, mTLS on every internal connection, etcd encryption at rest, and a bastion host for network access control. The difference between Simple and HA is infrastructure scale, not security posture.

The Security Model at a Glance

The homelab clusters implement five distinct security layers, each independent from the others:

Layer 1: Network access control (Bastion pattern). No cluster node is directly accessible from the Mac. All SSH access flows through a single jump server — the bastion host. The Mac’s ~/.ssh/config has one entry. The jump server’s SSH config has entries for every node. This is the same pattern used by AWS, GCP, and Azure for accessing private VPCs.

Layer 2: Transport encryption + mutual authentication (mTLS). Every connection between Kubernetes components uses TLS 1.3. But it’s not just encryption — it’s mutual TLS, meaning both sides present certificates. The API server verifies the kubelet’s identity. The kubelet verifies the API server’s identity. etcd verifies the API server’s identity before accepting writes. No anonymous connections, no one-way trust.

Layer 3: CA separation (three distinct trust domains). Certificates come from three different Certificate Authorities: the Kubernetes CA (for API server, kubelet, controller-manager, scheduler), the etcd CA (for etcd server, peer, and client certificates), and the Front Proxy CA (for the aggregation layer). A certificate issued by one CA cannot authenticate against another. See the Why Three CAs Matter section below.

Layer 4: Secrets encryption at rest. Kubernetes Secrets stored in etcd are encrypted with AES-256-CBC before being written to disk. The encryption key is managed by Vault and injected into the API server configuration at bootstrap. Raw etcd data on disk is ciphertext.

Layer 5: PKI lifecycle management (Vault). Certificates aren’t generated manually with openssl or cfssl and then copied around. They’re issued by HashiCorp Vault’s PKI secrets engine, which handles keypair generation, certificate signing, TTL enforcement, and audit logging. The same Vault instance that manages the PKI hierarchy also manages the etcd encryption key.

These layers are cumulative. Each one adds an independent barrier that an attacker would need to overcome. Even if someone compromises the jump server’s SSH access, they still need valid TLS certificates to talk to etcd or the API server. Even if they obtain a Kubernetes client certificate, they can’t use it to join the etcd cluster because etcd trusts a different CA. Even if they gain direct filesystem access to an etcd node, the Secrets on disk are AES-encrypted ciphertext — useless without the key.

etcd Encryption at Rest

The short version: the API server’s EncryptionConfiguration tells it to encrypt Secret resources using AES-256-CBC before writing to etcd. The encryption key comes from Vault, injected at cluster bootstrap via the vault-etcd-encryption-key.yml Ansible playbook. The result is that etcd data files on disk contain ciphertext, not plaintext Secrets.

The Vault PKI Hierarchy

The certificate infrastructure is the foundation that makes everything else work. The project uses a 3-tier CA hierarchy managed entirely by HashiCorp Vault:

Root CA (pki_root) — 365-day TTL, pathlen:2
└── Intermediate CA (pki_int) — 180-day TTL, pathlen:1
    ├── Kubernetes CA (pki_kubernetes) — 90-day TTL, pathlen:0
    ├── etcd CA (pki_etcd) — 90-day TTL, pathlen:0
    └── Front Proxy CA (pki_front_proxy) → 90-day TTL

The Root CA is offline after bootstrap — it signs only the Intermediate CA certificate and then its private key is not used again. The Intermediate CA signs the three leaf CAs. The leaf CAs issue all the component certificates. This structure means compromising a leaf CA doesn’t compromise the entire hierarchy.

The three leaf CAs are completely independent. The Kubernetes CA issues certificates for the API server, kubelet, controller-manager, scheduler, and admin kubeconfig. The etcd CA issues certificates for etcd server endpoints, etcd peers (Raft replication), and the API server’s etcd client connection. The Front Proxy CA issues the front-proxy-client certificate used by the API server’s aggregation layer.

Why Three CAs Matter

The single most important security decision in this project is CA separation. Three leaf CAs — Kubernetes, etcd, and Front Proxy — each with their own scope and trust boundary.

The reasoning is about blast radius. If all certificates came from one CA, a compromised certificate from any domain could potentially authenticate in any other domain. An etcd peer certificate could be presented to the Kubernetes API server. A kubelet client certificate could be used to join the etcd cluster. There’d be no cryptographic boundary between trust domains.

With separate CAs, each domain has its own root of trust. The API server is configured to trust the Kubernetes CA for client authentication and the etcd CA for its etcd connection. etcd is configured to trust only the etcd CA for peer and client authentication. The front-proxy uses its own CA so extension API servers can distinguish proxied requests from direct client requests.

This separation is what makes the “compromise one thing, compromise everything” scenario much harder. For a full analysis of what each CA is responsible for, what certificates it issues, and what would go wrong if they were combined, see Why Kubernetes Needs Three Separate CAs.

mTLS: Every Connection Authenticated

TLS in Kubernetes isn’t just encryption — it’s mutual authentication. In regular HTTPS (like visiting a website), only the server presents a certificate. The client verifies the server’s identity, but the server doesn’t verify the client’s. In mutual TLS (mTLS), both sides present certificates and both sides verify the other.

The homelab clusters use mTLS on every internal connection:

API server → etcd: The API server presents a client certificate signed by the etcd CA (apiserver-etcd-client.crt). etcd verifies it against the etcd CA. etcd presents its server certificate. The API server verifies it against the etcd CA. Both sides are authenticated.

kubelet → API server: The kubelet presents a client certificate signed by the Kubernetes CA. The API server verifies it. The API server presents its server certificate. The kubelet verifies it. Both sides authenticated, encrypted with TLS 1.3.

etcd peer replication: In HA clusters, etcd nodes replicate state to each other via Raft. Each peer presents a certificate signed by the etcd CA. Each peer verifies the other’s certificate. This is separate from the client interface — etcd peers use different certificates from the ones the API server uses to talk to etcd.

kubectl → API server: The admin kubeconfig contains a client certificate signed by the Kubernetes CA. When you run kubectl get pods, your client presents this certificate. The API server verifies it’s signed by the Kubernetes CA and grants access based on the certificate’s CN (Common Name) and O (Organization) fields, which map to RBAC roles.

The Bastion Layer

Before any certificate comes into play, there’s a network access layer. Every repo in this project — all six — includes a dedicated jump server that acts as the single entry point to the cluster network. The Mac connects to jump. Jump connects to everything else. No cluster node is directly reachable from outside.

This is the bastion host pattern. It reduces the external attack surface to one server. If you wanted to add firewall rules, IP whitelisting, multi-factor authentication, or intrusion detection, you’d do it on one machine instead of every node in the cluster. In cloud environments, this maps directly to jump servers in private subnets with security group rules that only allow SSH from specific IPs.

The bastion and TLS layers work together as defense in depth. SSH access to a node doesn’t give you etcd access — you’d still need valid client certificates. Certificate possession doesn’t give you network access — you’d still need to reach the node through the bastion. Each layer is an independent barrier.

See The Bastion Server Pattern for a deep dive into SSH ProxyJump configuration, security group design, and why this pattern matters even in a homelab context.

Certificate Lifecycle

Certificates aren’t static. They expire, they can be compromised, and they need to be rotated. The Vault-based approach handles this differently from the typical openssl workflow:

Issuance. Every certificate is issued by calling Vault’s PKI API through the k8s-certs Ansible role. Vault generates the keypair, signs the certificate with the appropriate leaf CA, and returns the cert + key + CA chain. The role deploys these to standardized paths on each node (/etc/etcd/pki/ for etcd nodes, /etc/kubernetes/pki/ for masters and workers).

Rotation. Re-running the k8s-certs playbook issues fresh certificates from Vault and overwrites the old ones. The same API call that issued the original cert issues a replacement. No manual openssl commands, no copying files between machines — the Ansible role handles distribution to all nodes in parallel with forks=12.

Revocation. Vault has built-in CRL (Certificate Revocation List) support. If a certificate is compromised, you revoke it through the Vault API and clients that check the CRL will reject it. With openssl, you’d need to maintain a CRL manually — which almost no one does in practice.

Audit. Vault logs every certificate issuance with timestamps, the requesting identity, and certificate details. With openssl, your audit trail is whatever your shell history shows.

The Vault PKI deep dive covers the full comparison between Vault and openssl/cfssl, including the tradeoffs — Vault adds operational complexity (initialization, unsealing, another service to run), but it’s the same tool used in production environments.

What the Security Architecture Looks Like in Practice

Here’s the concrete flow of how security layers interact when you deploy a workload on one of these homelab clusters:

# Step 1: SSH through the bastion
ssh jump
# Step 2: kubectl uses the admin kubeconfig (client cert signed by Kubernetes CA)
kubectl create deployment nginx --image=nginx
# What happens behind the scenes:
# - kubectl presents admin client cert → API server verifies against Kubernetes CA ✓
# - API server writes to etcd using etcd client cert → etcd verifies against etcd CA ✓
# - etcd replicates to peers using peer certs → peers verify against etcd CA ✓
# - Scheduler reads from API server (controller-manager cert) → API server verifies ✓
# - Kubelet on worker pulls the pod spec (kubelet client cert) → API server verifies ✓
# - Every connection: encrypted + mutually authenticated

Verifying the Security Setup

These commands work on any of the HA clusters (UTM, Vagrant, OrbStack). SSH to the jump server first — all cluster access goes through the bastion.

# Verify etcd encryption at rest (Secrets are ciphertext on disk)
ETCDCT0_API=https://192.168.64.11:2379 \
  ETCDCTL_CERT_FILE=/etc/etcd/pki/etcd-server.crt \
  ETCDCTL_KEY_FILE=/etc/etcd/pki/etcd-server.key \
  ETCDCTL_CA_FILE =/etc/etcd/pki/etcd-ca.crt \
  etcdctl get /test-key --print-value-only | xxd | head
# Should start with \"k\x8\x00\" (encryption prefix), not plaintext
# Verify the certificate hierarchy
openssl verify -CAfile /etc/vault/pki/root_ca.crt \
  /etc/vault/pki/intermediate_ca.crt
openssl verify -CAfile <(cat /etc/vault/pki/root_ca.crt /etc/vault/pki/intermediate_ca.crt) \
  /etc/kubernetes/pki/kubernetes.crt
# Verify mTLS is actually enforced (anonymous request should fail)
curl -k https://192.168.64.11:6443/api/v1/nodes
# Returns 401 Unauthorized -- muTLS is enforced
# Verify etcd peer TLS
openssl s_client -connect 192.168.64.12:2380 \
  -cert /etc/etcd/pki/etcd-peer.crt \
  -key /etc/etcd/pki/etcd-peer.key \
  -CAfile /etc/etcd/pki/etcd-ca.crt 2>&1 | grep Verify
# Should return: Verify return code: 0 (ok)

How This Applies Across All Six Repos

The security architecture is identical across UTM, Vagrant, and OrbStack. The same Ansible roles and playbooks — vault-bootstrap, vault-pki-setup, k8s-certs, vault-etcd-encryption-key, and the control plane role — run on all three platforms without modification. The only thing that changes between projects is the Ansible inventory file — different IP ranges for different virtualization tools:

# UTM inventory
master-1 ansible_host=192.168.64.11
# Vagrant inventory
master-1 ansible_host=192.168.105.11
# OrbStack inventory
master-1 ansible_host=192.168.139.11

Same Vault setup. Same PKI hierarchy. Same certificate paths. Same encryption configuration. The security model is substrate-independent.

The Simple clusters (no Vault, no HA) use a flat CA structure with openssl-generated certificates. They’re intentionally less secure — the point is to show what the HA clusters add. See From Simple to HA for the comparison.

Why a Homelab Needs This

The honest answer is: for security purposes, a homelab doesn’t need this. A cluster running on a Mac on a home network isn’t protecting production workloads.

But that’s not the point. The point is that every production Kubernetes cluster does need this — and if you’ve never seen how PKI actually works in Kubernetes, how CA separation prevents lateral movement, how Vault integrates with certificate issuance, or how etcd encryption is configured, you won’t know what to look for when you’re operating a real cluster.

Managed Kubernetes (EKS, GKE, AKS) hides all of this. The control plane is a black box. You don’t see the certificates. You don’t configure the CA hierarchy. You don’t know if etcd is encrypted or how. This project makes everything visible. You can inspect every certificate, trace every trust relationship, and understand exactly why each security decision was made.

When you do encounter PKI issues in a managed cluster — expired certificates, misconfigured RBAC, etcd backup/restore problems — having built this from scratch means you’ll know what you’re looking at.

Deep Dive Posts

Each security layer has its own detailed post:

• Vault PKI: 3-Tier CA the Right Way — Root CA, Intermediate CA, leaf CAs, pathlen constraints, TTL strategy

• Why Kubernetes Needs Three Separate CAs — CA separation, blast radius analysis, what breaks when you combine CAs

• The Bastion Server Pattern — SSH ProxyJump, security group design, why it matters even in a homelab

Getting Started

All six repos are on GitHub. The HA clusters (UTM, Vagrant, OrbStack) all implement the full security stack described here. Pick the virtualization tool you have available:

• k8s-utm-vault-ha — 11 VMs on UTM, full Vault PKI, 3-node etcd

• k8s-vagrant-vault-ha — same architecture, Vagrant + vagrant-qemu

• k8s-orbstack-vault-ha — same architecture, OrbStack Linux machines

For the full learning path from Simple clusters (no Vault, single etcd) to HA clusters (Vault PKI, 3-node etcd, bastion access), see From Simple to HA: A Learning Path for Kubernetes on Apple Silicon.

Big tech, small lab. One reel at a time.