Lab it, learn it

Most Kubernetes-the-hard-way tutorials generate all their certificates from a single CA. One root, one key, one trust domain for everything — API server, etcd, kubelets, front proxy. It works. The cluster comes up, TLS handshakes succeed, and everything looks green.

The homelab clusters in this project don’t do that. They use three separate leaf CAs — one for Kubernetes components, one for etcd, and one for the front-proxy aggregation layer — all managed through a 3-tier hierarchy in HashiCorp Vault. This post explains why that separation exists, what each CA is responsible for, what would go wrong if they were combined, and how the Vault PKI hierarchy enforces the boundaries.

This is a deep dive into one aspect of the security architecture that spans all six homelab repos. For the full Vault PKI automation — the five secrets engines, three Ansible roles, and certificate deployment — see the Vault PKI deep dive.

The Three CAs

The project’s PKI hierarchy has a single root CA and a single intermediate CA at the top, then branches into three leaf CAs at the bottom:

Root CA (pki_root)
└── Intermediate CA (pki_int)
    ├── Kubernetes CA (pki_kubernetes)
    ├── etcd CA (pki_etcd)
    └── Front Proxy CA (pki_front_proxy)

Each leaf CA has a specific scope — a defined set of certificates it can issue and a defined set of components that trust it:

Kubernetes CA (pki_kubernetes) signs certificates for the Kubernetes control plane and data plane: kube-apiserver, kube-controller-manager, kube-scheduler, kubelet (per-node server and client certs), kube-proxy, the admin client certificate, and the service account signing keypair. Any component that authenticates to the Kubernetes API server or that the API server authenticates to (other than etcd and extension API servers) uses certificates from this CA.

etcd CA (pki_etcd) signs certificates for the etcd cluster: per-node server certificates (presented to clients on port 2379), per-node peer certificates (used for Raft replication on port 2380), client certificates (presented by the API server when connecting to etcd), and healthcheck client certificates. See mTLS Between etcd Nodes Explained for how these certificates enable mutual TLS on every etcd connection.

Front Proxy CA (pki_front_proxy) signs a single certificate: the front-proxy-client cert used by the API server when it proxies requests to extension API servers (like metrics-server). This CA exists so extension API servers can verify that a request was proxied by the real API server, as opposed to coming from a regular client.

What Goes Wrong with a Single CA

The simplest way to understand why three CAs matter is to consider what happens with just one.

With a single CA, every certificate in the cluster shares the same root of trust. The API server’s certificate, the etcd peer certificates, the kubelet certificates, and the front-proxy certificate are all signed by the same CA key. This creates several concrete problems:

A compromised etcd certificate could authenticate to the API server. If all certs come from the same CA, and the API server is configured to trust that CA for client authentication, then any certificate signed by that CA is a valid client certificate for the API server. An etcd peer certificate — designed only for inter-node replication — could be presented to the API server as a client credential. Depending on the RBAC configuration, this might grant cluster-admin access.

A compromised Kubernetes component certificate could access etcd directly. The reverse is equally dangerous. If etcd trusts the same CA that signs kubelet and kube-proxy certificates, then any valid Kubernetes certificate could authenticate to etcd as a client. etcd doesn’t have RBAC — once you’re authenticated, you can read and write everything. Every secret, every deployment, every config map. A compromised worker node (which has a kubelet certificate) could bypass the API server entirely and modify cluster state in etcd.

Extension API servers can’t distinguish proxied requests from direct requests. The front-proxy-client certificate exists so extension API servers know the request came through the API server’s aggregation layer, not directly from a user. If the front-proxy cert is signed by the same CA as regular client certs, the extension API server has no way to tell the difference. This breaks the trust model for API aggregation.

CA key compromise is total cluster compromise. With one CA, a single key compromise lets an attacker mint certificates for any identity — forge an API server cert, create a fake kubelet, impersonate the controller manager, join the etcd cluster. With three CAs, compromising the etcd CA key is serious, but it only affects the etcd trust domain. The Kubernetes API and front proxy continue operating with their uncompromised CAs. The blast radius is contained.

How Components Use the CA Separation

The CA separation is enforced through configuration flags on each component. Every Kubernetes binary has explicit flags specifying which CA to trust for which type of connection.

kube-apiserver is the component that touches all three CAs:

# Kubernetes CA — for client authentication (kubelets, controllers, admin)
--client-ca-file=/etc/kubernetes/pki/ca.crt
# Kubernetes CA — API server's own TLS certificate
--tls-cert-file=/etc/kubernetes/pki/kube-apiserver.crt
--tls-private-key-file=/etc/kubernetes/pki/kube-apiserver.key
# etcd CA — for authenticating to etcd as a client
--etcd-cafile=/etc/kubernetes/pki/etcd-ca.crt
--etcd-certfile=/etc/kubernetes/pki/etcd-client.crt
--etcd-keyfile=/etc/kubernetes/pki/etcd-client.key
# Front Proxy CA — for proxying to extension API servers
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key

Notice the three separate -ca-file flags, each pointing to a different CA bundle. --client-ca-file uses the Kubernetes CA (so kubelets and controllers can authenticate). --etcd-cafile uses the etcd CA (so the API server can verify etcd’s server certificate). --requestheader-client-ca-file uses the Front Proxy CA (so extension API servers know which proxied requests are legitimate).

If these all pointed to the same CA file, the component would still work — but the trust boundaries would be gone.

etcd only knows about the etcd CA:

# Server-side TLS
--cert-file=/etc/etcd/pki/etcd-server.crt
--key-file=/etc/etcd/pki/etcd-server.key
--trusted-ca-file=/etc/etcd/pki/etcd-ca.crt
--client-cert-auth=true
# Peer TLS
--peer-cert-file=/etc/etcd/pki/etcd-peer.crt
--peer-key-file=/etc/etcd/pki/etcd-peer.key
--peer-trusted-ca-file=/etc/etcd/pki/etcd-ca.crt
--peer-client-cert-auth=true

Both --trusted-ca-file and --peer-trusted-ca-file point to the etcd CA. This means etcd only trusts certificates signed by its own CA — not the Kubernetes CA, not the Front Proxy CA. A kubelet certificate, a controller-manager certificate, or an admin certificate will be rejected. Only the etcd client certificate (deployed to master nodes specifically for API server → etcd authentication) will be accepted. The mTLS deep dive covers these flags in detail.

kubelet and other Kubernetes components only know about the Kubernetes CA:

# Kubelet kubeconfig — trusts the Kubernetes CA to verify the API server
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://haproxy:6443

Workers don’t even have the etcd CA bundle on disk — they never talk to etcd. Masters have the etcd CA bundle because the API server needs it, but workers only get the Kubernetes CA. This is least-privilege certificate distribution.

The Trust Boundaries Visualized

Think of the cluster as three trust domains, each with its own CA:

Trust Domain 1: Kubernetes Control Plane + Data Plane
  CA: pki_kubernetes
  Components: API server, controller-manager, scheduler, kubelets, kube-proxy, admin
  Boundary: Everything that authenticates to/from the Kubernetes API
Trust Domain 2: etcd Cluster
  CA: pki_etcd
  Components: etcd server, etcd peers, etcd clients (API server only)
  Boundary: Everything that reads/writes/replicates cluster state
Trust Domain 3: API Aggregation Layer
  CA: pki_front_proxy
  Components: API server (as proxy client), extension API servers
  Boundary: Proxied requests from the API server to extension services
Cross-domain connection:
  API server → etcd: Uses etcd CA (etcd client cert + etcd CA bundle)
  This is the ONLY bridge between trust domains 1 and 2.

The API server is the only component that spans trust domains. It has certificates from all three CAs. But it uses each certificate for a specific purpose — the Kubernetes cert for serving the API, the etcd client cert for connecting to etcd, and the front-proxy cert for proxying to extensions. No other component crosses boundaries.

How Vault Enforces the Separation

The CA separation isn’t just a configuration choice — it’s enforced by the Vault PKI infrastructure. Each CA is a separate Vault PKI secrets engine with its own CA private key, stored within Vault’s encrypted storage backend. There’s no API call that returns a CA private key — Vault simply doesn’t expose that endpoint. The keys exist on disk inside the storage backend, protected by Vault’s barrier encryption (which requires the unseal keys to access).

On top of CA separation, Vault PKI roles add another layer of policy enforcement:

# The etcd-server role on pki_etcd can only issue:
# - Certificates with etcd-related common names
# - Certificates with specific IP SANs (from the Ansible inventory)
# - Certificates with server auth + client auth extended key usage
# - Certificates with a maximum TTL of 720h
# Even with a valid Vault token, you can't use the etcd-server role to issue:
# - A certificate with a Kubernetes API server SAN
# - A certificate with a kubelet common name
# - A certificate with an arbitrary domain name

The roles are the second line of defense after CA separation. If someone obtains a Vault token, they can only issue certificates that match a role’s constraints. They can’t issue an etcd peer certificate from the Kubernetes CA engine, because the Kubernetes CA engine doesn’t have an etcd peer role. And they can’t issue an API server certificate from the etcd CA engine, because the etcd CA engine doesn’t have an API server role.

The Vault PKI deep dive covers the vault-pki-setup Ansible role that configures all five engines, creates the CA chain, and defines the roles. The three Ansible roles (vault-bootstrap, vault-pki-setup, k8s-certs) are shared across all three projects — UTM, Vagrant, and OrbStack — with only the inventory IPs changing between them.

The Shared Root: Why Not Three Independent CAs?

If separation is the goal, why not use three completely independent CA hierarchies? Why do all three leaf CAs chain back to the same root?

There are practical reasons. A shared root means you can verify any certificate in the cluster chains back to a single trust anchor, which simplifies auditing and troubleshooting. The intermediate CA acts as a rotation point — if you need to replace a leaf CA (say the etcd CA is compromised), you re-sign a new etcd CA from the intermediate without touching the root or the other leaf CAs. Clients that trust the root continue to trust the chain after the rotation.

The key insight is that the shared root doesn’t weaken the separation. The trust boundaries are enforced at the configuration level, not the chain level. etcd is configured to trust etcd-ca.crt, not root-ca.crt. Even though the etcd CA chains to the same root as the Kubernetes CA, etcd never sees the root — it only trusts its own leaf CA. The chain verification stops at the CA bundle specified in --trusted-ca-file.

This is the same pattern used by public CAs. Let’s Encrypt and DigiCert both chain to roots in your browser’s trust store, but a Let’s Encrypt certificate can’t impersonate a DigiCert-issued site. The roots are shared (your browser trusts both), but the leaf issuers are distinct. In the homelab clusters, the components are configured to trust specific leaf CAs, not the root — so the separation holds.

Verifying CA Separation on Your Homelab

These commands prove the CAs are separate on any HA cluster (UTM, Vagrant, OrbStack). SSH to jump first.

Compare the CA certificates — they should have different subjects and serial numbers:

# View the Kubernetes CA
ssh master-1 'openssl x509 -in /etc/kubernetes/pki/ca.crt -subject -serial -noout'
# View the etcd CA
ssh etcd-1 'openssl x509 -in /etc/etcd/pki/etcd-ca.crt -subject -serial -noout'
# View the Front Proxy CA
ssh master-1 'openssl x509 -in /etc/kubernetes/pki/front-proxy-ca.crt -subject -serial -noout'
# All three should show different subjects and serial numbers
# This proves they're distinct CAs, not copies of the same one

Prove cross-CA verification fails:

# Try to verify an etcd server cert with the Kubernetes CA — should fail
ssh master-1 'openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/etcd/pki/etcd-server.crt'
# Error: unable to get local issuer certificate
# Try to verify a Kubernetes cert with the etcd CA — should also fail
ssh master-1 'openssl verify -CAfile /etc/etcd/pki/etcd-ca.crt /etc/kubernetes/pki/kube-apiserver.crt'
# Error: unable to get local issuer certificate
# Verify each cert against its own CA — should succeed
ssh etcd-1 'openssl verify -CAfile /etc/etcd/pki/etcd-ca.crt /etc/etcd/pki/etcd-server.crt'
# OK
ssh master-1 'openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/kube-apiserver.crt'
# OK

Check which CAs are configured in the API server:

# View the kube-apiserver systemd unit and grep for CA flags
ssh master-1 'cat /etc/systemd/system/kube-apiserver.service | grep -i ca'
# You should see three separate CA file references:
# --client-ca-file        → Kubernetes CA
# --etcd-cafile           → etcd CA
# --requestheader-client-ca-file → Front Proxy CA

Verify in Vault — three separate PKI engines:

# From the jump server (Vault CLI is configured)
vault secrets list | grep pki
# You should see:
# pki_root/          pki
# pki_int/           pki
# pki_kubernetes/    pki
# pki_etcd/          pki
# pki_front_proxy/   pki
# List roles on each leaf CA
vault list pki_kubernetes/roles
vault list pki_etcd/roles
vault list pki_front_proxy/roles