High Availability Kubernetes Cluster
Context and motivation
At work I run production on my own, and the rule I hold myself to is simple: anything I operate has to survive me breaking it. This lab applies that rule to Kubernetes. Three server nodes with etcd, real quorum, because with a single master you can't kill the master and tell the story.
Everything starts from a `terraform apply` and a repository. ArgoCD compares the cluster against what Git says, and what isn't in Git doesn't exist, so a rollback is a revert. On top run an autoscaled application and a PostgreSQL with a replica and automatic failover, with real Let's Encrypt certificates solved over DNS and nothing exposed to the internet.
The four drills recreate the incidents every platform meets sooner or later: a node dies in the middle of the night, a broken deploy ships on a Friday, traffic spikes, everything is lost. Each one is a script with a stopwatch, and every number below came from running it, not estimating it. The project also handed me an incident of its own: on nested virtual machines, etcd took up to a second per write and the cluster ate itself in leader elections, so I measured, ruled out disk and memory, and moved the nodes to LXC containers. Hypervisor load fell from 8 to 3.5 and the cluster settled down.
Technologies
Main features
- Three server nodes with embedded etcd, created by Terraform as LXC containers on Proxmox
- GitOps with ArgoCD as an app of apps: the cluster state is whatever the repository says
- Drill 1, the node holding the PostgreSQL primary dies: 1 request lost out of 4,315, the replica promotes itself
- Drill 2, a broken deploy on a Friday: 0.0 s of downtime across 3,437 probes, and the rollback is a `git revert`
- Drill 3, a spike of 3,081 requests per second: from 3 to 6 replicas in 40 s with p95 at 22 ms
- Drill 4, total loss: `terraform destroy` of all three nodes and a full rebuild, data included, in 9 min 37 s
- Let's Encrypt certificates over a DNS challenge, and monitoring with Prometheus and Grafana, custom lab dashboard included
Gallery
Conclusion
It's a lab and I say so in the repository. It runs on nested virtualization on a home machine, and MetalLB announces the IP in L2 mode from a single node, so the outcome of drill 1 depends on which node you kill. The only secret, the Cloudflare DNS token, is created by hand and documented.
I ran every drill twice, the first pass discovers and the second one measures. The first pass of the total loss found two bugs that weren't in the cluster: SSH refused to talk to the freshly rebuilt nodes because their host keys had changed, and the ArgoCD install died because one of its CRDs is bigger than what a Kubernetes annotation allows. Both are fixed in the script, and the published numbers come from the second pass.