300 lines
14 KiB
Markdown
300 lines
14 KiB
Markdown
# ☸ Kubernetes — architecture, platforms, Cluster API
|
|
|
|
## Overview
|
|
|
|
Kubernetes (K8s) is an open-source container orchestrator — the de facto standard for deploying, scaling, and managing containerized applications. Built on declarative configuration and control loops (reconciliation).
|
|
|
|
## Kubernetes deployment methods
|
|
|
|
| Method | Description | Control plane | Best for |
|
|
|--------|-------------|--------------|----------|
|
|
| **kubeadm** | Official K8s cluster bootstrap tool | Self-managed (stacked/external etcd) | On-prem, lab, learning |
|
|
| **K3s** | Lightweight K8s (Rancher), single binary, embedded etcd/SQLite | Self-managed | Edge, IoT, low-resource, HA with embedded etcd |
|
|
| **RKE2** | Rancher Kubernetes Engine 2, CIS-hardened, FIPS-ready | Self-managed | Enterprise on-prem, air-gapped, regulatory |
|
|
| **OpenShift** | Red Hat enterprise K8s + operator lifecycle + SDN + routing | Self-managed (RHCOS) | Enterprise, multicluster, platform engineering |
|
|
| **Vanilla K8s (CAPI)** | Cluster API — declarative provisioning and lifecycle management | Self-managed (CAPI managed) | Fleet management, GitOps, multi-provider |
|
|
| **EKS** (AWS) | Managed K8s | AWS managed | AWS cloud-native, least ops |
|
|
| **AKS** (Azure) | Managed K8s | Azure managed | Azure cloud-native |
|
|
| **GKE** (GCP) | Managed K8s, auto-pilot, autopilot modes | GCP managed | GCP cloud-native |
|
|
| **SKE** (Sangfor) | Managed K8s on Sangfor HCI | Vendor managed | Sangfor HCI ecosystem |
|
|
|
|
---
|
|
|
|
## Cluster API (CAPI)
|
|
|
|
### What is Cluster API
|
|
|
|
Cluster API is a Kubernetes sub-project (SIG Cluster-Lifecycle) that brings declarative APIs for provisioning, upgrading, and operating Kubernetes clusters. Instead of Terraform scripts or manual `kubeadm`, you define clusters as Kubernetes Custom Resources — `Cluster`, `Machine`, `MachineDeployment`, etc.
|
|
|
|
Core principle: **A Kubernetes cluster that manages Kubernetes clusters.**
|
|
|
|
### Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────┐
|
|
│ Management Cluster │
|
|
│ │
|
|
│ ┌──────────────────────────────────┐ │
|
|
│ │ CAPI Controllers │ │
|
|
│ │ ┌──────┐ ┌──────┐ ┌─────────┐ │ │
|
|
│ │ │ Infra│ │Bootstrap│ │Control │ │ │
|
|
│ │ │ Prov │ │ Prov │ │Plane Pr │ │ │
|
|
│ │ └──────┘ └──────┘ └─────────┘ │ │
|
|
│ └──────────────────────────────────┘ │
|
|
│ │
|
|
│ CR: Cluster, Machine, MachineDeployment│
|
|
│ ... │
|
|
└────────────────┬────────────────────────┘
|
|
│ CAPI controller
|
|
│ creates / manages
|
|
┌────────┴────────┐
|
|
▼ ▼
|
|
┌───────────────┐ ┌───────────────┐
|
|
│ Workload │ │ Workload │
|
|
│ Cluster (dev) │ │ Cluster (prod)│
|
|
│ ┌───┐ ┌───┐ │ │ ┌───┐ ┌───┐ │
|
|
│ │ CP│ │ W │ │ │ │ CP│ │ W │ │
|
|
│ └───┘ └───┘ │ │ └───┘ └───┘ │
|
|
└───────────────┘ └───────────────┘
|
|
```
|
|
|
|
- **Management cluster** — a Kubernetes cluster running CAPI controllers. Can be a dedicated small admin cluster.
|
|
- **Workload (managed) cluster** — Kubernetes clusters managed by CAPI; each is a CRD inside the management cluster.
|
|
- **Machine** — abstraction of a compute unit (VM, bare metal) that becomes a K8s node.
|
|
|
|
### Key CRDs (Custom Resource Definitions)
|
|
|
|
| CRD | API group | Purpose |
|
|
|-----|-----------|---------|
|
|
| **Cluster** | `cluster.x-k8s.io` | Cluster representation (infra ref, control plane ref, networking) |
|
|
| **Machine** | `cluster.x-k8s.io` | Individual node (VM/BM instance) |
|
|
| **MachineDeployment** | `cluster.x-k8s.io` | Declarative scaling and rolling update of workers |
|
|
| **MachineSet** | `cluster.x-k8s.io` | Replica set for Machines (lower-level) |
|
|
| **MachineHealthCheck** | `cluster.x-k8s.io` | Auto-remediation (replace unhealthy nodes) |
|
|
| **ClusterClass** | `cluster.x-k8s.io` | Cluster template for reuse |
|
|
| **KubeadmControlPlane** | `controlplane.cluster.x-k8s.io` | Kubeadm-managed control plane (stacked/external etcd) |
|
|
| **KubeadmConfig / KubeadmConfigTemplate** | `bootstrap.cluster.x-k8s.io` | Bootstrap configuration (kubeadm init/join) |
|
|
|
|
### Provider model
|
|
|
|
CAPI uses a three-layer provider model:
|
|
|
|
#### 1. Infrastructure Provider
|
|
Creates and manages infrastructure (VM, networks, LB, storage).
|
|
|
|
| Provider | Platform | Status |
|
|
|----------|----------|--------|
|
|
| **AWS (CAPA)** | AWS EC2, VPC, ELB, EKS | Stable, SIG-sponsored |
|
|
| **Azure (CAPZ)** | Azure VM, VNet, LB, AKS | Stable, SIG-sponsored |
|
|
| **GCP (CAPG)** | GCP Compute, VPC, GKE | Beta |
|
|
| **vSphere (CAPV)** | VMware vSphere | Stable |
|
|
| **OpenStack (CAPO)** | OpenStack compute/network | Stable |
|
|
| **Metal3** | Bare metal (Ironic) | Stable |
|
|
| **Docker (CAPD)** | Docker containers (development) | Tilt/Dev only |
|
|
| **Akamai (Linode)** | Linode | Community |
|
|
| **Azure Stack HCI** | Azure Stack HCI | Community |
|
|
| **cloudscale** | cloudscale.ch | Community |
|
|
| **Exoscale** | Exoscale | Community |
|
|
| **IBM Cloud** | IBM Cloud | Community |
|
|
| **Equinix Metal** | Equinix (ex Packet) | Community |
|
|
| **Hetzner** | Hetzner Cloud | Community |
|
|
| **OpenNebula** | OpenNebula | Community |
|
|
|
|
#### 2. Bootstrap Provider
|
|
Handles K8s initialization on a node (kubeadm init/join, TLS certs, tokens).
|
|
|
|
| Provider | Description |
|
|
|----------|-------------|
|
|
| **Kubeadm** (built-in) | Standard kubeadm init/join, supports stacked/external etcd |
|
|
| **EKS** | Bootstrap for EKS managed control plane (AWS) |
|
|
| **K3s** | Lightweight K8s bootstrap (edge, IoT) |
|
|
| **RKE2** | Rancher K8s bootstrap, CIS-hardened |
|
|
| **Talos** | API-driven bootstrap (Sidero Labs), immutable OS |
|
|
| **k0smotron** | K0s-based bootstrap + hosted control plane |
|
|
| **MicroK8s** | Canonical MicroK8s bootstrap |
|
|
| **Canonical Kubernetes** | Canonical K8s (snap-based) |
|
|
|
|
#### 3. Control Plane Provider
|
|
Manages control plane nodes.
|
|
|
|
| Provider | Description |
|
|
|----------|-------------|
|
|
| **KubeadmControlPlane** (built-in) | Kubeadm-managed CP, stacked/external etcd |
|
|
| **EKS** | AWS EKS managed control plane |
|
|
| **Kamaji** | Hosted control plane (CP runs as deployment in management cluster) |
|
|
| **K3s** | K3s control plane (edge-optimized) |
|
|
| **RKE2** | RKE2 control plane |
|
|
| **Talos** | Talos control plane, API-based management |
|
|
| **k0smotron** | Hosted control plane (k0s-based) |
|
|
| **Nested** | Nested virtualization control plane |
|
|
|
|
### ClusterClass and Managed Topologies
|
|
|
|
ClusterClass (stable since CAPI v1beta1, CAPI v1.0+) allows defining a **cluster template**:
|
|
|
|
```yaml
|
|
apiVersion: cluster.x-k8s.io/v1beta1
|
|
kind: ClusterClass
|
|
metadata:
|
|
name: standard-aws-cluster
|
|
spec:
|
|
controlPlane:
|
|
ref:
|
|
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
|
|
kind: KubeadmControlPlaneTemplate
|
|
name: aws-cp-tmpl
|
|
machineInfrastructure:
|
|
ref:
|
|
kind: AWSMachineTemplate
|
|
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
|
|
name: aws-cp-machine-tmpl
|
|
workers:
|
|
machineDeployments:
|
|
- class: default-worker
|
|
template:
|
|
bootstrap:
|
|
ref:
|
|
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
|
|
kind: KubeadmConfigTemplate
|
|
name: aws-worker-bootstrap-tmpl
|
|
infrastructure:
|
|
ref:
|
|
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
|
|
kind: AWSMachineTemplate
|
|
name: aws-worker-machine-tmpl
|
|
variables:
|
|
- name: instanceType
|
|
required: true
|
|
schema:
|
|
openAPIV3Schema:
|
|
type: string
|
|
enum: ["t3.large", "m5.large", "m5.xlarge"]
|
|
```
|
|
|
|
Then create a cluster with variable overrides:
|
|
|
|
```yaml
|
|
apiVersion: cluster.x-k8s.io/v1beta1
|
|
kind: Cluster
|
|
metadata:
|
|
name: dev-team-alpha
|
|
namespace: clusters
|
|
spec:
|
|
topology:
|
|
class: standard-aws-cluster
|
|
version: v1.30.2
|
|
controlPlane:
|
|
replicas: 1
|
|
workers:
|
|
machineDeployments:
|
|
- class: default-worker
|
|
name: md-0
|
|
replicas: 2
|
|
variables:
|
|
- name: instanceType
|
|
value: "m5.xlarge"
|
|
```
|
|
|
|
### Cluster lifecycle with CAPI
|
|
|
|
| Phase | Action | CAPI mechanism |
|
|
|-------|--------|----------------|
|
|
| **Create** | `kubectl apply -f cluster.yaml` | Controller creates infra (VM, network), runs kubeadm init/join bootstrap |
|
|
| **Scale** | Update `replicas` in MachineDeployment | Controller creates/removes Machine → VM → node join/drain |
|
|
| **Upgrade** | Change `version` in KubeadmControlPlane / MachineDeployment | Rolling update: new CP node → upgrade → old drain & delete. Workers: MachineDeployment rolling update |
|
|
| **Health check** | MachineHealthCheck | If node unhealthy > timeout, controller creates replacement Machine |
|
|
| **Delete** | `kubectl delete cluster` | Controller drains, deletes VMs, cleans up infrastructure |
|
|
| **Template update** | Change AWSMachineTemplate / KubeadmConfigTemplate | New Machines use the new template; existing Machines only affected via rolling update |
|
|
|
|
### Auto-remediation (MachineHealthCheck)
|
|
|
|
```yaml
|
|
apiVersion: cluster.x-k8s.io/v1beta1
|
|
kind: MachineHealthCheck
|
|
metadata:
|
|
name: prod-mhc
|
|
namespace: clusters
|
|
spec:
|
|
clusterName: prod-us-east
|
|
selector:
|
|
matchLabels:
|
|
cluster.x-k8s.io/deployment-name: prod-us-east-workers
|
|
unhealthyConditions:
|
|
- type: Ready
|
|
status: "False"
|
|
timeout: 5m
|
|
- type: Ready
|
|
status: Unknown
|
|
timeout: 5m
|
|
maxUnhealthy: "40%"
|
|
nodeStartupTimeout: 10m
|
|
```
|
|
|
|
### CAPI + GitOps
|
|
|
|
CAPI integrates naturally with GitOps:
|
|
|
|
- **ArgoCD** — Cluster and MachineDeployment manifests in Git repo, ArgoCD applies them to the management cluster
|
|
- **Flux** — `Kustomization` + `OCIRepository` for CAPI objects
|
|
- **Crossplane** — can be combined: Crossplane provisions cloud resources (VPC, subnets), CAPI manages K8s clusters on top
|
|
|
|
Pattern: a dedicated "fleet management" cluster running CAPI + ArgoCD. All workload clusters are defined as YAML in Git.
|
|
|
|
### CAPI for on-prem
|
|
|
|
| Provider | Use case | Note |
|
|
|----------|----------|------|
|
|
| **Metal3** (Ironic) | Bare metal provisioning (PXE, IPMI, Redfish) | Automatically provisions BM servers as K8s nodes |
|
|
| **CAPV (vSphere)** | VMware VMs as K8s nodes | Most common enterprise on-prem |
|
|
| **CAPO (OpenStack)** | OpenStack VMs as K8s nodes | OpenStack-native |
|
|
| **Nutanix (CAPNX)** | Nutanix AHV/Prism | Community provider |
|
|
|
|
### CAPI for edge
|
|
|
|
| Provider | Use case | Note |
|
|
|----------|----------|------|
|
|
| **K3s bootstrap + control plane** | Lightweight K8s on edge devices | Single binary, SQLite/embedded etcd |
|
|
| **RKE2 bootstrap + control plane** | Enterprise edge, air-gapped | CIS-hardened, FIPS |
|
|
| **Talos** | Immutable OS, API-driven | Minimal footprint, no SSH |
|
|
| **k0smotron** | Hosted control plane for edge clusters | CP runs in management cluster, worker on edge |
|
|
|
|
### CAPI vs alternatives
|
|
|
|
| Tool | Approach | CAPI advantage | CAPI disadvantage |
|
|
|------|----------|----------------|-------------------|
|
|
| **Terraform/Pulumi** | Imperative/declarative IaC | CAPI is K8s-native — same tool for apps and clusters; GitOps ready | Terraform has broader non-K8s resource support |
|
|
| **kubeadm** | Manual or scripted | CAPI automates full lifecycle including upgrades and remediation | Higher complexity, requires management cluster |
|
|
| **Rancher** | Web UI + API for K8s cluster management | CAPI is open-source, vendor-neutral | Rancher has GUI, monitoring, app catalog |
|
|
| **OpenShift Hive/ACM** | Red Hat Advanced Cluster Management | CAPI is standard (SIG) — wider provider ecosystem | ACM has governance, policy, compliance |
|
|
|
|
### Limitations and maturity
|
|
|
|
- **Management cluster is SPOF** — needs its own HA and backup (etcd snapshots, certificates)
|
|
- **CAPI is not a cluster autoscaler** — it handles cluster lifecycle, not pod auto-scaling within a cluster (use Cluster Autoscaler separately)
|
|
- **Provider maturity varies** — AWS/Azure/vSphere stable, GCP/OpenStack beta, some community providers alpha
|
|
- **etcd backup is not built-in** — must be handled externally (Velero, etcd snapshot)
|
|
- **CAPI does not handle applications** — only K8s cluster lifecycle (monitoring, logging, ingress is user-managed)
|
|
- **Learning curve** — requires understanding management cluster, provider model, CRDs
|
|
- **CAPI v1.13+ (2026)** — stable release, v1beta1 API is GA, ClusterClass stable, EKS/AKS/GKE managed control plane support
|
|
|
|
### Recommended production CAPI stack
|
|
|
|
| Component | Recommendation |
|
|
|-----------|---------------|
|
|
| **Management cluster** | K3s (small footprint) or kubeadm (3 nodes HA) |
|
|
| **Infra provider** | CAPA (AWS) / CAPV (vSphere) / CAPO (OpenStack) — based on platform |
|
|
| **Bootstrap/CP provider** | Kubeadm or RKE2 |
|
|
| **GitOps** | ArgoCD or Flux |
|
|
| **Backup** | Velero + restic/Ceph |
|
|
| **Cluster autoscaler** | Cluster Autoscaler (via CAPI integration) |
|
|
| **Network** | Cilium (CAPI-native, support) |
|
|
| **Secrets** | External Secrets Operator / Sealed Secrets |
|
|
| **Monitoring** | Prometheus + Grafana (kube-prometheus-stack) |
|
|
| **Ingress** | ingress-nginx / Kong / Traefik |
|
|
|
|
## Sources
|
|
|
|
Links, books and standards: [sources/infrastructure/sources.en.md](sources/infrastructure/sources.en.md)
|
|
|
|
*Last revision: 2026-06-18*
|