knowledge-base/HYPERVISORS.en.md

# 🖥️ Hypervisors and Virtualization Platforms

## Hypervisor Types

| Type | Description | Examples |
|-----|-------|----------|
| **Type 1** (bare-metal) | Runs directly on hardware | VMware ESXi, Microsoft Hyper-V, KVM, Xen |
| **Type 2** (hosted) | Runs on top of host OS | VirtualBox, VMware Workstation, Parallels |

## Platform Overview

| Platform | Hypervisor | License | Note |
|-----------|-----------|---------|----------|
| **VMware vSphere** | ESXi | Proprietary (Subscription from 2024) | Market leader, wide adoption. After Broadcom acquisition (2023), switched to per-core subscription, perpetual license discontinued |
| **Microsoft Hyper-V** | Hyper-V | Windows Server / standalone | Integration with Azure, SCVMM |
| **Proxmox VE** | KVM + LXC | Open source | Debian-based, web UI, low cost |
| **Red Hat OpenStack / oVirt** | KVM | Open source | Open alternative, complex |
| **Nutanix AHV** | KVM (fork) | Part of Nutanix | Integrated HCI solution |
| **XCP-ng / Xen Server** | Xen | Open source | Successor to Citrix Hypervisor |
| **Oracle VM** | Xen | Proprietary | Oracle ecosystem |

## Key Concepts

- **VM — Virtual Machine** — full virtualization, own kernel
- **Container** — shared host kernel, lighter (Docker, LXC)
- **Paravirtualization** — guest OS knows it runs in a VM (better I/O performance)
- **NUMA** — Non-Uniform Memory Access, CPU/memory allocation optimization (see [SERVER-HW.en.md](SERVER-HW.en.md#numa))
- **Overcommit** — allocating more vCPU/RAM than physically available (ratio management)
- **Live Migration** — moving a running VM between hosts (vSphere vMotion, Hyper-V Live Migration)
- **HA (High Availability)** — VM restart on another host upon failure
- **DRS / Load Balancing** — automatic VM distribution based on load

## VMware vSphere

### VMware licensing (post-Broadcom 2024+)

Since 2024, VMware only sells subscription licenses; perpetual + SnS (Support & Subscription) have been discontinued.

| Product | Metric | Price (indicative) | What it includes |
|---------|---------|-------------------|-------------|
| **vSphere Standard** | Per core (min 16 cores/CPU) | ~$140/core/year | ESXi, vCenter, vMotion, HA, DRS basic |
| **vSphere Enterprise Plus** | Per core | ~$220/core/year | All above + DRS advanced, SIOC, NIOC, Big Data Extensions |
| **vSphere Foundation** | Per core (bundle) | ~$350/core/year | vSphere Enterprise Plus + Aria Operations, Aria Operations for Logs, Aria Automation |
| **VMware Cloud Foundation (VCF)** | Per core (bundle) | ~$700/core/year | vSphere + vSAN + NSX + Aria full suite. Required for vSAN and NSX from 2025 |
| **vSAN** | Per core (only as part of VCF from 2025) | No longer standalone | Storage virtualization, dedup, compression, encryption |
| **NSX** | Per core (only as part of VCF from 2025) | No longer standalone | SDN, micro-segmentation, firewall, load balancing |

**Key changes after Broadcom acquisition**:
- Discontinued perpetual license sales (May 2024)
- Discontinued standalone products: vSAN and NSX can no longer be purchased standalone (only within VCF)
- Desktop and ROBO variants cancelled (migrated to VCF)
- Average cost increase: 2–5× compared to the previous model (depends on size and product mix)
- **Impact**: Many customers are migrating to Proxmox VE, Nutanix AHV, or Hyper-V

**Per-core calculation**:
```text
Server: 2× EPYC 9654 (96C each) = 192 cores
  vSphere Standard: 192 × $140 = $26,880/year
  VCF: 192 × $700 = $134,400/year (incl. vSAN and NSX)
For comparison: previously perpetual + SnS ≈ $15,000 one-time + $3,000/year
```

### VMware Exit Strategy (post-Broadcom 2024+)

#### Context

After Broadcom's acquisition of VMware (completed November 2023), the virtualization market experienced the biggest upheaval in its history. Changes include:

- **Discontinuation of perpetual licenses** (February 2024) — mandatory subscription model
- **Forced bundling** — 8,000+ SKUs reduced to 4 bundles (VCF, VVF, vSphere Standard/Foundation)
- **Minimum 72-core commitment** (from April 2025) — small servers can no longer be licensed economically
- **20% late renewal penalty** — no tolerance
- **Price increase of 150–1,500%** depending on size and product mix
- **Standalone products discontinued** — vSAN and NSX only within VCF
- **Collapse of the partner ecosystem** — from 4,500+ partners to ~300 Premier

According to Foundry/CIO.com survey (2025): **56%** of organizations plan to reduce VMware usage, **71%** are actively looking for on-premise alternatives. Gartner predicts a loss of ~35% of workloads within 3 years.

#### Three Strategies

| Strategy | Description | Suitable for |
|-----------|-------|------------|
| **Stay** | Accept new pricing, renew VCF/VVF subscription | Large organizations with deep integration where migration costs more than new licenses |
| **Reduce** | Reduce VMware footprint, migrate part of workloads to alternatives, optimize the rest | Medium and large enterprises with heterogeneous environments |
| **Exit** | Complete migration to an alternative platform | SMEs, organizations facing 3–6× cost increases, greenfield projects |

#### Target Platforms — Comparison

| Criterion | Proxmox VE | Nutanix AHV | Microsoft Hyper-V | Red Hat OpenShift Virtualization | **Sangfor aSV (HCI)** |
|-----------|-----------|-------------|-------------------|----------------------------------|----------------------|
| **Hypervisor** | KVM + LXC | KVM (fork) | Hyper-V | KVM (KubeVirt) | **KVM (aSV)** |
| **License** | Open source (free), support ~€500/host/year | Per node subscription (30–60% savings vs VCF) | Windows Server license (Standard/Datacenter) | OpenShift subscription (core-based) | **Per node (Enterprise Pro), all-inclusive** |
| **Live Migration** | Live Migration (Proxmox 8+) | AHV Live Migration | Live Migration (SMB/RDMA) | KubeVirt (VMI live migration) | **Yes** |
| **HA** | Proxmox HA (watchdog, fencing) | Built-in HA (Prism) | Hyper-V HA (WS Failover Cluster) | OpenShift HA (self-healing) | **Built-in HA** |
| **Storage** | ZFS, Ceph, LVM | AOS (hybrid/SSD, erasure coding) | S2D, CSV, ReFS | OCS, Ceph, LSO | **aSAN (distributed SDS, locality-aware)** |
| **Backup** | Proxmox Backup Server (free) | Native snapshot + DR | Windows Server Backup / Veeam | OpenShift APIs + OADP | **Built-in backup + CDP** |
| **Price (3 years, 3 hosts)** | $0 + support $1,500 | ~$45,000–60,000 | $0 (Hyper-V Server free) or Windows Server license | ~$90,000+ (OpenShift) | **~$15,000–25,000** |
| **Price (3 years, 10 hosts)** | $0 + support $5,000 | ~$150,000–200,000 | Windows Server Datacenter for unlimited VMs | ~$300,000+ (OpenShift) | **~$50,000–80,000** |
| **Migration difficulty** | Medium (VMDK → QCOW2, VirtIO drivers) | Low (Nutanix Move tool) | Medium (V2V converter, SCVMM) | High (Kubernetes learning curve) | **Low (VMware import tool)** |
| **Linux support** | Excellent (native KVM) | Excellent (KVM-based) | Good (LIS drivers) | Excellent (KVM + OpenShift) | **Excellent (KVM-based)** |
| **Windows support** | Good (VirtIO drivers) | Excellent (ALAS drivers, svpd) | Excellent (native) | Good (KubeVirt + VirtIO) | **Good (VirtIO drivers)** |
| **GPU passthrough** | VFIO (excellent) | GPU passthrough | DDA (Direct Device Assignment) | VFIO + GPU Operator | **vGPU support (standard)** |
| **Integrated security** | — | — | — | — | **Yes (NGFW, IPS, WAF, EDR — aSEC)** |
| **Min. cluster (3 copies)** | 3 (Ceph) | 3 | 2–3 | 3 | **3** |

#### Migration Tools

| Tool | Source Platform | Target Platform | Method |
|---------|-------------------|-------------------|--------|
| **Proxmox VMware Import Wizard** | VMware ESXi | Proxmox VE | Web GUI import via NFS/ESXi API. Limitation: snapshots must be removed, UEFI not supported before Proxmox 8.1 |
| **Nutanix Move** | VMware ESXi, Hyper-V | Nutanix AHV | Virtual appliance, automated migration with minimal downtime, UEFI support, can retain IP/MAC |
| **Veeam Backup & Replication v12.2+** | VMware ESXi | Proxmox VE | Backup/restore via Veeam, hot migration, Proxmox support from v12.2 |
| **StarWind V2V Converter** | VMware ESXi | Proxmox, Hyper-V, XCP-ng | Free GUI tool, VMDK → QCOW2/raw/VHDX, CLI support, hot migrations |
| **virt-v2v** | VMware ESXi, Xen, Hyper-V | KVM (libvirt) | Open source CLI tool, disk + driver conversion (virtio), suitable for bulk migration |
| **Windows Admin Center VM Conversion Extension** | VMware ESXi | Hyper-V | Microsoft WAC extension, free, GUI-based, bulk migration |
| **Platform9 vJailbreak** | VMware ESXi | OpenStack / KVM | In-place migration (no swing gear), open source |
| **Sangfor VMware Import Tool** | VMware ESXi | Sangfor aSV (HCI) | VMware import tool, disk + driver conversion, can retain network config |

#### Cross-Hypervisor Migration Matrix

Comprehensive overview of all source→target pairs with methods, tools, limitations, and complexity.

| Source → Target | Method | Tools | Complexity | Limitations |
|-------------|--------|----------|-----------|---------|
| **VMware → Proxmox** | Disk conversion VMDK→QCOW2, driver reinstall | Proxmox Import Wizard, Veeam, StarWind, virt-v2v | Medium | VirtIO drivers required, UEFI not supported in Import Wizard (< 8.1), snapshots must be removed |
| **VMware → Hyper-V** | Disk conversion VMDK→VHDX, driver reinstall | StarWind, WAC Converter, SCVMM, Microsoft MTC | Medium | Integration Services required, network config differences (VMXNET3 → Hyper-V Synthetic) |
| **VMware → KVM/XCP-ng** | Disk conversion VMDK→raw/QCOW2, driver swap | virt-v2v, StarWind | Medium | VirtIO drivers, UEFI support (OVMF), host passthrough compatibility |
| **VMware → Nutanix AHV** | Automated migration via Move appliance | Nutanix Move, Veeam | Low | AHV is also KVM — minimal issues, retain IP/MAC, UEFI support |
| **VMware → Sangfor aSV** | Import via VMware Import Tool, disk + driver conversion | Sangfor VMware Import Tool | Low | Built-in tool, retain network config, UEFI support |
| **VMware → OpenStack** | In-place or swing | Platform9 vJailbreak, virt-v2v + Glance | High | Network redesign (Neutron), storage (Cinder), image format (Glance) required |
| **Hyper-V → VMware** | Disk conversion VHDX→VMDK, driver reinstall | StarWind, virt-v2v, VMware vCenter Converter (standalone) | Medium | VMware Tools required, network driver change (VMXNET3), UEFI/secure boot issues |
| **Hyper-V → Proxmox** | Disk conversion VHDX→QCOW2, driver swap | StarWind, virt-v2v, qemu-img | Medium–High | VirtIO drivers, integration services → guest agent, secure boot issues |
| **Hyper-V → KVM/XCP-ng** | Disk conversion VHDX→raw/QCOW2 | virt-v2v, qemu-img | Medium | VirtIO drivers, Linux generic drivers usually work |
| **Hyper-V → Nutanix AHV** | Automated migration | Nutanix Move | Low–Medium | Similar to VMware→Nutanix, UEFI support, retain IP |
| **Proxmox → VMware** | Export OVF/OVA, qemu-img convert | qemu-img (QCOW2→VMDK), ovftool, manual OVF export | High | VMware Tools required, storage format differences, no live migration, downtime required |
| **Proxmox → Hyper-V** | qemu-img convert, driver reinstall | qemu-img, manual VHDX conversion | High | Hyper-V Integration Services required, no automated tool, edge case |
| **Proxmox → KVM/XCP-ng** | Direct QCOW2 (same format), XML edit | libvirt, virsh dumpxml/define | Medium | libvirt XML/QEMU args differences (storage pool, network), validation required |
| **Proxmox → Nutanix AHV** | qemu-img + manual import | qemu-img, Nutanix Image Service CLI | High | No hot tool, conversion + manual VM reconfiguration required |
| **XCP-ng → VMware** | Disk conversion VHD→VMDK | qemu-img, StarWind, virt-v2v | High | VMware Tools required, paravirtualization differences (Xen PV vs VMware) |
| **XCP-ng → Proxmox** | Disk conversion or direct VHD | qemu-img, manual import | Medium | Disk conversion, VHD format not native in Proxmox |
| **XCP-ng → Hyper-V** | Disk conversion VHD→VHDX (direct) | StarWind, qemu-img | Medium | VHD/VHDX compatible, Integration Services required |
| **Nutanix AHV → VMware** | Export + conversion | qemu-img, Nutanix Export, VMware vCenter Converter | High | VMware Tools, AHV is KVM → usually easier than Hyper-V→VMware |
| **Nutanix AHV → Proxmox** | qemu-img + manual import | qemu-img, Nutanix self-service restore | Medium | AFS disks → QCOW2, metadata must be reconstructed |
| **Nutanix AHV → Hyper-V** | qemu-img + manual | qemu-img, StarWind | High | Edge case, no hot tool |
| **OpenStack → (any)** | Glance export + qemu-img | glance image-download, qemu-img, ovftool | Medium–High | Image format (raw/QCOW2), metadata (flavor, security groups) must be recreated |
| **Sangfor aSV → (any)** | qemu-img conversion + manual | qemu-img, manual OVF/OVA export | Medium–High | KVM-based → conversion to QCOW2/VMDK/VHDX via qemu-img, metadata must be recreated |
| **(any) → Sangfor aSV** | aSV API import + VMware Import Tool | Sangfor VMware Import Tool (for VMware), manual qemu-img import for others | Medium | KVM-based → standard formats supported, import tool for VMware only |

**Migration success keys:**

- **Drivers** — each platform requires its own paravirtual drivers (VMware Tools, VirtIO, Hyper-V Integration Services, Xen Tools). Always swap after migration.
- **UEFI / Secure Boot** — not all combinations support UEFI (Proxmox Import Wizard < 8.1 does not). Test UEFI VMs before migration.
- **Snapshots** — snapshots must be removed (merged) before migration. Most tools only migrate flat disks.
- **Network** — MAC addresses, IP addresses, VLAN tagging — verify after migration. Some tools (Nutanix Move, VMware Converter) can retain MAC.
- **Storage format** — VMDK ↔ VHDX ↔ QCOW2 ↔ raw are inter-convertible via `qemu-img`, but metadata differs (snapshots, backing files).
- **Live migration** — no live migration exists between different hypervisors. Downtime is always required (minutes to hours depending on VM size).
- **Migration temperature** — the "colder" the VM (fewer changes), the easier the migration. Real-time database applications require a separate DB migration plan.

| Platform | Year 1 | 3 Years Total | Note |
|-----------|--------|---------------|----------|
| **VMware VVF** (1-year rate) | $22,800 | $68,400 | 120 cores × $190/core/year |
| **VMware VCF** | $42,000 | $126,000 | 120 cores × $350/core/year |
| **Proxmox VE** (support) | $1,500 | $4,500 | 3× €500/host/year |
| **Nutanix AHV** (average) | ~$18,000 | ~$54,000 | Per node subscription, estimate |
| **Hyper-V** (Windows Server Datacenter) | $12,400 | $37,200 | One-time license per core, without SA |
| **Hyper-V** (Azure Stack HCI) | ~$7,200 | ~$21,600 | ~$10/core/month, 120 cores |
| **Sangfor HCI** (Enterprise Pro) | ~$5,000–8,000 | ~$15,000–25,000 | Per node, all-inclusive, 3 nodes |

**Real-world example from Spiceworks (2026)**: A user reports VMware Essentials+ increasing from $1,900/year to $14,000/year (VVF) — a 7.4× increase.

#### Decision Framework

```
1. Audit VMware environment
   ├─ Number of hosts, core count, utilization
   ├─ Feature dependency (vSAN, NSX, SRM)
   ├─ Workload profile (Windows vs Linux, DB, GPU)
   └─ Hardware refresh cycle

2. Calculate TCO for VMware renewal (3 years)
   ├─ VVF vs VCF vs current model
   └─ Include audit risk, late renewal penalty

3. Select target platform (1-2 candidates)
   ├─ Proxmox: lowest TCO, Linux-heavy shops
   ├─ Nutanix: enterprise HCI, low migration difficulty
    ├─ Hyper-V: Windows-centric, Azure hybrid
    ├─ Sangfor: HCI all-in-one, security-first, VMware exit (SMB/mid-market)
    └─ OpenShift: Kubernetes-first, platform engineering

4. Plan migration phases
   ├─ Wave 1: non-critical (dev/test, 1-2 months)
   ├─ Wave 2: standard production (3-6 months)
   ├─ Wave 3: mission-critical (6-12 months)
   └─ Coexistence: VMware + target running in parallel

5. Allow 18-48 months for complete exit (Gartner)
```

#### Real-World Case Studies

| Organization | Starting Point | Target | Scale | Result |
|-----------|---------|-----|--------|----------|
| **Stanford University** | VMware (60+ nodes) | Proxmox VE (6 clusters) | 1,500 VMs | Completed 2025, increased automation, lower costs |
| **Michelin** | VMware | Platform9 + OpenStack | Dozens of nodes | Platform engineering team, production workload migration |
| **Czech enterprise (50-100 servers)** | VMware | Proxmox VE | ~100 VMs | Annual savings of ~340,000–500,000 CZK on licenses |

#### Timing — Key Deadlines

| Event | Date | Impact |
|---------|-------|-------|
| **Discontinuation of perpetual licenses** | February 2024 | Already done |
| **72-core minimum** | April 2025 | Small server licensing became more expensive |
| **vSphere 7 EOS** | April 2025 | Upgrade to 8.x required |
| **ESXi 8.0 EOS** | October 2027 | Last supported version, migration deadline |
| **Windows Server 2025 Hyper-V** | December 2025 | 64-host cluster, 2,048 vCPU per VM |
| **Proxmox VE 9 + Datacenter Manager** | 2026 | Enterprise features, vCenter alternative |

#### Recommendations

| Scenario | Action |
|--------|------|
| **Small company (< 10 hosts), Linux workloads** | Migrate to Proxmox VE — immediate 100% license savings |
| **Medium company (10-50 hosts), mixed workloads** | Evaluate Nutanix AHV (easy migration) or Proxmox (lower TCO) |
| **Enterprise (50+ hosts), deep VMware integration** | Reduce strategy: optimize existing VMware + migrate selected workloads to OpenShift / Hyper-V |
| **Microsoft shop** | Hyper-V / Azure Stack HCI — native Azure hybrid, no additional hypervisor licenses |
| **Kubernetes-native team** | OpenShift Virtualization / KubeVirt — unify VM and container management |
| **MSP / hosting provider** | Nutanix or OpenStack — multi-tenancy, vCloud Director alternative |

#### Cluster Design

- **Max cluster size**: 64 hosts (vSphere 8/9), 96 hosts (vSphere 8 + enhanced)
- **Datastore limits**: max 256 datastores per host, max 65 TB per VMFS-6 datastore
- **vSAN ready capacity**: recommended max 60–64 hosts per vSAN cluster
- **Fault domains** — cluster division into host groups (rack awareness), min 3 fault domains for stretched cluster
- **Admission control** — resource reservation for HA failover:
  - **Host failures cluster tolerates** — most common (1–4 hosts)
  - **Percentage of cluster resources** — reserve % of CPU/memory
  - **Dedicated failover hosts** — dedicated host(s) for HA
- **Cluster limits (vSphere 8/9)**:
  - 960 VMs per host (vSphere 9 max)
  - 15,000 VMs per cluster (vCenter max)
  - 300 hosts per cluster (vSphere 8/9, hardware vMotion)

### Microsoft Hyper-V Licensing

| Variant | Metric | Price | What it includes |
|----------|---------|------|-------------|
| **Windows Server Standard** | Per core (min 16 licenses/server) + CAL | ~$1,000/core (one-time) + $200/CAL | 2 VM licenses (each with full Windows Server license) |
| **Windows Server Datacenter** | Per core (min 16 licenses/server) + CAL | ~$6,200/core (one-time) + $200/CAL | Unlimited VMs, Storage Spaces Direct, Shielded VMs |
| **Azure Stack HCI** | Per core (monthly) | ~$10–20/core/month (Azure hybrid benefit) | Hyper-V + S2D + Azure management, part of Azure subscription |
| **Hyper-V Server** | Free | $0 | Standalone hypervisor (no management, no GUI, limited support) — no longer distributed as of 2025 |

**Important**:
- Windows Server Standard = 2 VMs per license. If you need 3 VMs on a 2-socket server, you need 2× Standard license (4 VMs) or Datacenter
- **Azure Hybrid Benefit** — if you have Windows Server with SA (Software Assurance), you can use licenses in Azure at no additional cost
- **CAL (Client Access License)** — every user or device accessing Windows Server must have a CAL (except Azure Hybrid Benefit)

## Microsoft Hyper-V

| Feature | Hyper-V | Note |
|-----------|---------|----------|
| **Max hosts per cluster** | 64 (Windows Server 2025) | Shared Nothing Live Migration |
| **Max VMs per host** | 1,024 (WS 2022+) | Generation 2 VMs |
| **Max vCPU per VM** | 240 (WS 2022+) | 64-host cluster |
| **Max RAM per VM** | 12 TB (WS 2022+) | Dynamic memory |
| **Live Migration** | SMB, CSV, RDMA | Compressed or RDMA |
| **Storage** | CSV (Cluster Shared Volumes), ReFS | S2D for HCI |
| **Nested Virtualization** | Yes | Intel VT-x / AMD-V |
| **SCVMM** | System Center VMM | Enterprise management, fabric, P2V |

### Hyper-V vs VMware Comparison

| Feature | VMware vSphere | Microsoft Hyper-V |
|-----------|---------------|-------------------|
| **OS** | VMware ESXi (VMkernel) | Windows Server / Hyper-V Server |
| **License** | Per CPU (subscription) | Windows Server license / Datacenter |
| **Storage** | VMFS, NFS, vSAN, HCI | NTFS, ReFS, SMB, S2D |
| **Live Migration** | vMotion (cross-vSwitch, long distance) | Live Migration (SMB/RDMA) |
| **Storage Migration** | Storage vMotion (online) | Shared Nothing (data disk) |
| **Replication** | vSphere Replication | Hyper-V Replica (ASR) |
| **Management** | vCenter, vSphere Client | SCVMM, Hyper-V Manager, Admin Center |
| **Linux support** | Excellent (open-vm-tools) | Good (Linux Integration Services) |
| **TCO** | Higher | Lower (with Windows license) |

## KVM

### Architecture

```
Hardware ──> QEMU (I/O emulation) + KVM (kernel module, virtualization)
                │
           libvirt (API + management)
                │
        ┌───────┼───────────┐
  virt-manager  virsh    openstack/proxmox
```

### Tuning

- **CPU pinning** — `virsh vcpupin vm1 0 2` (vCPU 0 → physical core 2), prevents context switching
- **Huge pages** — 2 MB / 1 GB pages instead of 4 KB, reduces TLB misses (VMs with large RAM): `echo 2048 > /proc/sys/vm/nr_hugepages`
- **NUMA affinity** — VM pinned to one NUMA node (minimizes cross-NUMA memory access)
  - `numactl --cpunodebind=0 --membind=0`
  - `virsh numatune vm1 --nodeset 0`
- **VirtIO** — paravirtualized I/O (virtio-net, virtio-blk, virtio-scsi) for better performance
- **IO threads** — dedicated threads for QEMU I/O emulation

### KVM Tuning Checklist

- Verify HW virtualization: `lscpu | grep Virtualization`
- Load KVM modules: `kvm`, `kvm_intel`/`kvm_amd`, `vfio-pci`
- Optimize storage: raw/LVM (avoid qcow2 for performance workloads)

## Sangfor aSV (HCI)

[Chinese vendor](https://www.sangfor.com) — KVM-based hypervisor, part of Sangfor HCI stack (aSV + aSAN + aNet + aSEC). Distributed through partners in EMEA.

### Stack architecture

| Component | Role |
|-----------|------|
| **aSV** | Hypervisor (KVM-based) |
| **aSAN** | Distributed SDS (locality-aware, data tiering, dedup, compression) |
| **aNet** | Network virtualization (distributed switches and routers, WYDIWYG visual editor) |
| **aSEC** | Security (NGFW, IPS, WAF, EDR, east-west segmentation) |
| **Sangfor Cloud Platform** | Management orchestrator, unified dashboard |

### Key features

| Feature | Detail |
|-----------|--------|
| **Hypervisor** | KVM (aSV) — custom fork with HCI extensions |
| **License** | Enterprise Pro — per node, all-inclusive (compute + storage + network + security) |
| **Min. cluster** | 3 nodes (3 data copies) |
| **Live Migration** | Yes |
| **HA** | Built-in HA |
| **Storage** | aSAN — locality-aware, data tiering (SSD + HDD), dedup, compression, erasure coding |
| **Backup** | Built-in backup + CDP — no 3rd party needed |
| **Security** | Integrated NGFW, IPS, WAF, EDR — no external appliances |
| **VDI** | aDesk — integrated VDI solution |
| **Kubernetes** | SKE (Sangfor Kubernetes Engine) |
| **Migration** | Sangfor VMware Import Tool (from vCenter), qemu-img for others |
| **vGPU** | Standard support (no extra license) |

### Comparison with VMware

| Feature | Sangfor | VMware |
|---------|---------|--------|
| **License** | Per node, all-inclusive | Multi-tier (vSphere + vSAN + NSX + Aria) |
| **vGPU** | Included (standard) | Enterprise Plus only |
| **Backup + CDP** | Built-in | 3rd party or extra license |
| **Security (NGFW, IPS, WAF)** | Built-in (aSEC) | NSX + 3rd party |
| **Network management** | WYDIWYG visual editor | NSX Manager (more complex) |
| **Min. cluster (3 copies)** | 3 nodes | 5 nodes (vSAN) |
| **Data locality** | Yes | No |
| **SSD life prediction** | Yes | No |

### Use case

- **VMware exit** — VMware replacement for SMB and mid-market
- **Greenfield HCI** — new DCs, branch offices, remote sites
- **VDI** — aDesk integrated with HCI
- **Security-first** — organizations requiring integrated security
- **Asia-Pacific / EMEA** — strongest in Asia, expanding to Europe

### Risks and limitations

| Risk | Detail |
|--------|--------|
| **Geopolitical** | Chinese vendor — possible regulatory restrictions (GDPR, EU, NATO, government) |
| **Ecosystem** | Smaller community than VMware/Proxmox, less documentation and ISV certifications |
| **Support** | Primary support from Asia, local partner critical |
| **Vendor lock-in** | Closed ecosystem (aSV + aSAN + aNet + aSEC), harder to mix with 3rd party |
| **References in CZ/EU** | Very limited — pilot required before production |

## Storage in Hypervisors

See also: [STORAGE.en.md](STORAGE.en.md) — detailed overview of storage protocols and configurations.

| Type | Description | Protocols |
|-----|-------|-----------|
| **Local storage** | Disks directly in the server | SATA, SAS, NVMe |
| **Shared storage** | SAN / NAS accessible to all hosts | Fibre Channel, iSCSI, NFS, SMB |
| **vSAN / HCI** | Hyperconverged storage (server disks = single pool) | VMware vSAN, Nutanix, StarWind |
| **Software-Defined** | SDS separates storage software from hardware | Ceph, GlusterFS, MinIO |

## HCI Details

| Feature | Nutanix (AOS + AHV) | VMware vSAN | Azure Stack HCI |
|-----------|--------------------|-------------|----------------|
| **Hypervisor** | AHV (KVM fork), ESXi optional | ESXi (required) | Hyper-V |
| **Min. nodes** | 3 | 2 (witness) | 2 (witness) |
| **Max nodes** | 80+ | 64 | 16 (typical) |
| **Replication** | 2 or 3 copies + erasure coding | Mirroring (RAID 1), erasure coding | Mirroring + parity |
| **Deduplication** | Cluster-level (post-process) | Disk-level (capacity tier) | ReFS (real-time) |
| **Compression** | Inline (AOS 6+) | Dedup + compression combined | ReFS |
| **Management** | Prism (web UI) | vCenter + vSAN UI | Windows Admin Center |
| **Licensing** | Per node subscription | Per CPU subscription | Per core subscription |
| **Ecosystem** | Built-in DR, backup, security | Broad ISV ecosystem | Azure integration |
| **Use case** | Enterprise VDI, general VM | VMware-centric shops | Azure hybrid, branch offices |

## Virtualization Platforms — Comparison

| Capability | VMware vSphere | Microsoft Hyper-V | Proxmox VE | Nutanix AHV |
|-----------|---------------|-------------------|------------|-------------|
| Live Migration | vMotion | Live Migration | Live Migration | Live Migration |
| HA | vSphere HA | Hyper-V HA | Proxmox HA | Built-in |
| DRS/balancing | DRS | SCVMM / AKS | HA groups | Built-in |
| Storage vMotion | yes | when VM is off | ZFS send/recv | Built-in |
| Snapshots | yes | yes | yes | yes |
| Backup API | CBT (Changed Block Tracking) | Hyper-V WMI / RCT | Proxmox Backup Server | Native |
| GPU passthrough | vGPU (NVIDIA Grid) | DDA | VFIO passthrough | GPU passthrough |
| Licensing | Per CPU / subscription | Windows Server license | Open source (free) | Per node subscription |

## OpenStack

- **Distributions**: Red Hat OpenStack, Canonical Charmed OpenStack
- **Services**: Nova (compute), Cinder (block), Neutron (networking), Glance (images), Swift (object)
- **Use case**: Telco, large private clouds, MNO (MANO, NFVI)
- **Complexity**: High — complex deployment and maintenance

---

## Variant Hypervisor Configurations by Size and Storage Type

### Platform Selection by Use Case

| Use Case | Primary Choice | Alternative | Rationale |
|----------|---------------|-------------|------------|
| **VMware shop, enterprise** | vSphere 8/9 | Hyper-V | Most comprehensive ecosystem, vSAN, SRM, broadest ISV support |
| **Microsoft shop, Azure hybrid** | Hyper-V / Azure Stack HCI | vSphere | Windows Server CAL already in place, S2D, Azure Arc, native Hyper-V Replica |
| **SME / low budget** | Proxmox VE | XCP-ng / Hyper-V (free) | Open source, built-in Ceph, ZFS, PBS, no license costs |
| **HCI greenfield** | Nutanix AHV | VMware vSAN | All-in-one, simple management, built-in DR and backup |
| **Hyperscale / telco** | OpenStack (RHOSP) | — | Multi-tenancy, NFVI, MANO, Neutron SDN, Ceph integration |

### Variant A: Small Deployment (2-3 hosts, local storage)

For small companies, branch offices, edge, dev/test. No shared storage — HA provided at the application level or via VM replication.

| Parameter | Proxmox VE | VMware vSphere | Hyper-V |
|----------|-----------|---------------|---------|
| **CPU** | 1× EPYC 9124-9224 / Xeon 4410Y (8-16C) | 1× EPYC 9124-9224 / Xeon 4410Y | 1× Xeon 4410Y / EPYC 9124 |
| **RAM** | 64-128 GB (DDR5-4800, 1DPC) | 64-128 GB | 64-128 GB |
| **OS disk** | 2× SATA SSD RAID1 (240-480 GB) | 2× SATA SSD RAID1 | 2× SATA SSD RAID1 |
| **VM storage** | ZFS RAID10 (4-6× NVMe/SATA SSD) | VMFS local (4-6× SSD RAID5/10) | ReFS CSV (4-6× SSD RAID10) |
| **Network** | 2× 10/25 GbE LACP | 2× 10/25 GbE LACP + management | 2× 10/25 GbE LACP |
| **Management** | Proxmox web UI (1× node) | vCSA / vCenter (1× appliance) | Windows Admin Center / SCVMM |
| **HA** | Proxmox HA (watchdog, fencing) | vSphere HA (1 host failure) | Hyper-V HA (WS Failover Cluster) |
| **Backup** | Proxmox Backup Server | Veeam B&R (Community) | Windows Server Backup / Veeam |
| **License** | Free (support ~€500/host/year) | vSphere Essentials (~$600/3 hosts) | Windows Server Standard (2 VMs) |

**Use case**: Startup, branch office, dev/test, < 200 VMs, no SAN, minimal budget.

**Advantages**: Low cost, simple management. **Disadvantages**: Limited scalability, host failure = VM unavailability.

### Variant B: Medium HCI (3-6 hosts, vSAN / Ceph)

Hyperconverged infrastructure — storage runs on the same hosts as VMs.

| Parameter | VMware vSAN | Proxmox + Ceph | Nutanix AHV |
|----------|------------|----------------|-------------|
| **CPU** | 1-2× EPYC 9334-9654 (16-32C) | 1-2× EPYC 9224-9334 (12-24C) | 1-2× EPYC 9334-9654 |
| **RAM** | 256-512 GB | 128-256 GB | 256-512 GB |
| **Cache tier** | 1-2× NVMe cache (write buffer) | — (Ceph uses RAM/OSD) | 1-2× NVMe (oplog) |
| **Capacity tier** | 4-8× SSD (SAS/SATA) | 4-8× HBA NVMe/SSD (OSD) | 4-6× SSD (extent store) |
| **Network** | 4× 25 GbE (vSAN + VM + mgmt) | 4× 25 GbE (Ceph public + cluster) | 4× 25 GbE (storage + VM) |
| **Fault domain** | Rack awareness (3 racks min) | CRUSH rack level | Rack awareness |
| **Replication** | RAID-1 mirroring (FTT=1) | 3× replication / EC 8+3 | 2× copies + EC |
| **Dedupe/Compress** | Dedup + compression (capacity) | ZFS / Ceph compression (inline) | Inline compression |
| **HA limit** | 1-3 host failures | 1-2 host failures (replication) | 1-2 host failures |
| **Min. hosts** | 2 + witness | 3 (MON + OSD) | 3 |

**Use case**: Medium company, VDI, general virtualization, 50-500 VMs.

**Recommendation**: For vSAN → min. 4 hosts for FTT=1 with erasure coding. For Ceph → min. 3 hosts, ideally 5+, each OSD host = 1 OSD per NVMe for maximum IOPS.

### Variant C: Enterprise FC SAN (6+ hosts)

Classic 3-tier architecture — compute (hosts) + storage (SAN) + network separated.

| Parameter | VMware vSphere | Hyper-V |
|----------|---------------|---------|
| **CPU** | 2× EPYC 9654-9965 (32-64C) | 2× EPYC 9654-9965 / Xeon 8592+ |
| **RAM** | 512-2048 GB (DDR5) | 512-2048 GB |
| **OS disk** | 2× SATA SSD RAID1 (480 GB) | 2× SATA SSD RAID1 |
| **Storage** | FC SAN LUN (2× FC HBA 32/64G) | FC SAN LUN or CSV over SMB |
| **App network** | 2-4× 25/100 GbE LACP | 2-4× 25/100 GbE LACP |
| **Storage network** | 2× FC 32/64G (multipath) | 2× FC 32/64G or SMB Multichannel |
| **vMotion / Live Migration** | 2× 25 GbE dedicated (vMotion) | 2× 25 GbE dedicated (SMB/RDMA) |
| **Management** | vCenter (VCSA), NSX, Aria | SCVMM, Azure Arc |
| **Cluster max** | 64-96 hosts (vSphere 8/9) | 64 hosts (WS 2025) |
| **Admission control** | 1-4 host failures | Nodes reserve |
| **DRS / Balancing** | DRS (fully automated) | SCVMM / AKS load balancing |

**Use case**: Enterprise, databases, critical applications, 500-5000 VMs.

**Storage variants**: FC SAN (lowest latency), iSCSI (lower CAPEX), NFS (simpler management).

**FC SAN topology**:
```
                 ┌─────────────────────────────────────┐
                 │              FC Fabric              │
                 │   ┌─────────┐       ┌─────────┐    │
                 │   │ Switch 1│       │ Switch 2│    │
                 │   └────┬────┘       └────┬────┘    │
                 └────────┼─────────────────┼──────────┘
                    ┌─────┴─────┐     ┌─────┴─────┐
               ┌───┤ FC HBA 1  ├─┐ ┌─┤ FC HBA 2  ├───┐
               │   └───────────┘ │ │ └───────────┘   │
            ┌──┴──┐          ┌──┴──┴──┐          ┌──┴──┐
            │Host1│          │Host2   │          │Host3│ ...
            └─────┘          └────────┘          └─────┘
```

### Variant D: Hyperscale OpenStack (20+ hosts)

For telco, large private clouds, MANO/NFVI environments.

| Parameter | Red Hat OpenStack | Canonical Charmed OpenStack |
|----------|-------------------|-----------------------------|
| **Compute** | Nova + KVM | Nova + KVM |
| **Storage** | Ceph (Cinder/RBD) + Swift | Ceph + Swift |
| **Network** | Neutron + OVN/OVS + DPDK | Neutron + OVN/OVS |
| **CPU per host** | 2× EPYC 9654-9965 (64-128C) | 2× EPYC 9654-9965 |
| **RAM per host** | 512-1024 GB | 512-1024 GB |
| **Storage per host** | Ceph OSD (4-12× NVMe/SSD) | Ceph OSD |
| **Network per host** | 4-8× 100 GbE (DPDK/VPP) | 4× 100 GbE |
| **Control plane** | 3-9× control node (HA) | 3-7× control node |
| **Orchestration** | TripleO / OpenStack Kolla | Juju + charms |
| **SDN** | OVN, OpenDaylight | OVN |
| **NFVI ready** | Yes (SR-IOV, NUMA, huge pages) | Yes |
| **Min. size** | 9 nodes (3 ctl + 3 compute + 3 ceph) | 7 nodes |

**Use case**: Telco (5G UPF, MNO), hyperscale private cloud, > 5000 VMs.

### Connectivity Summary by Platform

| Platform | App / VM Network | Storage Network | Replication / HA | Management |
|-----------|-------------|-------------|----------------|------------|
| **Proxmox small** | 2× 10/25 GbE LACP | — (local ZFS) | — | 1× 1 GbE |
| **vSAN (3-6)** | 2× 25 GbE LACP | 2× 25 GbE (vSAN) | vSAN traffic | 1× 1 GbE |
| **Proxmox Ceph (3-6)** | 2× 25 GbE | 2× 25 GbE (Ceph public) | 2× 25 GbE (Ceph cluster) | 1× 1 GbE |
| **Nutanix (3-6)** | 2× 25 GbE | Dedicated storage VLAN | Replication traffic | 1× 1 GbE |
| **vSphere FC SAN (6+)** | 2-4× 25/100 GbE LACP | 2× FC 32/64G multipath | 2× 25 GbE (vMotion) | 1× 1 GbE + SAN mgmt |
| **Hyper-V FC SAN (6+)** | 2-4× 25/100 GbE LACP | 2× FC 32/64G or SMB | 2× 25 GbE (Live Migration) | 1× 1 GbE |
| **OpenStack (20+)** | 2-4× 100 GbE | 2× 100 GbE (Ceph) | 2× 100 GbE (OVN) | 1× 1 GbE |

## Resources

Links, books and standards: [sources/infrastructure/sources.en.md](sources/infrastructure/sources.en.md)

### Recommended Reading

| Book | Authors | ISBN | Description |
|-------|--------|------|-------|
| Virtualization Essentials (3rd ed., 2023) | Matthew Portnoy | 978-1119481513 | Practical guide to virtualization: from hypervisor basics (Type 1/Type 2), VM configuration (CPU, memory, storage, networking) to cloud computing and DevOps. "Learning-by-doing" approach with tutorials. Author is a Senior System Engineer at VMware/Splunk. |
| VMware vSphere Design (2nd ed.) | Guthrie, Lowe, Coleman | 978-1119130312 | Comprehensive guide to vSphere infrastructure design: hardware selection, network layout, security, storage and hypervisors. Describes a framework for design, decision analysis and best practices from experienced VMware architects. |

*Last revision: 2026-06-04*