602 lines
30 KiB
Markdown
602 lines
30 KiB
Markdown
# 🧠 Infrastruktura pro AI/ML
|
||
|
||
## Přehled komponent
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph Compute
|
||
GPU["GPU (H100/B200/Instinct)"]
|
||
CPU["CPU (AMD EPYC / Intel Xeon)"]
|
||
ASIC["ASIC (TPU, Trainium, Inferentia)"]
|
||
end
|
||
subgraph Network
|
||
IB["InfiniBand NDR/XDR"]
|
||
ROCE["RoCEv2"]
|
||
NVL["NVLink / NVSwitch"]
|
||
end
|
||
subgraph Storage
|
||
FS["Parallel FS (Lustre, GPFS, Weka)"]
|
||
OBJ["Object Store (S3, MinIO)"]
|
||
NVME["Local NVMe cache"]
|
||
end
|
||
subgraph Orchestration
|
||
S["Slurm"]
|
||
K["Kubernetes + Volcano/Kueue"]
|
||
end
|
||
subgraph Cooling
|
||
DLC["Direct-to-chip liquid"]
|
||
IMM["Immersion"]
|
||
AIR["Air (high-density)"]
|
||
end
|
||
|
||
Compute --> Network --> Storage
|
||
Orchestration --> Compute
|
||
Cooling --> Compute
|
||
```
|
||
|
||
---
|
||
|
||
## GPU compute
|
||
|
||
### NVIDIA
|
||
|
||
| GPU | Architektura | FP8 | FP16/BF16 | FP64 | HBM | NVLink | TDP | Rack |
|
||
|-----|-------------|-----|-----------|------|-----|--------|-----|------|
|
||
| **H100 SXM** | Hopper | 3 958 TFLOPS | 1 979 TFLOPS | 67 TFLOPS | 80 GB HBM3 | 900 GB/s | 700 W | 6–8× v DGX H100 |
|
||
| **H200 SXM** | Hopper (HBM3e) | 3 958 TFLOPS | 1 979 TFLOPS | 67 TFLOPS | 141 GB HBM3e | 900 GB/s | 700 W | 6–8× v DGX H200 |
|
||
| **B200** | Blackwell | ~9 000 TFLOPS | ~4 500 TFLOPS | ~40 TFLOPS | 192 GB HBM3e | 1 800 GB/s | 1 000 W | 6–8× v DGX B200 |
|
||
| **GB200 Grace Hopper** | Blackwell | ~18 000 TFLOPS | ~9 000 TFLOPS | — | 192 GB + 480 GB (Grace) | NVLink-C2C | 1 000 W (GPU) + 500 W (CPU) | DGX GB200 (36× GPU) |
|
||
| **L40S** | Ada Lovelace | 733 TFLOPS | 367 TFLOPS | — | 48 GB GDDR6 | N/A | 350 W | Inference, enterprise |
|
||
| **A100 SXM** | Ampere | 1 248 TFLOPS | 624 TFLOPS | 19,5 TFLOPS | 80 GB HBM2e | 600 GB/s | 400 W | DGX A100 |
|
||
|
||
### AMD
|
||
|
||
| GPU | Architektura | FP8 | FP16/BF16 | FP64 | HBM | Infinity Fabric | TDP |
|
||
|-----|-------------|-----|-----------|------|-----|----------------|-----|
|
||
| **MI300X** | CDNA 3 | 2 615 TFLOPS | 1 307 TFLOPS | 81 TFLOPS | 192 GB HBM3 | 896 GB/s | 750 W |
|
||
| **MI250** | CDNA 2 | — | 383 TFLOPS | 95,7 TFLOPS | 128 GB HBM2e | 400 GB/s | 500 W |
|
||
|
||
### Intel
|
||
|
||
| GPU | Architektura | FP16/BF16 | FP32 | HBM | TDP |
|
||
|-----|-------------|-----------|------|-----|-----|
|
||
| **Gaudi 3** | Custom | 1 835 TFLOPS | — | 144 GB HBM2e | 600 W |
|
||
| **Max 1550** | Xe HPC | 600+ TFLOPS | 200 TFLOPS | 128 GB HBM2e | 600 W |
|
||
|
||
### Cloud ASIC
|
||
|
||
| ASIC | Provider | Use case | Výkon |
|
||
|------|----------|----------|-------|
|
||
| **TPU v5p** | Google | Training | ~4 600 TFLOPS (BF16) per pod |
|
||
| **Trainium 2** | AWS | Training | ~1 000 TFLOPS (BF16) per chip |
|
||
| **Inferentia 2** | AWS | Inference | ~400 TOPS (INT8) per chip |
|
||
| **Maia 100** | Microsoft | Training + inference | Custom, 800 W TDP |
|
||
|
||
---
|
||
|
||
## AI networking
|
||
|
||
### Srovnání technologií
|
||
|
||
| Technologie | Bandwidth per link | Latence | Topologie | Use case |
|
||
|-------------|-------------------|---------|-----------|----------|
|
||
| **InfiniBand NDR200** | 200 Gb/s | < 1 µs | Fat-tree, Dragonfly+ | Training (NVIDIA) |
|
||
| **InfiniBand NDR400** | 400 Gb/s | < 1 µs | Fat-tree, Dragonfly+ | Training (NVIDIA) |
|
||
| **InfiniBand XDR** | 800 Gb/s (planned) | < 1 µs | Dragonfly+ | Next-gen training |
|
||
| **RoCEv2** (CX-7/8) | 200–400 Gb/s | 1–2 µs | Fat-tree, Spine-leaf | Training (AMD, Intel, open) |
|
||
| **NVLink 4.0** | 900 GB/s per GPU | < 0,5 µs | NVSwitch full-mesh | Intra-node GPU comm |
|
||
| **NVLink 5.0** | 1 800 GB/s per GPU | < 0,5 µs | NVSwitch full-mesh | Intra-node (Blackwell) |
|
||
| **Ethernet (400 GbE)** | 400 Gb/s | 2–5 µs | Spine-leaf | Inference, data pipeline |
|
||
|
||
### Principy AI fabric
|
||
|
||
- **Rail-optimized topology** — každá GPU komunikuje na dedikovaném "rails" (stejné GPU indexy napříč uzly jsou na stejném switchi)
|
||
- **Fat-tree (Clos)** — standard pro InfiniBand a RoCE, non-blocking bisection bandwidth
|
||
- **Dragonfly+** — redukce počtu hopů při zachování bandwidth (používáno v největších clusterech)
|
||
- **GPU Direct RDMA** — přímá komunikace GPU ↔ GPU bez CPU involvementu, podpora InfiniBand a RoCE
|
||
- **SHARP (Scalable Hierarchical Aggregation and Reduction Protocol)** — in-network reduction pro AllReduce (pouze InfiniBand)
|
||
|
||
### Bandwidth dimenzování
|
||
|
||
```text
|
||
Pravidlo: InfiniBand bandwidth ≥ 50 % GPU HBM bandwidth pro škálovatelné training
|
||
|
||
Příklad: H100 má 3,35 TB/s HBM
|
||
→ Potřebuje min. 1,6 TB/s bisection bandwidth per GPU
|
||
→ 8× H100 v DGX: 4× NDR400 IB na GPU = 4 × 50 GB/s = 200 GB/s
|
||
→ Reálně: 8× 200 Gb/s (25 GB/s) per GPU v typické konfiguraci = ~6 % HBM → bottleneck
|
||
```
|
||
|
||
---
|
||
|
||
## AI storage
|
||
|
||
### Požadavky
|
||
|
||
| Dataset size | IO pattern | Doporučený storage | Bandwidth |
|
||
|-------------|-----------|-------------------|-----------|
|
||
| < 10 TB | Sequential read (data loading) | Local NVMe | > 10 GB/s per node |
|
||
| 10–100 TB | Random read (checkpointing) | Parallel FS (Lustre, Weka) | > 100 GB/s cluster-wide |
|
||
| 100 TB–10 PB | Mixed (training + checkpoint) | Parallel FS + object store | > 500 GB/s |
|
||
| 10 PB+ | Multi-modal, video, LLM | Tiered (NVMe cache + parallel FS + object) | > 1 TB/s |
|
||
|
||
### Srovnání storage řešení
|
||
|
||
| Řešení | Typ | Bandwidth per node | Max capacity | Škálování | Use case |
|
||
|--------|-----|-------------------|-------------|-----------|----------|
|
||
| **Lustre** | Parallel FS (POSIX) | > 100 GB/s (cluster) | 100s PB | OST + MDS | HPC, LLM training (standard) |
|
||
| **GPFS / StorageScale** | Parallel FS (POSIX) | > 100 GB/s | 100s PB | NSD servers | HPC, AI (IBM) |
|
||
| **WekaFS** | Parallel FS (POSIX + NFS/SMB) | ~80 GB/s per 10 nodes | 10s PB | Container-native | AI/ML, NVIDIA DGX preferred |
|
||
| **VAST Data** | Universal storage (NVMe + QLC) | ~100 GB/s per cluster | 10s PB | Scale-out | AI, checkpoint, data lake |
|
||
| **Pure Storage//E** | All-flash (NVMe) | ~50 GB/s | ~30 PB | Scale-out | Enterprise AI, database |
|
||
| **MinIO / S3** | Object store | ~20 GB/s per gateway | EB | Erasure coding | Dataset repository, checkpoint |
|
||
| **NetApp AFF** | NAS + S3 | ~10 GB/s per controller | ~50 PB | HA pair | Enterprise, NFS baseline |
|
||
|
||
### Checkpointing strategie
|
||
|
||
| Strategie | RPO | Storage impact | Popis |
|
||
|-----------|-----|---------------|-------|
|
||
| **Full checkpoint** | každý N step | Vysoký (zastaví training) | Celý model + optimizer state |
|
||
| **Async checkpoint** | každý N step | Střední (non-blocking) | Kopie do staging bufferu, zápis na pozadí |
|
||
| **Distributed checkpoint** (NVIDIA NeMo) | každý N step | Nízký | Každá rank zapisuje svůj shard |
|
||
| **In-memory checkpoint** (IBM) | při failover | Minimální (DRAM) | Replikace do DRAM jiného node |
|
||
| **Continuous checkpoint** (Microsoft) | každý 1–5 min | Nízký (delta) | Jen changed shardy |
|
||
|
||
---
|
||
|
||
## AI cluster architektura
|
||
|
||
### Fyzická topologie — DGX H100 example
|
||
|
||
```
|
||
┌──────── DGX H100 (8× GPU) ────────┐
|
||
│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │
|
||
│ │GPU 0│ │GPU 1│ │GPU 2│ │GPU 3│ │
|
||
│ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ │
|
||
│ ┌──┴──┐ ┌──┴──┐ ┌──┴──┐ ┌──┴──┐ │
|
||
│ │GPU 4│ │GPU 5│ │GPU 6│ │GPU 7│ │
|
||
│ └─────┘ └─────┘ └─────┘ └─────┘ │
|
||
│ NVSwitch (NVLink 4.0, 900 GB/s) │
|
||
│ InfiniBand CX-7: 8× NDR400 │
|
||
└────────────────────────────────────┘
|
||
│ 8× IB rails
|
||
┌────┴──────────────┐
|
||
│ IB NDR400 Switches │ (rail-optimized)
|
||
└────────────────────┘
|
||
```
|
||
|
||
### Kubernetes pro AI
|
||
|
||
| Komponenta | Role |
|
||
|-----------|------|
|
||
| **Volcano** | Batch scheduling, gang scheduling, queue management |
|
||
| **Kueue** | Multi-tenant admission, resource quotas, fair sharing |
|
||
| **NVIDIA GPU Operator** | Driver, container toolkit, MIG, DCGM, monitoring |
|
||
| **HAMi** (ex k8s-vGPU-scheduler) | GPU sharing, MIG partitioning, fractional GPU |
|
||
| **Node Feature Discovery** | Detekce GPU typu, NUMA topologie |
|
||
| **Topology Manager** | NUMA-aware pod placement |
|
||
| **DPDK / SR-IOV** | High-performance networking pro GPU Direct RDMA |
|
||
|
||
### Slurm pro AI
|
||
|
||
| Komponenta | Role |
|
||
|-----------|------|
|
||
| **slurm.conf** | Partition pro GPU nodes, GRES (Generic Resource) |
|
||
| **gres.conf** | GPU typ, počet GPU na node |
|
||
| **srun --gres=gpu:8** | Alokace 8 GPU pro job |
|
||
| **sbatch --nodes=64 --ntasks=512** | 64 uzly, 512 ranků (8 GPU/node) |
|
||
| **Pixis** | NVIDIA orchestrace plugin pro Slurm |
|
||
|
||
---
|
||
|
||
## Chlazení AI clusterů
|
||
|
||
### Power density srovnání
|
||
|
||
| Konfigurace | TDP per node | Racků | kW/rack | Poznámka |
|
||
|-------------|-------------|-------|---------|----------|
|
||
| Standardní server (2U) | 1 kW | 20 | 5–10 | Běžné DC |
|
||
| GPU server (DGX H100, 6×) | 42 kW | 6 | 45–50 | Air cooling limit |
|
||
| GPU server (DGX B200, 6×) | 72 kW | 6 | 90–100 | Liquid cooling nutný |
|
||
| GPU server (GB200 NVL72) | 120 kW | — | ~120 | Liquid cooling mandatory |
|
||
| NVIDIA NVL72 rack | 120 kW | 1 | 120 | Plně liquid cooled |
|
||
|
||
### Chladící technologie
|
||
|
||
| Metoda | Max kW/rack | CAPEX | OPEX | Komplexita |
|
||
|--------|-------------|-------|------|-----------|
|
||
| **Air cooling (CRAC/CRAH)** | < 15 | Nízká | Střední | Nízká |
|
||
| **Air cooling (in-row)** | 15–30 | Střední | Střední | Nízká |
|
||
| **Rear-door heat exchanger** | 30–50 | Střední | Nízká | Střední |
|
||
| **Direct-to-chip liquid (cold plate)** | 50–150 | Vysoká | Nízká | Vysoká |
|
||
| **Immersion (single-phase)** | 100–200 | Vysoká | Nízká | Vysoká |
|
||
| **Immersion (two-phase)** | 200+ | Velmi vysoká | Nízká | Velmi vysoká |
|
||
|
||
---
|
||
|
||
## Inference infrastruktura
|
||
|
||
### Srovnání inference serverů
|
||
|
||
| Nástroj | Frameworky | Optimalizace | Use case |
|
||
|---------|-----------|-------------|----------|
|
||
| **vLLM** | Megatron, HF, AWQ, GPTQ | PagedAttention, KV cache, continuous batching | LLM inference (open source) |
|
||
| **TensorRT-LLM** | TensorRT | INT4/INT8/FP8, inflight batching, attention optimizations | Produkce (NVIDIA) |
|
||
| **Triton Inference Server** | Vše (TensorRT, vLLM, PyTorch) | Model ensemble, model caching, concurrent execution | Enterprise, multi-model |
|
||
| **SageMaker** | Managed | Auto-scaling, model parallelism | AWS managed |
|
||
| **OpenAI API / TGI** | HF Transformers | Continuous batching, flash attention | Hosting |
|
||
|
||
### Optimalizace pro inference
|
||
|
||
| Technika | Latence zlepšení | Propustnost zlepšení | Memory reduction |
|
||
|----------|-----------------|---------------------|------------------|
|
||
| **FP8/INT8 quantization** | — | 2× | 2× |
|
||
| **INT4 quantization** | — | 4× | 4× |
|
||
| **Flash Attention 2/3** | 2–4× | — | 50 % (KV cache) |
|
||
| **PagedAttention** | — | 2–5× | 95 % (KV cache fragmentation) |
|
||
| **Continuous batching** | — | 10–20× | — |
|
||
| **Speculative decoding** | 2–3× | — | — |
|
||
| **Multi-LoRA / S-LoRA** | — | 8–16× | — |
|
||
|
||
---
|
||
|
||
## Distribuované training techniky
|
||
|
||
| Technika | Popis | Frameworky |
|
||
|----------|-------|------------|
|
||
| **Data Parallelism (DDP/FSDP)** | Každá GPU má kopii modelu, různé batch | PyTorch DDP, FSDP |
|
||
| **Tensor Parallelism (TP)** | Model rozdělen po vrstvách (intra-node) | Megatron-LM, DeepSpeed |
|
||
| **Pipeline Parallelism (PP)** | Vrstvy rozděleny napříč uzly | Megatron-LM, DeepSpeed |
|
||
| **Sequence Parallelism (SP)** | Sekvence rozdělena napříč GPU | Megatron-LM |
|
||
| **Expert Parallelism (EP)** | Různé expertní subsítě na různých GPU | Mixture-of-Experts (MoE) |
|
||
| **3D Parallelism** | TP + PP + DP kombinace | Megatron-LM, NeMo |
|
||
| **ZeRO (1/2/3)** | Optimalizátor/gradient/parametry sharding | DeepSpeed |
|
||
| **NCCL / RCCL** | GPU collective communication library | NVIDIA/AMD |
|
||
|
||
---
|
||
|
||
## Operační systémy pro AI
|
||
|
||
### Srovnání distribucí
|
||
|
||
| OS | GPU driver | CUDA | Container toolkit | IB/RoCE | Lustre klient | Produkční podpora |
|
||
|----|-----------|------|-------------------|---------|--------------|-------------------|
|
||
| **Ubuntu 22.04 LTS** | NVIDIA 525+ | 12.x | nvidia-container-toolkit | MLNX_OFED, rdma-core | Ano (lustre-client) | NVIDIA DGX standard |
|
||
| **Ubuntu 24.04 LTS** | NVIDIA 550+ | 12.5+ | nvidia-container-toolkit | MLNX_OFED, rdma-core | Ano | Nejnovější GPU podpora |
|
||
| **RHEL 9 / Rocky 9** | NVIDIA 525+ | 12.x | nvidia-container-toolkit | MLNX_OFED | Ano (EL repo) | Red Hat, enterprise |
|
||
| **DGX OS** (Ubuntu-based) | NVIDIA custom | 12.x | Pre-installed | Pre-configured | Ano | NVIDIA DGX jediná podporovaná |
|
||
| **SLES 15 SP5** | NVIDIA 525+ | 12.x | nvidia-container-toolkit | MLNX_OFED | Ano | HPC, některé Lustre clustery |
|
||
| **Debian 12** | NVIDIA 525+ | 12.x | nvidia-container-toolkit | rdma-core | Ano (backports) | Community, research |
|
||
| **Flatcar / Bottlerocket** | Container-host | — | nvidia-container-toolkit | Omezeně | Ne | K8s-only, minimal footprint |
|
||
|
||
### Omezení a limity
|
||
|
||
#### GPU drivery a CUDA
|
||
|
||
| Omezení | Detail |
|
||
|----------|--------|
|
||
| **Driver-CUDA kompatibilita** | NVIDIA driver major verze musí odpovídat CUDA toolkit (driver ≥ CUDA req). Např. CUDA 12.5 vyžaduje driver ≥ 550 |
|
||
| **Kernel version** | NVIDIA driver není kompatibilní se všemi kernely. Nový kernel (6.8+) může vyžadovat DKMS build nebo opožděnou podporu |
|
||
| **Secure Boot** | NVIDIA driver vyžaduje podepsaný modul (MOK, shim) nebo vypnutý Secure Boot — častý problém v enterprise |
|
||
| **Open vs Proprietary driver** | NVIDIA `nvidia-open` (od R515) — open source kernel modul. Podpora GPU: datové centrum (H100+) → OK, starší GPU → proprietary nutný |
|
||
| **nvidia-persistenced** | Nutný pro udržení GPU initialization, bez něj GPU po idle timeout usnou (`nvidia-smi -pm 1`) |
|
||
| **GPU reset** | Po crash training jobu může GPU viset. `nvidia-smi --gpu-reset` nebo reboot node, někdy i power cycle |
|
||
| **Multi-instance GPU (MIG)** | Vyžaduje specifický driver, MIG mode na GPU, restart GPU. Nelze měnit za běhu. Podpora jen A100, H100, B200 |
|
||
|
||
#### Network (InfiniBand / RoCE)
|
||
|
||
| Omezení | Detail |
|
||
|----------|--------|
|
||
| **MLNX_OFED vs rdma-core** | MLNX_OFED (NVIDIA) — plná podpora, ale vlastní kernel moduly, nutná compatibility s kernel verzí. `rdma-core` (open) — omezená podpora, ale bez modulů |
|
||
| **Kernel compatibility** | MLNX_OFED podporuje jen specifické kernel verze (major.minor). Upgrade kernelu → nutný rebuild MLNX_OFED |
|
||
| **NCCL** | Verze NCCL musí být kompatibilní s CUDA a IB firmware. `nccl-tests` jako validace |
|
||
| **SHARP** | In-network reduction vyžaduje specifickou MLNX_OFED + IB switch firmware kombinaci |
|
||
| **GPU Direct RDMA** | Vyžaduje `nvidia-peermem` modul + MLNX_OFED. Nefunguje se všemi GPU a IB kartami |
|
||
| **RoCE v PFC/ECN** | RoCE vyžaduje lossless fabric (PFC, ECN, DCQCN). Nastavení switch i host — komplexní tuning |
|
||
|
||
#### Storage
|
||
|
||
| Omezení | Detail |
|
||
|----------|--------|
|
||
| **Lustre klient** | Verze klienta musí odpovídat serveru. Upgrade serveru → upgrade všech klientů. Kompatibilní jen s RHEL/Debian deriváty |
|
||
| **POSIX locking** | NFS a Lustre mají odlišné POSIX locking chování. Distributed training spoléhá na flock → problém při smíšených FS |
|
||
| **Filesystem cache** | Page cache může maskovat IO bottleneck. Training joby často vyžadují `O_DIRECT` nebo `sync` IO |
|
||
| **Local NVMe vs parallel FS** | Dataset staging na lokální NVMe eliminuje síťovou závislost, ale vyžaduje prostor a pre-fetch pipeline |
|
||
|
||
#### Kontejnerový runtime
|
||
|
||
| Omezení | Detail |
|
||
|----------|--------|
|
||
| **Docker + GPU** | `nvidia-container-toolkit` (dříve nvidia-docker2). Nutná instalace runtime a config v `/etc/docker/daemon.json` |
|
||
| **Podman + GPU** | Vyžaduje `nvidia-container-toolkit` + podman hook. Méně testováno než Docker |
|
||
| **containerd + GPU** | Standart pro K8s. Vyžaduje `cdi` (Container Device Interface) nebo `nvidia-container-runtime` |
|
||
| **Enroot + Pyxis** | NVIDIA container stack pro Slurm (Enroot = container runtime bez daemona, Pyxis = Slurm plugin) |
|
||
| **User namespace mapping** | Kontejnerové GPU access vyžaduje device cgroup a rootless může selhat (výjimka pro /dev/dri a /dev/nvidia*) |
|
||
|
||
#### Kernel parametry
|
||
|
||
```text
|
||
# AI workload recommended sysctl
|
||
net.core.rmem_max = 134217728 # dostatečný pro NCCL
|
||
net.core.wmem_max = 134217728
|
||
net.ipv4.tcp_rmem = 4096 87380 134217728
|
||
net.ipv4.tcp_wmem = 4096 65536 134217728
|
||
net.core.netdev_budget = 600 # pro vysokou packet rate
|
||
vm.max_map_count = 1048576 # PyTorch DataLoader workers
|
||
kernel.numa_balancing = 0 # vypnout NUMA balancing (ruší locality)
|
||
kernel.sched_min_granularity_ns = 10000000
|
||
|
||
# Disable security mitigations pro perf (pouze na dedicated AI clusterech)
|
||
mitigations=off
|
||
transparent_hugepages=never # nebo madvise — THP může způsobovat latency spiky
|
||
intel_idle.max_cstate=1 # redukce C-state transition latency
|
||
```
|
||
|
||
#### Firmware a HW
|
||
|
||
| Omezení | Detail |
|
||
|----------|--------|
|
||
| **GPU firmware (VBIOS)** | NVIDIA datacenter GPU (H100, B200) mají VBIOS updates přes NVFlash. Bez update → chybí podpora partitioning nebo novějších CUDA feature |
|
||
| **InfiniBand firmware** | IB switch a HCA firmware musí být kompatibilní. Mix starého switch + nového HCA → degraded perf |
|
||
| **NVSwitch firmware** | DGX systémy mají NVSwitch firmware updatovatelný jen přes NVIDIA DGX tools |
|
||
| **Power capping (nvidia-smi)** | `nvidia-smi -pl <power>` — omezení TDP pro power budget management. Nutné testovat vliv na training throughput |
|
||
| **GPU clock locking** | `nvidia-smi -ac <clock,mem>` — locked clock frekvence pro stabilní benchmarky. Aplikace až po `nvidia-persistenced` |
|
||
| **PCIe Gen** | GPU v PCIe Gen4 slotu (místo Gen5) → bottleneck pro data transfer CPU↔GPU. Důležité pro FSDP sharding |
|
||
|
||
### Doporučené OS per use case
|
||
|
||
| Use case | OS | Zdůvodnění |
|
||
|----------|-----|-------|
|
||
| **DGX cluster (produkce)** | DGX OS / Ubuntu 22.04 LTS | NVIDIA standard, nejlepší driver support |
|
||
| **Enterprise K8s (OpenShift)** | RHEL 9 / RHCOS | Red Hat support, GPU Operator kompatibilní |
|
||
| **Vanilla K8s (on-prem)** | Ubuntu 22.04 LTS + Flatcar (workers) | Nejširší community support, Flatcar pro minimal footprint |
|
||
| **Slurm cluster (HPC/AI)** | Rocky Linux 9 / Ubuntu 22.04 LTS | EL ekosystém (Lustre, OFED) nebo Ubuntu (community) |
|
||
| **Výzkum / rapid prototyping** | Ubuntu 24.04 LTS | Nejnovější CUDA, PyTorch, driver support |
|
||
| **Edge inference** | NVIDIA JetPack / Ubuntu (ARM) | Embedded GPU (Jetson Orin, AGX) |
|
||
|
||
---
|
||
|
||
## AI-ready datové centrum — check-list
|
||
|
||
| Oblast | Požadavek |
|
||
|--------|-----------|
|
||
| **Power** | 30–120 kW/rack, HVDC (400 V DC), UPS s podporou GPU špiček |
|
||
| **Cooling** | Liquid cooling ready (direct-to-chip), rear-door pro 30+ kW |
|
||
| **Network** | InfiniBand (NDR/XDR) nebo RoCEv2, rail-optimized fat-tree |
|
||
| **Storage** | Parallel FS (Lustre/Weka), checkpoint bandwidth > 100 GB/s |
|
||
| **GPU density** | Max GPU/rack, minimalizace NVSwitch hopů |
|
||
| **Physical** | Podlaha nosnost 1 500+ kg/m², rack 52U–60U |
|
||
| **Security** | Tenant isolation, network segmentation, data encryption |
|
||
| **Monitoring** | DCGM, NCCL health checks, thermals, power capping |
|
||
|
||
---
|
||
|
||
## Omezení modelů a propustnosti
|
||
|
||
### Model size per GPU
|
||
|
||
Maximální velikost modelu, který se vejde na jednu GPU, závisí na HBM kapacitě a precision:
|
||
|
||
| GPU | HBM | FP32 | FP16/BF16 | INT8 | INT4 |
|
||
|-----|-----|------|-----------|------|------|
|
||
| **H100 80GB** | 80 GB | ~10B | ~40B | ~80B | ~160B |
|
||
| **H200 141GB** | 141 GB | ~18B | ~70B | ~140B | ~280B |
|
||
| **B200 192GB** | 192 GB | ~24B | ~96B | ~192B | ~384B |
|
||
| **MI300X 192GB** | 192 GB | ~24B | ~96B | ~192B | ~384B |
|
||
| **A100 80GB** | 80 GB | ~10B | ~40B | ~80B | ~160B |
|
||
| **GB200 (192+480)** | 192 GB GPU + 480 GB Grace | — | ~96B + CPU offload | — | — |
|
||
|
||
*Hodnoty orientační: 1B parametrů ≈ 2 GB FP16 ≈ 4 GB FP32 ≈ 1 GB INT8 ≈ 0,5 GB INT4. Reálně odečíst ~10–15 % HBM pro activations, KV cache, optimizer states.*
|
||
|
||
### Memory breakdown inference
|
||
|
||
| Komponenta | Llama 3 70B (FP16) | Llama 3 8B (FP16) |
|
||
|------------|-------------------|-------------------|
|
||
| Model weights | 140 GB | 16 GB |
|
||
| KV cache (4K context, batch 1) | ~2 GB | ~0,2 GB |
|
||
| KV cache (128K context, batch 1) | ~60 GB | ~6,5 GB |
|
||
| Activations (peak) | ~5 GB | ~1 GB |
|
||
| **Celkem 4K ctx** | ~147 GB | ~17 GB |
|
||
| **Celkem 128K ctx** | ~205 GB | ~23 GB |
|
||
|
||
**Závěr:** Llama 3 70B v FP16 se nevejde na jednu H100 (80 GB). Nutné: INT8 (170 GB → 2× H100), INT4 (85 GB → 1× H200), nebo tensor parallelism.
|
||
|
||
### Context length vs memory
|
||
|
||
| Context | KV cache 70B (FP16) | KV cache 8B (FP16) | Poznámka |
|
||
|---------|-------------------|-------------------|----------|
|
||
| 4K | ~2,2 GB | ~0,25 GB | Běžný chat |
|
||
| 32K | ~18 GB | ~2 GB | Dokumenty |
|
||
| 128K | ~72 GB | ~8 GB | Long-context (Claude, Gemini) |
|
||
| 1M | ~560 GB | ~64 GB | Experimentální (Gemini 1.5 Pro) |
|
||
|
||
KV cache je **lineární s délkou kontextu** a kvadratická s počtem hlav pozornosti. Pro long-context je kritická.
|
||
|
||
### Throughput inference
|
||
|
||
| Model | GPU | Precision | Batch size | Tokens/s | QPS (1K output) |
|
||
|-------|-----|-----------|-----------|----------|-----------------|
|
||
| Llama 3 8B | H100 | FP16 | 1 | ~800 | ~0,8 |
|
||
| Llama 3 8B | H100 | FP16 | 128 | ~4 500 | ~35 |
|
||
| Llama 3 8B | H100 | INT4 | 128 | ~8 000 | ~62 |
|
||
| Llama 3 70B | 4× H100 | FP16 | 1 | ~180 | ~0,18 |
|
||
| Llama 3 70B | 4× H100 | INT4 | 64 | ~1 200 | ~19 |
|
||
| Llama 3 70B | 8× H100 | FP16 (TP=8) | 128 | ~2 500 | ~20 |
|
||
| DeepSeek-R1 671B | 8× H200 | FP8 (MoE) | 64 | ~500 | ~8 |
|
||
| GPT-4 class (est.) | — | — | — | ~100–300 | ~1–3 |
|
||
|
||
**Poznámky:**
|
||
- QPS (queries per second) závisí na output délce (1K tokenů ≈ ~1 query)
|
||
- Batch size zvyšuje throughput, ale zvyšuje TTFB (time to first token)
|
||
- Tensor Parallelism (TP) škáluje, ale komunikační režba roste lineárně
|
||
|
||
### Training limits
|
||
|
||
#### Scaling efficiency
|
||
|
||
| Počet GPU | Model | Efficiency | Důvod |
|
||
|-----------|-------|-----------|-------|
|
||
| 8 (1 node) | Llama 3 8B | ~95 % | NVLink intra-node |
|
||
| 64 (8 nodes) | Llama 3 8B | ~85 % | IB inter-node |
|
||
| 512 (64 nodes) | Llama 3 70B | ~75 % | Komunikační režie |
|
||
| 4 096 (512 nodes) | Llama 3 70B | ~60 % | Pipeline bubble, network |
|
||
| 16 384 (2 048 nodes) | Llama 3 405B | ~45 % | Synchronous SGD overhead |
|
||
|
||
**Poznámka:** Efficiency = (actual throughput) / (ideal linear speedup). Klesá logaritmicky s počtem GPU.
|
||
|
||
#### Memory breakdown training
|
||
|
||
| Komponenta | Llama 3 70B (BF16) | Llama 3 8B (BF16) |
|
||
|------------|-------------------|-------------------|
|
||
| Model weights | 140 GB | 16 GB |
|
||
| Optimizer states (Adam) | 280 GB | 32 GB |
|
||
| Gradients | 140 GB | 16 GB |
|
||
| Activations (peak) | ~30 GB | ~4 GB |
|
||
| **Celkem (DDP)** | ~590 GB | ~68 GB |
|
||
| **Celkem (FSDP shard=8)** | ~74 GB | ~8,5 GB |
|
||
|
||
**Závěr:** FSDP (Fully Sharded Data Parallelism) je nutný pro trénování modelů > 10B. Adam optimizer zdvojnásobuje memory oproti inference (weights + optimizer + gradients).
|
||
|
||
#### Time to train
|
||
|
||
| Model | GPU count | GPU type | Precision | Time | Cost (on-prem odhad) |
|
||
|-------|-----------|---------|-----------|------|---------------------|
|
||
| Llama 3 8B | 64 | H100 | BF16 | ~3 dny | ~$5 000 |
|
||
| Llama 3 70B | 512 | H100 | BF16 | ~14 dní | ~$100 000 |
|
||
| Llama 3 405B | 16 384 | H100 | BF16 | ~60 dní | ~$14 M |
|
||
| DeepSeek-R1 671B (MoE) | 2 048 | H800 | BF16 | ~30 dní | ~$6 M |
|
||
| GPT-4 (est.) | 25 000 | A100/H100 | Mixed | ~90–100 dní | ~$100 M |
|
||
|
||
### Power a thermal limity
|
||
|
||
| Konfigurace | TDP limit | Throughput ztráta | Důvod |
|
||
|-------------|-----------|------------------|-------|
|
||
| H100 SXM | 700 W (default) | 0 % | Nominální |
|
||
| H100 SXM | 600 W (-15 %) | ~5–8 % | Power capping |
|
||
| H100 SXM | 500 W (-30 %) | ~15–25 % | Výrazný throttling |
|
||
| H100 SXM | 400 W (-43 %) | ~30–50 % | Jen pro emergency |
|
||
| DGX H100 (8×) | 5,6 kW (max) | 0 % | Nutné liquid cooling |
|
||
| DGX H100 (8×) | 4,5 kW (air) | ~10–15 % | Rear-door heat exchanger |
|
||
|
||
GPU throttluje při překročení TDP nebo teploty (85°C+). Power capping je lineární korelace s frekvencí, ale nelineární s propustností.
|
||
|
||
### API a provozní limity
|
||
|
||
| Limit | Popis | Typická hodnota |
|
||
|-------|-------|-----------------|
|
||
| **Rate limit** | Max requestů za minutu/hodinu | 100–10 000 RPM (dle tieru) |
|
||
| **Tokens per minute (TPM)** | Max tokenů za minutu | 1M–300M (dle modelu) |
|
||
| **Context window** | Max vstupních tokenů | 4K–2M (dle modelu) |
|
||
| **Max output tokens** | Max vygenerovaných tokenů | 4K–32K (dle modelu) |
|
||
| **Concurrent requests** | Počet paralelních requestů | 10–10 000 (dle backendu) |
|
||
| **Batch window** | Čas na sebírání batch | 0–20 s (vLLM, TGI) |
|
||
| **TTFB timeout** | Max latence na první token | 30–120 s |
|
||
| **Idle timeout** | GPU idle → škálování na 0 | 5–15 min (cloud) |
|
||
|
||
### Limity per deployment model
|
||
|
||
| Model | Samostatný HW | Managed cloud (SageMaker, Vertex) | API (OpenAI, Anthropic) |
|
||
|-------|--------------|----------------------------------|------------------------|
|
||
| **Model size** | Limitován HBM (max 192 GB/GPU) | Neomezen (škálování cluster) | Neomezen |
|
||
| **Queries** | Limitován GPU count | Auto-scaling | Rate limit (dle tieru) |
|
||
| **Latency** | < 10 ms (same node) | 10–100 ms (network hop) | 100 ms – 10 s |
|
||
| **Customization** | Plná (fine-tuning, quantization) | Managed (SageMaker, Bedrock) | Pouze prompt engineering |
|
||
| **Data privacy** | Ano (on-prem) | Smluvní (region, encryption) | Omezená |
|
||
| **Cost per 1M tokens** | ~$0,10–0,50 (FP16 inference) | ~$0,20–1,00 | ~$0,15–15,00 |
|
||
| **Max context** | 128K+ (dle GPU count) | 128K+ | 32K–2M |
|
||
| **Cold start** | 0 (always-on) | 30 s – 5 min | 0 (shared infra) |
|
||
|
||
---
|
||
|
||
## Ceny GPU a poměr cena/výkon (2026)
|
||
|
||
> Ceny jsou orientační — NVIDIA nezveřejňuje oficiální ceník pro datacenter GPU. Cloud ceny dle veřejných providerů (Q2 2026). Při koupi HW se cena liší dle objemu, resellera a regionu.
|
||
|
||
### Pořizovací cena (buy)
|
||
|
||
| GPU | Cena/GPU | Cena 8× GPU baseboard | $/PFLOPS (FP16) | Poznámka |
|
||
|-----|---------|----------------------|----------------|----------|
|
||
| **H100 SXM** | $27 000–40 000 | ~$200 000 | $25 000 | Scareita 2023–2024, nyní stabilizace |
|
||
| **H200 SXM** | $35 000–50 000 | ~$280 000 | ~$35 000 | Upgrade H100, HBM3e |
|
||
| **B200** | ~$60 000–70 000 | ~$500 000+ | ~$31 000 | Blackwell, FP4 support |
|
||
| **B100** | ~$30 000 | ~$240 000 | ~$20 000 | Nižší cena než B200, podobný výkon FP8 |
|
||
| **GB200** (Grace+Blackwell) | ~$70 000–100 000 | ~$2 000 000 (rack) | — | CPU+GPU unified, high-density |
|
||
| **A100 80GB** | ~$10 000–15 000 | ~$120 000 | ~$19 200 | Předchozí generace, stále relevantní |
|
||
| **MI300X** | ~$12 000–18 000 | ~$100 000 | ~$9 600 | AMD, 192 GB HBM3 |
|
||
| **Gaudi 3** | ~$15 625 | ~$125 000 | **$8 515** | Intel, nejlepší $/PFLOPS |
|
||
| **L40S** | ~$8 000–10 000 | — | — | Inference, enterprise |
|
||
|
||
### Cloud ceny (on-demand $/GPU/hr)
|
||
|
||
| GPU | Nejdostupnější | Mid-range (CoreWeave, Lambda) | Hyperscaler (AWS, GCP, Azure) |
|
||
|-----|--------------|-------------------------------|-------------------------------|
|
||
| **H100 SXM** | $1.38 (Thunder) | $2.89–3.29 | $4.15–6.88 |
|
||
| **H100 PCIe** | $2.01 (Spheron) | $2.50 | — |
|
||
| **H200 SXM** | $3.89 (Spheron) | $4.54 | $5.00+ |
|
||
| **B200** | **$3.39** (Spheron) | $6.02 | $14.24 (AWS) |
|
||
| **B200** | **$2.12** (spot) | — | — |
|
||
| **GB200** | $3.50 (Runcrate) | $5.85 (Oracle) | $6.95 (GCP) |
|
||
| **MI300X** | **$1.50** (TensorWave) | $1.85 (Vultr) | $7.86 (Azure) |
|
||
| **A100 80GB** | $1.07 (Spheron) | $1.50–2.00 | $3.00+ |
|
||
| **Gaudi 3** | ~$1.50–2.50 | — | — |
|
||
| **L40S** | $0.91 (Spheron) | $1.50–2.00 | — |
|
||
|
||
### Cena za inferenci ($/M tokenů)
|
||
|
||
| GPU | Provider | $/hr | Est. tok/s | $/M tok |
|
||
|-----|----------|------|-----------|--------|
|
||
| **B200** | Spheron | $3.39 | ~4 000 | **$0.42** |
|
||
| **B200** (spot) | Spheron | $2.12 | ~4 000 | **$0.15** |
|
||
| **H100 PCIe** | Spheron | $2.01 | ~1 200 | $0.47 |
|
||
| **A100 80GB** | Spheron | $1.07 | ~520 | $0.57 |
|
||
| **H100 SXM** | AWS | $6.88 | ~1 200 | $1.59 |
|
||
| **H200 SXM** | Spheron | $4.54 | ~1 800 | $0.70 |
|
||
| **L40S** | Spheron | $0.91 | ~450 | $0.56 |
|
||
|
||
*Hodnoty pro Llama 3 70B (INT8, batch=1, output 1K tok). Reálné hodnoty se liší dle batch size, kontextu a kvantizace.*
|
||
|
||
### Cena za GB HBM
|
||
|
||
| GPU | HBM | Cena/hr cloud | $/GB/hr | Vhodnost pro memory-bound workloady |
|
||
|-----|-----|-------------|--------|-----------------------------------|
|
||
| **MI300X** | 192 GB | $1.50 | **$0.0078** | ✅ Nejlepší |
|
||
| **B200** | 192 GB | $3.39 | $0.0177 | ✅ Dobrý |
|
||
| **H200** | 141 GB | $3.89 | $0.0276 | ⚠️ |
|
||
| **H100 SXM** | 80 GB | $1.38 | $0.0173 | ⚠️ Jen do 70B modelů |
|
||
| **GB200** | 384 GB | $3.50 | $0.0091 | ✅✅ (2× MI300X kapacita) |
|
||
|
||
### Poměr cena/výkon dle scénáře
|
||
|
||
| Scénář | Vítěz | Zdůvodnění |
|
||
|--------|-------|-------|
|
||
| **Absolutní výkon** (cena není limit) | **GB200 DGX NVL72** | 72× GPU, 18 PFLOPS FP8, 384 GB HBM/GPU |
|
||
| **Cloud inference** — nejlepší $/token | **B200 spot** | $0.15/M tok; 4× throughput H100 při nižší ceně |
|
||
| **Cloud inference** — on-demand | **B200** | $0.42/M tok |
|
||
| **Cloud inference** — rozpočet | **A100 / L40S** | $0.57–0.56/M tok |
|
||
| **Training** — cena/výkon při koupi | **Gaudi 3** | $8 515/PFLOPS, 2.5–3× lepší než H100 |
|
||
| **Training** — cloud | **H100 SXM** | $1.38/hr, CUDA ekosystém, NCCL |
|
||
| **Memory-bound** — long context, 70B+ | **MI300X / GB200** | 192–384 GB, $0.0078–0.0091/GB |
|
||
| **Ekosystém + bezpečná volba** | **H100/H200** | CUDA, nejširší SW, NVIDIA tools |
|
||
| **Spot / preemptible** — nejnižší cena | **A100 / H100** | $1.07–1.38/hr, 50–90 % sleva oproti on-demand |
|
||
|
||
### Trendy 2026
|
||
|
||
- **H100** — cena klesla o 64 % z peaku $8/hr na $1.38–2.89/hr, pak rebound o 40 % díky inference boomu
|
||
- **B200** — nový high-end, $3.39/hr cloud → ~$0.15/M tok na spotu — benchmark pro inference
|
||
- **MI300X** — nabídka roste (TensorWave, Vultr, CoreWeave, Oracle, Azure), cena od $1.50/hr
|
||
- **Gaudi 3** — nejlepší $/PFLOPS při koupi, ale úzký ekosystém a omezená cloud dostupnost
|
||
- **Market se bifurkoval** — starší generace (H100, A100) komoditizují, nová (B200, GB200) drží prémii
|
||
|
||
## Související
|
||
|
||
- [GPU.md](GPU.md) — GPU architektura, NVIDIA/AMD, vGPU, MIG
|
||
- [NETWORKING.md](NETWORKING.md) — InfiniBand, RoCE, network topologie
|
||
- [STORAGE.md](STORAGE.md) — parallel filesystem, object store
|
||
- [DATACENTERS.md](DATACENTERS.md) — DC layout, power, cooling
|
||
- [CLOUD.md](CLOUD.md) — cloud AI služby (SageMaker, Vertex AI)
|
||
|
||
## Zdroje
|
||
|
||
Odkazy, knihy a standardy: [sources/infrastructure/sources.md](sources/infrastructure/sources.md)
|
||
|
||
*Poslední revize: 2026-06-18* |