1064 lines
55 KiB
Markdown
1064 lines
55 KiB
Markdown
# 🏭 Datacenters
|
||
|
||
## Tier classification (TIA-942 / Uptime Institute)
|
||
|
||
| Tier | Availability | Downtime / year | Redundancy |
|
||
|------|-------------|-----------------|------------|
|
||
| **Tier I** | 99.671 % | 28.8 h | N — no redundancy |
|
||
| **Tier II** | 99.741 % | 22.7 h | N+1 — redundant components |
|
||
| **Tier III** | 99.982 % | 1.6 h | N+1 — concurrently maintainable |
|
||
| **Tier IV** | 99.995 % | 26.3 min | 2N+1 — fault tolerant |
|
||
|
||
## Key subsystems
|
||
|
||
| System | Description |
|
||
|--------|-------------|
|
||
| **Power** | UPS, generators (diesel), ATS, PDU, redundant feeds (A/B feed) |
|
||
| **Cooling** | CRAC/CRAH, chilled water, free cooling, containment (hot/cold aisle) |
|
||
| **Physical security** | CCTV, biometric access, mantrap, rack security locks |
|
||
| **Cabling** | Structured cabling (Cat6A/7/8, OM3/OM4 single-mode fiber), patch panels |
|
||
| **Fire suppression** | Alarm, inert gases (Novec, FM-200), VESDA (very early smoke detection) |
|
||
| **Monitoring** | DCIM (Data Center Infrastructure Management), SNMP, BMS (Building Management System) |
|
||
|
||
## Aisle containment
|
||
|
||
```
|
||
┌────────────────────────────────────┐
|
||
│ Rack Row │
|
||
│ ┌──┐ ┌──┐ ┌──┐ ┌──┐ ┌──┐ ┌──┐ │
|
||
Cold │ │ │ │ │ │ │ │ │ │ │ │ │ │ Cold
|
||
Aisle <──│ └──┘ └──┘ └──┘ └──┘ └──┘ └──┘ ──> Aisle
|
||
│ ┌──┐ ┌──┐ ┌──┐ ┌──┐ ┌──┐ ┌──┐ │
|
||
Hot │ │ │ │ │ │ │ │ │ │ │ │ │ │ Hot
|
||
Aisle ──>│ └──┘ └──┘ └──┘ └──┘ └──┘ └──┘ <── Aisle
|
||
└────────────────────────────────────┘
|
||
```
|
||
|
||
## Environmental classes (ASHRAE TC 9.9)
|
||
|
||
ASHRAE Technical Committee 9.9 defines temperature and humidity envelopes for IT equipment in DC.
|
||
|
||
| Class | Temperature (recommended) | Temperature (allowable) | Usage |
|
||
|-------|--------------------------|-------------------------|-------|
|
||
| **A1** | 18-27 °C | 15-32 °C | Enterprise DC, strict control |
|
||
| **A2** | 18-27 °C | 10-35 °C | Standard DC |
|
||
| **A3** | 18-27 °C | 5-40 °C | Looser environment |
|
||
| **A4** | 18-27 °C | 5-45 °C | Maximum cooling savings |
|
||
| **H1** | 18-22 °C | 5-25 °C | High-density air-cooled (AI/ML) |
|
||
|
||
- 5th edition (2021) added class H1 for high-density and extended liquid cooling W-classes (W17, W27, W32, W40, W45, W+)
|
||
- 2024: new S-classes for Technology Cooling System (TCS) liquid cooling
|
||
- Humidity: recommended −9 °C DP to 70 % RH (at low pollutants); max 50 % RH at high corrosivity
|
||
|
||
## Power
|
||
|
||
### Power chain
|
||
|
||
```
|
||
Grid ──> Transformer ──> UPS ──> PDU ──> Rack PDU ──> Server PSU
|
||
│
|
||
├──> Generator (ATS switches on outage)
|
||
└──> STS/ATS (Static Transfer Switch)
|
||
```
|
||
|
||
A/B feed topology:
|
||
```
|
||
Grid A ──> UPS A ──> PDU A1 ──> Rack PDU A ──> PSU A (server)
|
||
│
|
||
Grid B ──> UPS B ──> PDU B1 ──> Rack PDU B ──> PSU B (server)
|
||
```
|
||
Each server has 2 PSUs — each powered from a different branch (A/B). On failure of one branch, the server continues without interruption.
|
||
|
||
### UPS types
|
||
|
||
| Classification | IEC 62040-3 | Description | Switching | Use case |
|
||
|--------------|-------------|-------------|-----------|----------|
|
||
| **VFD** (Voltage & Frequency Dependent) | Passive standby | UPS in bypass, switches to inverter on failure | 4-10 ms | SOHO, edge |
|
||
| **VI** (Voltage Independent) | Line-interactive | Voltage regulation via autotransformer | 2-4 ms | Smaller racks, office |
|
||
| **VFI** (Voltage & Frequency Independent) | Double-conversion | AC → DC → AC, full isolation, zero switching time | 0 ms | Enterprise DC, Tier III/IV |
|
||
|
||
For DC the standard is **VFI (double-conversion)** — online UPS, zero switching time, full isolation from the grid.
|
||
|
||
### Battery technologies
|
||
|
||
| Type | Density (Wh/L) | Lifespan (cycles) | Lifespan (years) | Temperature | Cost/kWh | Note |
|
||
|------|---------------|-------------------|------------------|-------------|----------|------|
|
||
| **VRLA** (AGM/Gel) | 50-80 | 200-500 | 3-5 | 20-25 °C | ~$150-200 | Cheap, large, heavy, temperature sensitive |
|
||
| **Li-ion (LFP)** | 200-350 | 3000-5000 | 10-15 | 0-40 °C | ~$300-500 | Small, light, long life, BMS required |
|
||
| **Li-ion (NMC)** | 250-400 | 1000-2000 | 8-12 | 0-40 °C | ~$250-400 | Higher density, thermal runaway risk |
|
||
| **NiCd** | 80-150 | 1000-2000 | 10-15 | −20-50 °C | ~$400-600 | Extreme temperatures, memory effect |
|
||
| **Flow battery** (V/Zn/Br) | 20-40 | 10,000+ | 20+ | 10-35 °C | ~$500-800 | Unlimited cycles, large, long-term backup |
|
||
|
||
Li-ion (LFP) is becoming the standard for new DCs due to longer life, smaller footprint, and better behavior at high temperatures.
|
||
|
||
### Generator sizing
|
||
|
||
| Variant | Size | Fuel | Start time | Run time | Use case |
|
||
|---------|------|------|------------|----------|----------|
|
||
| **Diesel** | 500-2500 kVA | Diesel | 10-30 s | 24-72 h (depending on tank) | Standard for enterprise DC |
|
||
| **Nat. gas** | 200-1500 kVA | Natural gas | 10-30 s | Unlimited (pipeline) | Less common, lower emissions |
|
||
| **CHP** (cogeneration) | 500-2000 kVA | Natural gas | 5-15 min | Unlimited | Combined power + cooling (absorption chiller) |
|
||
|
||
Sizing: Generator should cover 100 % IT load + 100 % cooling load (incl. chillers) — typically 1.3-1.8× IT load. Diesel tank min. for 24 h operation, commonly 48-72 h. Daily consumption ~0.3-0.4 L/kWh.
|
||
|
||
### ATS vs STS
|
||
|
||
| Feature | ATS (Automatic Transfer Switch) | STS (Static Transfer Switch) |
|
||
|---------|--------------------------------|-----------------------------|
|
||
| **Switching** | 4-10 ms (mechanical relay) | < 4 ms (thyristor) |
|
||
| **Lifespan** | ~10,000 switches | Unlimited (solid-state) |
|
||
| **Cost** | Low | High (~3-5× ATS) |
|
||
| **Use case** | Generator → UPS feed | Between two UPS outputs |
|
||
|
||
### PDU types
|
||
|
||
| Type | Description | Use case |
|
||
|------|-------------|----------|
|
||
| **Basic** | Passive splitter (no monitoring) | Edge, office |
|
||
| **Metered** | Current measurement at PDU level | Standard DC |
|
||
| **Monitored** | Measurement per outlet, SNMP, web GUI | Enterprise DC |
|
||
| **Switched** | On/off per outlet, remote reboot | Enterprise DC, colo |
|
||
| **High-density** | 3-phase, 60-100 A, C19 outlets | GPU/HPC/AI racks |
|
||
|
||
### Power calculation
|
||
|
||
```
|
||
Total Power = Σ(P_server + P_storage + P_network + P_cooling + P_losses)
|
||
|
||
P_server = P_idle + (P_max - P_idle) × Utilization%
|
||
P_cooling = P_IT / PUE
|
||
|
||
Example:
|
||
100 servers × 500 W (avg) = 50 kW IT load
|
||
PUE = 1.5 → total 75 kW
|
||
UPS + generator → sized for 75 kW × 1.2 (safety factor) = 90 kW
|
||
```
|
||
|
||
### PUE (Power Usage Effectiveness)
|
||
|
||
```
|
||
PUE = Total Facility Energy / IT Equipment Energy
|
||
```
|
||
|
||
| PUE | Efficiency | Type |
|
||
|-----|-----------|------|
|
||
| 1.0-1.1 | Excellent | Hyperscale (Google, Meta) |
|
||
| 1.1-1.3 | Very good | Modern DC |
|
||
| 1.3-1.6 | Good / average | Enterprise DC |
|
||
| 1.6-2.0 | Below average | Older DC |
|
||
| >2.0 | Poor | Legacy |
|
||
|
||
PUE is measured at the whole DC level, not per rack. Includes: UPS losses, cooling, lighting, distribution losses. Excludes: well-to-tank fuel production, embodied carbon. Target for modern DC: PUE < 1.2.
|
||
|
||
### WUE and CUE
|
||
|
||
| Metric | Description | Formula | Target |
|
||
|--------|-------------|---------|--------|
|
||
| **WUE** (Water Usage Effectiveness) | Water consumption per IT energy | WUE = Annual Water Usage / IT Energy (L/kWh) | < 0.5 L/kWh |
|
||
| **CUE** (Carbon Usage Effectiveness) | CO₂ emissions per IT energy | CUE = Total CO₂ / IT Energy (kg CO₂/kWh) | < 0.2 kg CO₂/kWh |
|
||
|
||
WUE is critical in dry regions (southwest US, Australia, Middle East). Adiabatic cooling consumes significantly more water than closed-loop cooling.
|
||
|
||
### 3-phase vs Single-phase
|
||
|
||
| Feature | Single-phase (230 V) | 3-phase (400 V) |
|
||
|---------|---------------------|-----------------|
|
||
| **Voltage** | 230 V (L-N) | 230/400 V (L-N/L-L) |
|
||
| **Power per feed** | ~7.4 kW (32 A) | ~22 kW (32 A, 3-ph) |
|
||
| **Efficiency** | Lower (more losses) | Higher (lower current) |
|
||
| **Use case** | Smaller racks, office | Standard in DC, high-density |
|
||
| **PDU** | Single-phase (C13/C19) | 3-phase (C13/C19, 3-ph monitoring) |
|
||
| **Balancing** | Automatic | Phase balancing required (L1/L2/L3) |
|
||
|
||
### Rack power density
|
||
|
||
| Cat. | Type | kW/rack | Power | Cooling |
|
||
|------|------|---------|-------|---------|
|
||
| Low | Office, storage | 1-3 kW | 1-ph, 16 A | Air (free cooling) |
|
||
| Medium | Standard compute | 5-10 kW | 3-ph, 32 A | Air (CRAC/CRAH) |
|
||
| High | GPU, HPC | 15-30 kW | 3-ph, 60 A | Air + liquid assist |
|
||
| Ultra | AI/ML clusters | 40-100+ kW | 3-ph, 100+ A | Direct-to-chip / immersion |
|
||
|
||
### Rack PDU connectors
|
||
|
||
| Connector | Max current | Device type |
|
||
|-----------|-------------|-------------|
|
||
| **C13** | 10 A (250 V) | Servers, switches, 1U |
|
||
| **C19** | 16 A (250 V) | Higher power servers, UPS |
|
||
| **IEC 60309** (3-ph) | 16-125 A | Rack PDU inputs |
|
||
| **NEMA L6-30** | 30 A (250 V) | US spec |
|
||
|
||
## Cooling
|
||
|
||
### Cooling — technology overview
|
||
|
||
| Technology | Type | Output (kW/rack) | Typical PUE | CAPEX | Use case |
|
||
|-----------|------|-----------------|-------------|-------|----------|
|
||
| **Free air cooling** | Air | < 5 | 1.05-1.15 | Low | Climatically suitable locations |
|
||
| **CRAC (DX)** | Air | 5-10 | 1.4-1.8 | Medium | Smaller DC, retrofit |
|
||
| **CRAH (CW)** | Air | 5-15 | 1.2-1.5 | High | Enterprise DC |
|
||
| **In-row cooling** | Air | 10-25 | 1.2-1.4 | High | High-density racks |
|
||
| **Rear-door HX** | Hybrid | 15-30 | 1.1-1.3 | Medium | Retrofits, GPU |
|
||
| **Direct-to-chip** | Liquid | 40-100+ | 1.05-1.15 | High | AI/ML, HPC |
|
||
| **Immersion (single-phase)** | Liquid | 50-100+ | 1.03-1.10 | High | Bitcoin, hyperscale |
|
||
| **Immersion (two-phase)** | Liquid | 100-200+ | 1.03-1.08 | Very high | Extreme density |
|
||
|
||
### Chilled water vs Direct Expansion (DX)
|
||
|
||
| Feature | Chilled water (CW) | Direct Expansion (DX) |
|
||
|---------|-------------------|----------------------|
|
||
| **Medium** | Water + glycol | Refrigerant (R134a, R410A, R454B) |
|
||
| **CRAC/CRAH** | CRAH (Coolant-based) | CRAC (refrigerant compressor) |
|
||
| **Efficiency** | Higher (COP 5-7) | Lower (COP 2-4) |
|
||
| **Water temperature** | 7-12 °C (standard), 18-22 °C (high-temp) | −5-10 °C (evaporator) |
|
||
| **Complexity** | Higher (chillers, pumps, pipes, cooling tower) | Simpler |
|
||
| **Maintenance** | Higher (water treatment, legionella prevention) | Lower |
|
||
| **Use case** | Large DC > 500 kW, enterprise | Smaller DC, edge, retrofit |
|
||
|
||
### Containment types
|
||
|
||
| Type | Description | Efficiency | Implementation |
|
||
|------|-------------|------------|----------------|
|
||
| **Cold aisle containment (CAC)** | Enclosed cold aisle, warm air returns to room | High | Doors at aisle ends, ceiling panels |
|
||
| **Hot aisle containment (HAC)** | Enclosed hot aisle, warm air goes directly to return | Higher | Doors + ceiling panels, return to CRAH |
|
||
| **Chimney / rear duct** | Each rack has its own exhaust chimney to ceiling | Highest | Individual ducts per rack, expensive |
|
||
| **Open aisle** | No containment, cold and warm air mix | Low | Legacy, cheap |
|
||
|
||
Recommendation: CAC/HAC at density > 5 kW/rack. HAC is 5-10 % more efficient than CAC (warm air is directly extracted, does not mix with room).
|
||
|
||
### CFD modeling
|
||
|
||
Computational Fluid Dynamics (CFD) simulates airflow in DC before physical implementation:
|
||
- Identification of hot spots (warm air recirculation into cold aisle)
|
||
- Optimization of perforated tile positions
|
||
- Design of bypass airflow (cable openings, uncovered positions)
|
||
- Simulation of CRAH unit failure (what-if scenarios)
|
||
- Tools: Future Facilities (6Sigma DC), Ansys Fluent, OpenFOAM
|
||
|
||
### Free cooling
|
||
|
||
- **Air-side** — intake of outside air at suitable temperature (filtration, humidification)
|
||
- **Water-side** — use of cold water from outdoor chillers (strainer cycle) without compressor
|
||
- **Climate zone** — free cooling usable ~2000-8000 hours/year depending on location
|
||
- Scandinavia: 7000-8000 h/year
|
||
- Central Europe: 4000-6000 h/year
|
||
- Southern Europe: 2000-4000 h/year
|
||
- **Hybrid** — combination of free cooling + mechanical cooling (most common)
|
||
- **Economizer types**: Class A1 (dry cooler), Class A2 (evaporative), Class B (air-side)
|
||
|
||
### Liquid cooling detail
|
||
|
||
| Type | Inlet temperature | Capacity (kW/rack) | Medium | Installation |
|
||
|------|-----------------|-------------------|--------|-------------|
|
||
| **Cold plate (D2C)** | 20-45 °C | 40-100+ | Water, propylene glycol | CDU per rack or per row |
|
||
| **Rear-door HX** | 18-27 °C | 15-30 | Water | Passive, no server modification |
|
||
| **Immersion (1-ph)** | 35-50 °C | 50-100+ | Dielectric oil | Tank, CDU, heat exchanger |
|
||
| **Immersion (2-ph)** | 25-35 °C | 100-200+ | Dielectric (boiling) | Tank + condenser |
|
||
|
||
**CDU (Coolant Distribution Unit)**:
|
||
- Provides coolant temperature and pressure to racks
|
||
- Primary loop (facility water) + secondary loop (rack coolant)
|
||
- Sizing: 1 CDU per 4-8 racks (40-100 kW per CDU)
|
||
- Redundancy: N+1 CDU, dual coolant loops
|
||
|
||
**Water quality requirements**:
|
||
- Conductivity: < 1 µS/cm (demineralized water)
|
||
- pH: 6.5-8.0
|
||
- Particulates: < 50 µm (filtration)
|
||
- Corrosion prevention: inhibitors, glycol (10-30 %)
|
||
- Biological growth prevention: UV, biocides
|
||
|
||
### Adiabatic cooling
|
||
|
||
Using water evaporation to cool air:
|
||
- **Direct adiabatic** — air passes through water (media pad), cools and humidifies
|
||
- **Indirect adiabatic** — air cools via heat exchanger without direct contact with water
|
||
- **Water consumption**: 3-5 L/kWh (direct), 1-2 L/kWh (indirect)
|
||
- Efficiency depends on air humidity — more effective in dry climates
|
||
|
||
## Cabling and structured cabling
|
||
|
||
### TIA-942 cabling hierarchy
|
||
|
||
```
|
||
Entrance Room (ER)
|
||
│
|
||
├── Backbone cabling (fiber single-mode / multi-mode)
|
||
│ │
|
||
│ ├── Main Distribution Area (MDA)
|
||
│ │ │
|
||
│ │ ├── Horizontal Distribution Area (HDA)
|
||
│ │ │ │
|
||
│ │ │ └── Equipment Distribution Area (EDA) → rack
|
||
│ │ │
|
||
│ │ └── Intermediate Distribution Area (IDA) — optional
|
||
│ │
|
||
│ └── Telecommunication Room (TR) — for office
|
||
│
|
||
└── Backbone cabling (fiber / copper)
|
||
```
|
||
|
||
### Copper cabling categories
|
||
|
||
| Category | Frequency | Speed | Length | Connector | Use case |
|
||
|----------|-----------|-------|--------|-----------|----------|
|
||
| **Cat5e** | 100 MHz | 1 GbE | 100 m | RJ45 | Legacy, voice |
|
||
| **Cat6** | 250 MHz | 1 GbE (10 GbE up to 55 m) | 100 m (10 GbE: 55 m) | RJ45 | Standard DC, enterprise |
|
||
| **Cat6A** | 500 MHz | 10 GbE | 100 m | RJ45 | Standard for new DC |
|
||
| **Cat7** (GG45) | 600 MHz | 10 GbE | 100 m | GG45/TERA | Niche, replaced by Cat6A/8 |
|
||
| **Cat8.1** | 2000 MHz | 25/40 GbE | 30 m | RJ45 | Top-of-rack, storage |
|
||
| **Cat8.2** | 2000 MHz | 25/40 GbE | 30 m | GG45/TERA | Top-of-rack, storage |
|
||
|
||
In DC, **Cat6A** (10 GbE up to 100 m) is standard for horizontal cabling. Cat8 only for patch cables within a rack (up to 30 m).
|
||
|
||
### Fiber optic types
|
||
|
||
| Type | Core | Modal BW | Speed | Max length | Use case |
|
||
|------|------|----------|-------|-----------|----------|
|
||
| **OS1** (SM) | 9 µm | — | 100 GbE - 800 GbE | 10-80 km | Backbone, campus, WAN |
|
||
| **OS2** (SM) | 9 µm | — | 100 GbE - 800 GbE | 2-80 km (CWDM/DWDM) | Backbone, DWDM |
|
||
| **OM1** (MM) | 62.5 µm | 200 MHz·km | 1 GbE | 275 m | Legacy |
|
||
| **OM2** (MM) | 50 µm | 500 MHz·km | 10 GbE | 82 m | Legacy |
|
||
| **OM3** (MM) | 50 µm | 2000 MHz·km | 10 GbE up to 300 m, 100 GbE up to 100 m | 300 m (10G) | Standard DC, VCSEL |
|
||
| **OM4** (MM) | 50 µm | 4700 MHz·km | 100 GbE up to 150 m, 400 GbE up to 100 m | 550 m (10G) | High-performance DC standard |
|
||
| **OM5** (MM) | 50 µm | 4700+ MHz·km | 200/400 GbE SWDM | 150 m (100G) | Emerging, SWDM |
|
||
|
||
For new DC: **OM4** as standard for multi-mode, **OS2** for single-mode backbone (LR, DWDM). OM5 is not widely deployed — OM4 + parallel optics (SR4) is more common.
|
||
|
||
### Connector types
|
||
|
||
| Connector | Type | Insertion loss | Fiber count | Use case |
|
||
|-----------|------|---------------|-------------|----------|
|
||
| **LC** | Duplex | < 0.15 dB | 2 | Standard for SFP/SFP+/QSFP |
|
||
| **SC** | Duplex | < 0.2 dB | 2 | Older installations, patch panels |
|
||
| **MPO/MTP** (12-f) | Multi-fiber | < 0.35 dB | 12/24 | 40/100/400 GbE parallel |
|
||
| **MPO/MTP** (24-f) | Multi-fiber | < 0.5 dB | 24 | 400 GbE (SR4.2, DR4) |
|
||
| **SN** | Duplex (mini) | < 0.15 dB | 2 | High-density (QSFP-DD, OSFP) |
|
||
| **CS** | Duplex (mini) | < 0.15 dB | 2 | High-density (QSFP-DD, OSFP) |
|
||
|
||
### MPO/MTP polarity
|
||
|
||
| Method | Description | Use case |
|
||
|--------|-------------|----------|
|
||
| **Type A** (Straight) | Fiber 1→1, 2→2, ... | Duplex applications with cross-over at both ends |
|
||
| **Type B** (Crossed) | Fiber 1→12, 2→11, ... | Parallel optics (SR4, SR8) — standard |
|
||
| **Type C** (Pairs crossed) | Pairs 1-2→2-1, 3-4→4-3 | 40 GbE SR4 (4×10G) |
|
||
|
||
### Breakout cassettes
|
||
|
||
```
|
||
MPO (12-f) ──> Breakout cassette ──> 6× LC duplex (12 fibers = 6× duplex)
|
||
MPO (24-f) ──> Breakout cassette ──> 12× LC duplex (24 fibers = 12× duplex)
|
||
```
|
||
|
||
Use case: Connecting MPO ports (switch) with LC ports (servers, storage). Cassettes are in the patch panel, not in the active path.
|
||
|
||
### Copper vs fiber decision
|
||
|
||
| Criterion | Copper (Cat6A/8) | Fiber (OM4/OS2) |
|
||
|-----------|-----------------|-----------------|
|
||
| **Reach** | 30-100 m | 100 m - 80 km |
|
||
| **Speed** | 1-40 GbE | 1-800 GbE |
|
||
| **Transceiver cost** | Lower (RJ45) | Higher (SFP+/QSFP) |
|
||
| **Cable cost** | Lower | Higher (patch cord) |
|
||
| **Port power** | 2-5 W (25 GbE) | 1-3 W (25 GbE SR) |
|
||
| **EMI immunity** | Susceptible | Immune |
|
||
| **Weight (100 m)** | ~3-4 kg | ~0.5-1 kg |
|
||
| **Recommendation** | Up to 30 m, server→ToR switch | Backbone, storage, >30 m |
|
||
|
||
### Cabling best practices
|
||
|
||
- **Horizontal cabling**: max 90 m permanent link + 10 m patch cords (TIA-942)
|
||
- **Fiber management**: slack spools, cable managers, minimum bend radius 10× cable diameter
|
||
- **Color coding**: OS1/OS2 (yellow), OM3 (aqua), OM4 (magenta/purple), OM5 (lime green)
|
||
- **Labeling**: both ends, patch panels, faceplates — standard ANSI/TIA-606-B
|
||
- **Overhead vs underfloor**: overhead (ladder rack) is preferred in DC (better airflow, easier changes)
|
||
- **MPO cassettes**: plan 15-20 % fiber reserve for future needs
|
||
|
||
## Physical security
|
||
|
||
### Multi-layer security model (defense in depth)
|
||
|
||
```
|
||
Layer 1: Perimeter (fence, gate, guards)
|
||
Layer 2: Building (walls, locks, CCTV, card readers)
|
||
Layer 3: DC hall (biometrics, mantrap, CCTV, motion detection)
|
||
Layer 4: Rack / Cage (electronic locks, sensors)
|
||
Layer 5: Data (encryption, HSM, access control)
|
||
```
|
||
|
||
### Access control
|
||
|
||
| Method | Factor | Level | Note |
|
||
|--------|--------|-------|------|
|
||
| **RFID / proximity card** | Something you have | Standard | Basic access, cheap |
|
||
| **Smart card (PKI)** | Something you have + PIN | Medium | Certificate on card, anti-passback |
|
||
| **Biometric (fingerprint)** | Something you are | High | Fast, hygienic (touchless readers) |
|
||
| **Biometric (palm/finger vein)** | Something you are | Very high | Hard to forge, contactless |
|
||
| **Biometric (iris/retina)** | Something you are | Highest | Very accurate, slow, expensive |
|
||
| **Multi-factor** | 2+ factors | Highest | Card + biometrics + PIN — Tier IV DC |
|
||
|
||
### Mantrap design
|
||
|
||
```
|
||
Outer door ──> Mantrap (vestibule) ──> Inner door
|
||
│
|
||
├── Weight sensor (anti-tailgating)
|
||
├── CCTV (both doors)
|
||
├── Intercom (emergency exit)
|
||
└── Motion detector (in mantrap)
|
||
```
|
||
|
||
- Only one door opens at a time
|
||
- Anti-tailgating: weight sensor detects multiple persons
|
||
- Exit via breakout button + motion detection
|
||
- Emergency exit: panic bar + alarm
|
||
|
||
### CCTV
|
||
|
||
| Element | Recommendation |
|
||
|---------|----------------|
|
||
| **Resolution** | Min. 1080p, ideally 4K (6 MP+) |
|
||
| **FPS** | 15-30 FPS (recording), 30+ FPS (realtime monitoring) |
|
||
| **Retention** | Min. 30 days (90 days for audit) |
|
||
| **Storage** | NVR (on-prem), cloud (AWS KVS, Azure Video Indexer) |
|
||
| **AI analytics** | Face detection, ANPR (license plate), object detection |
|
||
| **Field of view** | Every door, every aisle — no blind spots |
|
||
|
||
### Asset tracking
|
||
|
||
| Technology | Accuracy | Cost | Use case |
|
||
|-----------|----------|------|----------|
|
||
| **Barcode** | Rack-level | Very low | Manual inventory |
|
||
| **RFID (passive)** | Rack-level (door sweep) | Low | Automatic rack open detection |
|
||
| **RFID (active, UWB)** | 10-30 cm | Medium | Real-time tracking |
|
||
| **Bluetooth BLE** | 1-3 m | Low | Approximate position |
|
||
| **GPS** | 1-10 m | Medium | Outdoor tracking |
|
||
|
||
## DC layout and design
|
||
|
||
### Raised floor vs Slab
|
||
|
||
| Feature | Raised floor | Slab (solid floor) |
|
||
|---------|-------------|-------------------|
|
||
| **Airflow** | Underfloor air distribution (raised floor as plenum) | Overhead air, in-row cooling |
|
||
| **Flexibility** | Easy addition of perforated tiles | Limited (overhead cooling required) |
|
||
| **Weight** | Limit 500-1000 kg/m² (depends on height) | Unlimited |
|
||
| **Cost** | Higher (~$200-400/m²) | Lower (~$100-200/m²) |
|
||
| **Height** | 600-900 mm (standard), 900-1200 mm (high-density) | — |
|
||
| **Trend** | Declining (shift to in-row/overhead cooling) | Growing (new DC, high-density) |
|
||
|
||
Modern high-density DC (AI/ML, GPU) are moving away from raised floor to slab + overhead/in-row cooling — higher rack weights (1000-2000 kg), inability to provide sufficient airflow through floor.
|
||
|
||
### Rack layout and dimensions
|
||
|
||
| Parameter | Standard | High-density | Note |
|
||
|-----------|----------|-------------|------|
|
||
| **Rack width** | 600 mm (19") | 600-750 mm | 750 mm for GPU (cabling, cooling) |
|
||
| **Rack depth** | 1000-1200 mm | 1200-1500 mm | GPU servers, longer cables |
|
||
| **Rack height** | 42U | 48U / 52U | Higher rack = better power density |
|
||
| **Aisle width (cold)** | 1200-1500 mm | 1500-1800 mm | Service access, airflow |
|
||
| **Aisle width (hot)** | 900-1200 mm | 1200-1500 mm | Narrower than cold |
|
||
| **Max rack load** | 500-800 kg | 1000-2000 kg | Floor reinforcement required |
|
||
|
||
### Space planning
|
||
|
||
```
|
||
For Tier III DC (example):
|
||
IT space: 1000 m²
|
||
└── 20 rows × 10 racks = 200 racks at 42U
|
||
└── 200 racks × 5 kW avg = 1 MW IT load
|
||
└── PUE 1.4 → 1.4 MW facility
|
||
Support spaces:
|
||
└── UPS + batteries: 200 m²
|
||
└── Generators: 100 m² (outdoor)
|
||
└── Cooling (chillers, cooling tower): 300 m²
|
||
└── Offices, storage, loading dock: 400 m²
|
||
Total: ~2000 m² (50% IT, 50% support)
|
||
```
|
||
|
||
### Zone approach (TIA-942)
|
||
|
||
| Zone | Description | Access | Security |
|
||
|------|-------------|--------|----------|
|
||
| **Z1** (Public) | Reception, offices | Free | Minimal |
|
||
| **Z2** (Office) | Administration, NOC | Employees + guests | RFID |
|
||
| **Z3** (DC support) | UPS, generators, cooling | DC operators | RFID + biometrics |
|
||
| **Z4** (DC hall) | Servers, storage, networking | DC operators + approved | RFID + biometrics + mantrap |
|
||
| **Z5** (Rack/cage) | Specific rack or cage | Only authorized personnel | Electronic lock |
|
||
|
||
## Fire suppression
|
||
|
||
### Detection
|
||
|
||
| System | Type | Detection time | False alarms | Use case |
|
||
|--------|------|----------------|--------------|----------|
|
||
| **VESDA** (Very Early Smoke Detection) | Aspiration, laser sensor | < 30 s (4 alarm levels) | Very low | Standard for DC |
|
||
| **Spot detection** | Ionization / optical smoke detector | 2-5 min | Medium | Legacy, smaller DC |
|
||
| **Heat detection** | Thermal detector (temperature / rate of rise) | 5-10 min | Very low | Backup for VESDA |
|
||
| **Line-type (LHD)** | Linear heat detection cable | 2-5 min | Low | Cable trays, above ceiling |
|
||
|
||
VESDA is the standard — active aspiration draws air from DC, laser sensor detects smoke particles at 4 levels (Alert → Action → Fire 1 → Fire 2). Enables intervention before visible smoke.
|
||
|
||
### Suppression systems
|
||
|
||
| System | Medium | Advantages | Disadvantages | Typical DC |
|
||
|--------|--------|------------|---------------|-----------|
|
||
| **Novec 1230** (FK-5-1-12) | Gas | Safe for people, zero ODP, short atmospheric lifetime (5 days) | Higher cost | Enterprise DC |
|
||
| **FM-200** (HFC-227ea) | Gas | Fast (10 s), effective | High GWP (3220), no ODP | Legacy DC |
|
||
| **Inergen** (IG-541) | Inert gas (52% N₂, 40% Ar, 8% CO₂) | Completely safe, natural gas | Large volume, high pressure | Enterprise DC |
|
||
| **Argonite** (IG-55) | 50% Ar, 50% N₂ | Safe, natural | Large volume, higher pressure | Enterprise DC |
|
||
| **Water mist** | Water (fine mist) | Cooling, smoke suppression, low cost | Water in DC (risk), local application only | Retrofits |
|
||
| **Pre-action sprinkler** | Water | Dual activation (detection + sprinkler) | Water risk, drainage required | Tier I-II |
|
||
|
||
**Concentration**: Novec (4-6 % volume), FM-200 (7-9 %), Inergen (35-50 %). Novec and Inergen are safe for breathing (min. 5-7 min evacuation).
|
||
|
||
### Detection zones
|
||
|
||
```
|
||
DC hall ──> zones of ~200 m² (max)
|
||
│
|
||
├── VESDA (each zone its own aspirator)
|
||
├── Smoke detectors (ceiling + floor)
|
||
└── Heat detection (backup)
|
||
```
|
||
|
||
## DCIM (Data Center Infrastructure Management)
|
||
|
||
### What DCIM covers
|
||
|
||
| Area | Metrics | Output |
|
||
|------|---------|--------|
|
||
| **Power** | Per PDU, per outlet, per rack, total | Capacity planning, PUE, kW/rack |
|
||
| **Cooling** | Temperature, humidity, airflow (sensors per rack) | Hot spot maps, airflow optimization |
|
||
| **Asset** | What is in which rack, U position, serial, warranty | Asset inventory, lease management |
|
||
| **Network** | Port utilization, patch panel connections | Patch management, port tracking |
|
||
| **Space** | Free U in rack, free racks | Capacity planning, "what-if" simulations |
|
||
|
||
### Tools
|
||
|
||
| Tool | Type | Platform | Cost | Note |
|
||
|------|------|----------|------|------|
|
||
| **Nlyte (Carrier)** | Enterprise DCIM | On-prem / Cloud | $$$ | Market leader, complex |
|
||
| **Sunbird DCIM** | Enterprise DCIM | Cloud | $$$ | Power monitoring, asset tracking |
|
||
| **Device42** | DCIM + IPAM | On-prem / Cloud | $$ | Integrated IPAM, CMDB |
|
||
| **NetBox** | Open source DCIM | On-prem | Free | IPAM, DCIM, asset tracking |
|
||
| **OpenDCIM** | Open source | On-prem | Free | Basic DCIM, asset management |
|
||
| **RackTables** | Open source | On-prem | Free | Simple, asset + networking |
|
||
| **Vendor-specific** | Dell OME, HPE OneView | On-prem | Part of HW | Vendor-specific only |
|
||
|
||
## Site selection
|
||
|
||
### Criteria for DC site selection
|
||
|
||
| Category | Criterion | Weight |
|
||
|----------|-----------|--------|
|
||
| **Power** | Electricity availability (grid capacity), cost/kWh, possibility of two independent feeds | High |
|
||
| **Connectivity** | Fiber backbone availability, number of connectivity providers, latency to major POP | High |
|
||
| **Natural risks** | Earthquakes, floods, hurricanes, tornadoes, wildfires — historical data + predictions | High |
|
||
| **Climate** | Average temperature, humidity (free cooling potential) | Medium |
|
||
| **Workforce** | Availability of technicians, DC operators, network/admin engineers | Medium |
|
||
| **Taxes and regulation** | Tax incentives, environmental regulations, building permits | Medium |
|
||
| **Security** | Crime, political stability, terrorist risk | High |
|
||
| **Transport accessibility** | Proximity to airport, highway (for HW deliveries, personnel) | Low |
|
||
|
||
### Natural risks — mapping
|
||
|
||
| Risk | Areas | Mitigation |
|
||
|------|-------|------------|
|
||
| **Earthquakes** | Pacific Ring of Fire (CA, Japan, Chile) | Base isolation, seismic bracing, flexible connections |
|
||
| **Hurricanes** | Caribbean, southeastern US, southeast Asia | Reinforced construction, generators above flood level |
|
||
| **Floods** | River valleys, coastal areas | Location outside flood zone, barriers |
|
||
| **Wildfires** | California, Australia, Mediterranean | Defensive zones, air filtration, monitoring |
|
||
|
||
### Power availability by region
|
||
|
||
| Region | Grid reliability | Cost/kWh (industrial) | Note |
|
||
|--------|-----------------|------------------------|------|
|
||
| **Northern Europe** (SE, NO, FI) | High (99.99 %) | $0.04-0.08 | Cheap green energy, cool climate |
|
||
| **Central Europe** (DE, NL, CZ) | High (99.99 %) | $0.10-0.20 | Stable, growing renewables |
|
||
| **Eastern US** (VA, NC) | High | $0.05-0.08 | Largest DC hub (Ashburn, VA) |
|
||
| **Western US** (CA, OR) | Medium (PG&E issues) | $0.10-0.15 | CALISO grid, blackout risk |
|
||
| **Singapore** | High | $0.15-0.20 | Moratorium on new DC (2023), water |
|
||
| **Dubai / UAE** | High | $0.06-0.10 | Cheap energy, high temperature (cooling) |
|
||
|
||
## Compliance and certification
|
||
|
||
| Standard / Certification | Area | Description |
|
||
|-------------------------|------|-------------|
|
||
| **TIA-942** (Rated 1-4) | DC design | Classification of redundancy, cabling, security (analogous to Uptime Tier) |
|
||
| **Uptime Institute** (Tier I-IV) | DC design | Operational certification, construction documentation |
|
||
| **ISO 27001** | ISMS | Information security, risk management |
|
||
| **ISO 27701** | Privacy | Extension of ISO 27001 for GDPR compliance |
|
||
| **SOC 2** (Type I/II) | Service org | Controls: Security, Availability, Confidentiality, Integrity, Privacy |
|
||
| **PCI DSS** | Payment cards | Physical security, access to cardholder data |
|
||
| **HIPAA** | Healthcare | USA, health data protection |
|
||
| **FedRAMP** | US government | Cloud service authorization, DC security |
|
||
| **GDPR** | EU | Personal data protection, data residency |
|
||
| **NIST SP 800-53** | DC security | Security control catalog for US federal |
|
||
| **ISO 14001** | EMS | Environmental management, sustainability |
|
||
|
||
## Sustainability
|
||
|
||
### Carbon footprint of DC
|
||
|
||
```
|
||
Total emissions = Scope 1 (direct) + Scope 2 (energy) + Scope 3 (supply chain)
|
||
Scope 1: Generators (diesel), refrigerant leaks
|
||
Scope 2: Purchased electricity (grid mix)
|
||
Scope 3: HW manufacturing, transport, EOL recycling (~60-80 % of total emissions)
|
||
```
|
||
|
||
### Emission reduction
|
||
|
||
| Measure | Impact on PUE | Emission reduction | Payback |
|
||
|---------|--------------|-------------------|---------|
|
||
| **Temperature increase** (22→27 °C) | −0.1-0.2 | 10-20 % cooling | Immediate |
|
||
| **Free cooling** | −0.1-0.3 | 20-40 % cooling | 1-2 years |
|
||
| **Liquid cooling** | −0.2-0.4 | 30-50 % cooling | 2-4 years |
|
||
| **LED lighting + sensors** | −0.01-0.02 | < 1 % | 1 year |
|
||
| **PPA (Power Purchase Agreement)** | — | 100 % Scope 2 | Variable |
|
||
| **Renewable sources** (rooftop solar) | — | 5-15 % consumption | 5-10 years |
|
||
| **Green generator** (HVO biodiesel) | — | 90 % CO₂ reduction | +30 % fuel cost |
|
||
|
||
### Sustainability certifications
|
||
|
||
| Certification | Description |
|
||
|--------------|-------------|
|
||
| **LEED** (BD+C: DC) | U.S. Green Building Council — design and construction |
|
||
| **BREEAM** | UK, European sustainability assessment |
|
||
| **Climate Neutral Data Centre Pact** (EU) | Self-regulatory, PUE < 1.4 by 2030 |
|
||
| **ISO 50001** | Energy management system |
|
||
| **Energy Star** | EPA, energy efficiency (US only) |
|
||
|
||
## Decision diagram — DC topology design
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Start(["DC design"]) --> TIER{"Required Tier?"}
|
||
TIER -->|"Tier I / II"| T1["N / N+1 redundancy<br/>Simple power, single path<br/>CRAC/CRAH, free cooling<br/>PUE 1.4-1.6, cost 1×"]
|
||
TIER -->|"Tier III"| T3["N+1, concurrently maintainable<br/>Dual path (A/B feed)<br/>Hot aisle containment<br/>PUE 1.2-1.4, cost 2×"]
|
||
TIER -->|"Tier IV"| T4["2N+1, fault tolerant<br/>Dual redundant + STS<br/>Hot + cold containment<br/>PUE 1.1-1.3, cost 3×"]
|
||
|
||
TIER --> POWER{"Power chain"}
|
||
POWER -->|"UPS"| UPS{"UPS type"}
|
||
UPS -->|"Enterprise DC"| UPS1["VFI double-conversion<br/>Li-ion (LFP), 10-15 years<br/>N+1 or 2N modular"]
|
||
UPS -->|"Edge / office"| UPS2["VI line-interactive<br/>VRLA, 3-5 years"]
|
||
POWER -->|"Generator"| GEN["Diesel 500-2500 kVA<br/>Tank for 24-72 h<br/>ATS 4-10 ms switching"]
|
||
POWER -->|"PDU"| PDU["3-phase 400 V<br/>Monitored/Switched<br/>A/B feed to racks"]
|
||
|
||
Start --> DENS{"Power density"}
|
||
DENS -->|"< 10 kW/rack"| COOL1["Air cooling<br/>CRAC/CRAH, raised floor<br/>Hot aisle containment<br/>ASHRAE A1-A2"]
|
||
DENS -->|"10-25 kW/rack"| COOL2["Hybrid<br/>In-row cooling<br/>Rear door HX<br/>ASHRAE A1-H1"]
|
||
DENS -->|"> 25 kW/rack"| COOL3["Liquid cooling<br/>CDU, direct-to-chip<br/>Immersion single/two-phase<br/>ASHRAE W-classes"]
|
||
|
||
Start --> CLIM{"Climate zone"}
|
||
CLIM -->|"Moderate (CZ, DE)"| FC1["Free cooling 4000-6000 h/year<br/>Chiller + economizer<br/>PUE saving 0.2-0.3"]
|
||
CLIM -->|"Warm (ES, US South)"| FC2["Chiller year-round<br/>Adiabatic cooling<br/>PUE 1.3-1.6"]
|
||
CLIM -->|"Cold (SE, NO)"| FC3["Free cooling 7000+ h/year<br/>Air-side economizer<br/>PUE < 1.2"]
|
||
```
|
||
|
||
## Secondary data center topologies
|
||
|
||
When planning a second DC, the choice of topology is key based on distance, RPO/RTO, and budget.
|
||
|
||
### Distance classification
|
||
|
||
| Category | Distance | Latency (round-trip) | Use case |
|
||
|-----------|-----------|---------------------|----------|
|
||
| **Metro (Campus)** | 1–20 km | < 1 ms | Synchronous replication, stretched cluster |
|
||
| **Metro** | 20–100 km | 1–5 ms | Metro cluster, mostly sync replication |
|
||
| **Regional** | 100–500 km | 5–20 ms | Asynchronous replication, warm standby |
|
||
| **Continent** | 500–3000 km | 20–100 ms | Asynchronous replication, cold standby |
|
||
| **Global** | 3000+ km | > 100 ms | Async only, no real-time dependencies |
|
||
|
||
### Topologies by operational mode
|
||
|
||
#### Active-Active (Hot-Hot)
|
||
|
||
```
|
||
DC-A (Primary) DC-B (Active)
|
||
┌────────────────────┐ ┌────────────────────┐
|
||
│ App Active │ │ App Active │
|
||
│ DB Active │◄─sync─►│ DB Active │
|
||
│ Users → LB → A │ │ Users → LB → B │
|
||
└────────────────────┘ └────────────────────┘
|
||
│ │
|
||
└──── Global Load Balancer ────┘
|
||
```
|
||
|
||
| Parameter | Value |
|
||
|----------|---------|
|
||
| **RTO** | 0–seconds (automatic failover, traffic is redirected) |
|
||
| **RPO** | 0 (sync replication, commit is confirmed only after write to both DCs) |
|
||
| **Max distance** | < 100 km (latency < 5 ms RTT for sync DB replication) |
|
||
| **Operating costs** | 2× (both DCs fully active, both fully equipped) |
|
||
| **Advantages** | Zero downtime, instant switchover, full utilization of both DCs |
|
||
| **Disadvantages** | Requires synchronous replication → distance limit, complex networking, split-brain risk |
|
||
|
||
**Split-brain solutions**: STONITH (Shoot The Other Node In The Head), watchdog, quorum (3rd node in 3rd location / cloud), fencing, SCSI-3 persistent reservation.
|
||
|
||
**Use case**: Financial services, telco, payment gateways — where even a minute of downtime = millions.
|
||
|
||
#### Active-Passive (Hot-Warm, MetroCluster)
|
||
|
||
```
|
||
DC-A (Primary) DC-B (Standby)
|
||
┌────────────────────┐ ┌────────────────────┐
|
||
│ App Active │ │ App Standby │
|
||
│ DB Primary │──sync──►│ DB Standby │
|
||
│ Users → LB → A │ │ ~~~ (waiting) ~~~ │
|
||
│ DNS: A-record │ │ DNS: health check │
|
||
└────────────────────┘ └────────────────────┘
|
||
```
|
||
|
||
| Parameter | Value |
|
||
|----------|---------|
|
||
| **RTO** | tens of seconds–minutes (DNS failover + App startup) |
|
||
| **RPO** | 0 (sync) or seconds (async) |
|
||
| **Max distance** | sync < 100 km, async unlimited |
|
||
| **Operating costs** | 1.5–1.8× (second DC has reduced or idle compute) |
|
||
| **MetroCluster** | Specific implementation: FC SAN over DWDM, sync mirror, automatic failover |
|
||
|
||
**MetroCluster** (NetApp, Dell EMC, HPE):
|
||
- Storage-based cluster with synchronous mirroring between DCs
|
||
- Automatic failover on entire DC failure
|
||
- Requires dedicated DWDM or dark fiber interconnection
|
||
- Typical distance: up to 50 km (for latency < 1 ms RTT)
|
||
- Use case: enterprise storage, primary+secondary DC in metropolitan area
|
||
|
||
#### Hot-Cold (Warm Standby → Cold)
|
||
|
||
```
|
||
DC-A (Primary) DC-B (Cold Standby)
|
||
┌────────────────────┐ ┌────────────────────┐
|
||
│ App Active │ │ ~~~ powered off ~~~│
|
||
│ DB Active │──async─►│ Backup storage │
|
||
│ Users → A │ │ ~~~ no compute ~~~│
|
||
└────────────────────┘ └────────────────────┘
|
||
```
|
||
|
||
| Parameter | Value |
|
||
|----------|---------|
|
||
| **RTO** | hours–days (purchase/rent HW, restore from backup) |
|
||
| **RPO** | hours (last backup) |
|
||
| **Max distance** | unlimited |
|
||
| **Operating costs** | 1.1–1.3× (only storage and facility, compute only at failover) |
|
||
| **Typical use case** | Low-cost DR, compliance, last resort |
|
||
|
||
#### Pilot Light
|
||
|
||
```
|
||
DC-A (Primary) DC-B (Pilot Light)
|
||
┌────────────────────┐ ┌────────────────────┐
|
||
│ App Active │ │ ~~~ off ~~~ │
|
||
│ DB Active │──async─►│ DB replica (mini) │
|
||
│ All services │ │ Core services only│
|
||
│ │ │ (DNS, LDAP, mon) │
|
||
└────────────────────┘ └────────────────────┘
|
||
On DR: spin-up compute
|
||
from IaC, rest from backup
|
||
```
|
||
|
||
- DC-B runs with minimum compute (only core services and DB replica)
|
||
- Application layer is spun up from IaC (Terraform, Ansible) only during DR
|
||
- Compromise between cost and RTO
|
||
|
||
### Comparison table
|
||
|
||
| Topology | RTO | RPO | Cost (× primary) | Max distance | Failover |
|
||
|-----------|-----|-----|-------------------|-------------|----------|
|
||
| **Active-Active** | 0–s | 0 | 2.0× | < 100 km | Auto (traffic) |
|
||
| **MetroCluster** | s–min | 0 | 1.8–2.0× | < 50 km | Auto (storage) |
|
||
| **Active-Passive (sync)** | min | 0 | 1.5–1.8× | < 100 km | Semi-auto |
|
||
| **Active-Passive (async)** | min–h | s–min | 1.3–1.5× | unlimited | Semi-auto |
|
||
| **Pilot Light** | h | min–h | 1.2–1.4× | unlimited | Manual |
|
||
| **Warm Standby** | min–h | s–min | 1.5–1.8× | unlimited | Semi-auto |
|
||
| **Cold Standby** | days | h | 1.1–1.3× | unlimited | Manual |
|
||
|
||
### Stretched Cluster
|
||
|
||
```
|
||
┌──── Site A (50 km) ────┐ ┌──── Site B ──────────┐
|
||
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
|
||
│ │ ESXi / Hyper-V │ │ │ │ ESXi / Hyper-V │ │
|
||
│ │ VM │ │ │ │ VM (complement) │ │
|
||
│ └────────┬─────────┘ │ │ └────────┬─────────┘ │
|
||
│ │ │ │ │ │
|
||
│ ┌────────▼─────────┐ │ │ ┌────────▼─────────┐ │
|
||
│ │ Storage (SAN) │──┼────┼──│ Storage (SAN) │ │
|
||
│ │ MetroCluster │ │ │ │ MetroCluster │ │
|
||
│ └──────────────────┘ │ │ └──────────────────┘ │
|
||
└────────────────────────┘ └────────────────────────┘
|
||
│
|
||
┌─────▼──────┐
|
||
│ vCenter / │
|
||
│ Cluster │
|
||
│ (single) │
|
||
└────────────┘
|
||
```
|
||
|
||
- One cluster stretched across two sites (single management domain)
|
||
- VMs can live-migrate between sites (vMotion over distance)
|
||
- Storage synchronously mirrored (MetroCluster, VPLEX, vSAN延伸)
|
||
- **Requirements**: dark fiber / DWDM, low latency (< 5 ms), high link reliability
|
||
- **Risks**: split-brain, brain drain (split-site cluster), network dependency
|
||
- **Use case**: enterprise with own dark fiber between two DCs in a metropolitan area
|
||
|
||
### Decision tree
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Start(["Secondary DC"]) --> RPO{"Required RPO?"}
|
||
RPO -->|"0 (no data loss)"| SYNC{"Sync replication possible?"}
|
||
SYNC -->|"Yes, < 100 km"| ACT{"Want zero downtime?"}
|
||
ACT -->|"Yes"| AA["Active-Active<br/>RTO=0, RPO=0, 2× cost"]
|
||
ACT -->|"No"| AP["Active-Passive<br/>RTO=min, RPO=0, 1.5×"]
|
||
SYNC -->|"No, > 100 km"| ASYNC["Active-Passive (async)<br/>RTO=min, RPO=s, 1.3×"]
|
||
|
||
RPO -->|"minutes–hours"| WARM{"Want fast failover?"}
|
||
WARM -->|"Yes"| PILOT["Pilot Light<br/>RTO=h, RPO=min, 1.2×"]
|
||
WARM -->|"No"| COLD["Cold Standby<br/>RTO=days, RPO=h, 1.1×"]
|
||
|
||
Start --> DIST{"Distance between DCs"}
|
||
DIST -->|"< 50 km, own fiber"| MC["MetroCluster / Stretched Cluster<br/>Single management, sync storage"]
|
||
DIST -->|"50–300 km"| REG["Regional DR<br/>Active-Passive, async replication"]
|
||
DIST -->|"> 300 km"| GLOBAL["Global DR<br/>Cold standby, backup & restore"]
|
||
```
|
||
|
||
### Physical infrastructure for DC interconnection
|
||
|
||
| Technology | Bandwidth | Max distance | Latency | Use case |
|
||
|------------|-----------|-------------|---------|----------|
|
||
| **Dark fiber** | 100 GbE–800 GbE | 10–80 km (single-mode) | < 0.1 ms | MetroCluster, stretched cluster |
|
||
| **DWDM** | 400 GbE–1.6 TbE (per lambda) | 80–120 km (without amplifier) | < 0.5 ms | Metro, metro cluster |
|
||
| **CWDM** | 10–25 GbE (per channel) | 10–40 km | < 0.3 ms | Campus, smaller metro |
|
||
| **MPLS L2VPN** | 10–100 GbE | unlimited | 1–10 ms | Regional DR, async replication |
|
||
| **Internet IPsec** | 1–10 GbE | unlimited | 5–50 ms | Cold standby, backup |
|
||
|
||
### Impact of individual technologies on DC topology selection
|
||
|
||
Choosing a secondary DC topology is not purely an infrastructure decision — each layer (DB, hypervisor, orchestration, messaging) brings its own constraints.
|
||
|
||
#### Databases
|
||
|
||
| DB technology | Sync replication | Max distance | Auto-failover | Split-brain handling | Note |
|
||
|---------------|---------------|-------------|---------------|-------------------|----------|
|
||
| **PostgreSQL** | Synchronous commit (synchronous_standby_names) | < 100 km (latency < 10 ms) | Patroni / repmgr + etcd | Quorum (etcd, 3+ node) | Streaming replication, needs wal_keep_segments |
|
||
| **MySQL** | Group Replication (multi-primary, single-primary) | < 100 km | MySQL InnoDB Cluster + MySQL Router | Paxos (Group Replication, 3+ node) | Semi-sync as compromise |
|
||
| **Oracle** | Data Guard (SYNC/FASTSYNC/ASYNC), RAC extended | sync < 100 km, async unlimited | Data Guard Broker / FSFO (Fast Start Failover) | Observer (3rd node) | Far Sync for remote DCs |
|
||
| **MSSQL** | AlwaysOn Availability Groups (SYNCHRONOUS_COMMIT) | < 100 km | AlwaysOn + Cluster quorum | File share majority / cloud witness | Multi-site cluster support |
|
||
| **MongoDB** | Majority write concern + journaling | < 100 km | Replica set auto-election | Arbitration node (voting member) | Priority-based failover |
|
||
| **Cassandra** | N/A (multi-master, eventual consistency) | unlimited | Yes (peer-to-peer) | None (multi-master, gossip protocol) | Snitch-aware topology, NetworkTopologyStrategy |
|
||
| **Redis** | Redis Sentinel / Redis Cluster (async) | unlimited (async) | Sentinel / Cluster failover | Quorum (Sentinel, majority) | PSYNC replication, replication lag |
|
||
|
||
Key limitation for **sync replication**: latency < 5 ms RTT (commit must wait for confirmation from both DCs). At 100 km RTT ~1 ms — OK. At 1000 km (~10 ms RTT) sync replication reduces transaction throughput by 80+ %.
|
||
|
||
Suitable for **Active-Active**:
|
||
- **Cassandra / ScyllaDB** — native multi-DC, eventual consistency, no split-brain
|
||
- **MySQL Group Replication (multi-primary)** — 3+ DC for quorum
|
||
- **CockroachDB / TiDB** — native multi-region, ACID across DCs
|
||
- **Redis Enterprise** — Active-Active (CRDT-based)
|
||
|
||
Suitable for **Active-Passive**:
|
||
- **PostgreSQL + Patroni** — auto-failover, etcd quorum
|
||
- **Oracle Data Guard** — FSFO, far sync for remote DCs
|
||
- **MSSQL AlwaysOn** — cloud witness
|
||
- **MongoDB Replica Set** — arbitration node in 3rd location
|
||
|
||
#### Hypervisors
|
||
|
||
| Hypervisor | Cluster technology | Stretched cluster | Max distance | Split-brain |
|
||
|-----------|-------------------|-------------------|-------------|-------------|
|
||
| **VMware vSphere** | vSAN延伸, Metro vCenter, Site Recovery Manager | Yes (vSAN延伸, Metro Cluster) | < 50 km (vSAN延伸), < 10 ms RTT | Fencing (STONITH), witness host |
|
||
| **Hyper-V** | Storage Replica + Failover Cluster | Yes (Cluster Sets) | < 50 km (sync), unlimited (async) | File share witness / cloud witness |
|
||
| **Proxmox VE** | Proxmox HA + Ceph | Limited (Ceph stretch cluster) | < 50 km (Ceph sync) | Ceph monitor quorum (3+ DC) |
|
||
| **XCP-ng / XenServer** | Xen Orchestra HA + SR (Storage Repository) replication | Limited | depends on storage replication | — |
|
||
| **Nutanix AHV** | Metro Availability (sync), Async DR | Yes (Metro) | < 100 km (sync), unlimited (async) | Witness VM (cloud / 3rd site) |
|
||
| **KVM / oVirt** | oVirt HA + GlusterFS / NFS | Limited | depends on storage replication | — |
|
||
|
||
**vSAN延伸 specific requirements:**
|
||
- Dedicated vSAN network (25 GbE min., < 5 ms RTT)
|
||
- Witness host in 3rd location (or cloud witness)
|
||
- All VM policies (FTT=1, mirroring striped)
|
||
- Storage policy: `site-A + site-B + witness`
|
||
|
||
#### Kubernetes and container platforms
|
||
|
||
| Platform | Multi-cluster DR | Replication | Max distance | Failover |
|
||
|-----------|-----------------|-----------|-------------|----------|
|
||
| **Vanilla K8s** | KubeFed, Cluster API, Velero + Restic | Velero (backup/restore), Rook (Ceph) | unlimited | Manual (Velero restore) |
|
||
| **OpenShift** | ACM (Advanced Cluster Management), Velero | OADP (OpenShift API for Data Protection) | unlimited | ACM failover (subscription) |
|
||
| **Rancher** | Rancher Multi-Cluster App, Velero | Longhorn (sync/async DR), Velero | unlimited | Semi-auto |
|
||
| **Google GKE** | Multi-cluster Services, Backup for GKE | Config Sync, Backup for GKE | unlimited | Manual |
|
||
| **Azure AKS** | Azure ARC + Velero + Azure Traffic Manager | AKS backup (velero), Azure Site Recovery | unlimited | Manual (Velero) |
|
||
| **AWS EKS** | EKS multi-cluster, Velero + S3 cross-region | Velero (S3), Rook (EBS snapshots) | unlimited | Manual |
|
||
|
||
**Key K8s DR principles:**
|
||
- **Applications must be stateless** (or state externalized to DB/storage)
|
||
- **Velero** — backup/restore entire cluster (PV, resources, helm releases)
|
||
- **Rook/Ceph** — cross-region mirroring RBD volumes
|
||
- **KubeFed / ACM** — subscription-based deploy to multiple clusters
|
||
- **Ingress/Gateway API** — traffic routing between clusters
|
||
- **External DNS** — DNS failover on cluster outage
|
||
|
||
#### Messaging / streaming
|
||
|
||
| Platform | Replication | Topology | DR support | Max distance |
|
||
|-----------|-----------|-----------|------------|-------------|
|
||
| **Apache Kafka** | MirrorMaker 2, Confluent Cluster Linking, KRaft quorum | Active-Passive (MM2), Active-Active (Cluster Linking) | MM2: async, Cluster Linking: async | unlimited |
|
||
| **RabbitMQ** | Classic Queue Mirroring, Quorum Queues | Active-Passive (Warm Standby) | Federation / Shovel (async) | unlimited |
|
||
| **Red Hat AMQ** | (Artemis) Cluster + HA | Active-Passive (shared store / replication) | Live-backup pair | < 100 km (sync) |
|
||
| **NATS** | NATS JetStream (cluster + cross-account) | Active-Active (Leaf nodes, cross-account) | Super-cluster, failover | unlimited |
|
||
| **Apache Pulsar** | BookKeeper (bookie rack-aware), geo-replication | Active-Active (geo-replication) | Built-in (cluster-level) | unlimited (async) |
|
||
| **AWS SQS/SNS** | Managed, AWS region pairs | Active-Active (multi-region) | Built-in (AWS managed) | unlimited |
|
||
| **Azure Service Bus** | Managed, paired region | Active-Passive (paired region) | Built-in (geo-recovery) | unlimited |
|
||
| **Oracle Service Bus (OSB)** | Oracle WebLogic Cluster + JDBC store + AQ | Active-Passive (WebLogic Cluster + Data Guard) | OSB/WLS cluster + Oracle RAC/Data Guard sync | < 100 km (Data Guard sync), unlimited (async) |
|
||
|
||
**Messaging DR recommendations:**
|
||
- **Kafka**: use Cluster Linking for Active-Active, or MirrorMaker 2 for Active-Passive; replicate only critical topics
|
||
- **RabbitMQ**: Quorum Queues + Federation upstream for DR; avoid Classic Queue Mirroring (deprecated)
|
||
- **Pulsar**: native geo-replication, bookie rack-aware for stretched cluster; easiest DR among messaging platforms
|
||
- **OSB**: WebLogic cluster + Oracle RAC/Data Guard; DR depends on DB layer, not on OSB itself
|
||
|
||
### Per-layer limitations summary table
|
||
|
||
| Layer | Limiting factor for secondary DC | Max distance for sync | Impact on topology selection |
|
||
|--------|-----------------------------------|----------------------|--------------------------|
|
||
| **Storage** | Sync mirror latency, DWDM cost | < 50 km (MetroCluster) | Stretched cluster only in metro |
|
||
| **Databases** | Commit wait for sync replication | < 100 km (5 ms RTT) | Active-Active only with multi-master DB |
|
||
| **Hypervisor** | Stretched cluster quorum + fencing | < 50 km (vSAN, 5 ms) | MetroCluster / stretched cluster |
|
||
| **Kubernetes** | Velero restore time, Rook mirror latency | unlimited (async) | Active-Passive, cold standby |
|
||
| **Messaging** | Replication lag, offset management | unlimited (async) | Active-Active (Kafka, Pulsar, NATS) or Active-Passive |
|
||
| **Network** | Dark fiber/DWDM cost, latency | < 100 km (metro fiber) | Limits sync replication options |
|
||
| **Application** | Stateful/stateless, connection draining | depends on architecture | Stateless app → any topology |
|
||
|
||
## Disk monitoring — S.M.A.R.T.
|
||
|
||
Self-Monitoring, Analysis and Reporting Technology — predictive monitoring of HDD/SSD.
|
||
|
||
| Key attribute | ID | Description |
|
||
|--------------|----|-------------|
|
||
| Reallocated Sectors Count | 5 | Number of remapped sectors (increase = end of disk life) |
|
||
| Power-On Hours | 9 | Total operating time in hours |
|
||
| Reported Uncorrectable Errors | 187 | Uncorrectable errors (red flag) |
|
||
| CRC Error Count | 199 | Errors on SATA link (cable/controller) |
|
||
| SSD Life Left | 231 | % remaining SSD life |
|
||
| Media Wearout Indicator | 233 | Total NAND writes |
|
||
|
||
Tools: `smartmontools` (smartctl, smartd), Prometheus exporter (`node_exporter`), OTeL collector.
|
||
|
||
## Sources
|
||
|
||
Links, books and standards: [sources/infrastructure/sources.en.md](sources/infrastructure/sources.en.md)
|
||
|
||
### Recommended literature
|
||
|
||
| Book | Authors | ISBN | Description |
|
||
|------|---------|------|-------------|
|
||
| The Data Center as a Computer (4th ed., 2025) | Barroso, Hölzle, Ranganathan | 978-3-031-99488-3 | Comprehensive design evolution of warehouse-scale computer (WSC) by Google architects. Covers hardware, software, power, cooling, networking and 25 years of WSC experience. Key publication for datacenter architecture. |
|
||
| Electronics Cooling: From the Chip to the Datacenter (Vol. 62) | Abraham et al. | 978-0-443-47084-4 | Practical guide to thermal management from transistor level to datacenter. Covers conduction, convection, liquid immersion and phase change cooling. Essential resource for DC cooling design. |
|
||
|
||
## Datacenter backbone services
|
||
|
||
When building a new DC, basic infrastructure services must be deployed first — without them, higher layers cannot operate:
|
||
|
||
### DNS
|
||
|
||
| Role | Service | Description |
|
||
|------|---------|-------------|
|
||
| **Authoritative** | Bind, PowerDNS, NSD | Primary DNS zone for internal domains |
|
||
| **Recursive** | Unbound, Bind (caching), CoreDNS | Resolver for internal + external queries |
|
||
| **Anycast** | DNS anycast (BGP) | Redundancy, lower latency |
|
||
| **Integration** | Infoblox, BlueCat, dnsmasq | IPAM + DNS + DHCP in one |
|
||
|
||
Best practices: separate auth and recursive resolvers, DNSSEC, split-horizon (internal vs external view), TSIG for zone transfers, monitoring (DNS query latency, NXDOMAIN rate).
|
||
|
||
### NTP (time synchronization)
|
||
|
||
- **Primary**: GPS-disciplined NTP servers (Microchip S600, Meinberg)
|
||
- **Secondary**: Stratum 1/2 NTP (ntpd, chrony, NTPsec)
|
||
- **All nodes**: chrony (modern replacement for ntpd), local NTP server on each rack switch (boundary clock)
|
||
- **Precision**: PTP (IEEE 1588) for telco/fintech — sub-microsecond accuracy
|
||
- **DC topology**: GPS antenna → Grandmaster (PTP) → Boundary clock (rack switch) → Ordinary clock (server)
|
||
|
||
### DHCP + IPAM
|
||
|
||
| Tool | Description |
|
||
|------|-------------|
|
||
| **ISC DHCP** | Legacy, still widely deployed |
|
||
| **Kea** | Modern replacement for ISC DHCP (ISC + Linux Foundation) |
|
||
| **Infoblox / BlueCat** | Enterprise IPAM + DHCP + DNS |
|
||
| **NetBox / phpIPAM** | Open-source IPAM |
|
||
|
||
### LDAP / Identity Management
|
||
|
||
| Tool | Description |
|
||
|------|-------------|
|
||
| **FreeIPA** | Integrated IDM (LDAP + Kerberos + DNS + CA) — Linux |
|
||
| **Active Directory** | Microsoft, LDAP + Kerberos + Group Policy |
|
||
| **389 Directory Server** | Open-source LDAP (Red Hat) |
|
||
| **OpenLDAP** | Classic open-source LDAP |
|
||
| **Keycloak / Authentik** | Modern OIDC/SAML/LDAP gateways |
|
||
|
||
### PKI and certificates
|
||
|
||
- **Enterprise CA**: EJBCA, Smallstep, HashiCorp Vault (PKI engine)
|
||
- **ACME**: Cert-Manager (Kubernetes), certbot (Let's Encrypt)
|
||
- **mTLS**: Vault PKI, spire (SPIFFE), Cilium
|
||
- **Best practices**: root CA offline, intermediate CA per environment, short-lived certificates (max 90 days), revocation (CRL/OCSP)
|
||
|
||
### Monitoring and observability
|
||
|
||
See [MONITORING.en.md](MONITORING.en.md). Before running first workloads, DC must have:
|
||
- Metric collection (Prometheus, Zabbix)
|
||
- Centralized logs (Loki, ELK)
|
||
- Alerting (Alertmanager, PagerDuty)
|
||
- Uptime monitoring (heartbeat checks)
|
||
|
||
### Deployment logistics — step order
|
||
|
||
```
|
||
1. DNS (at least recursive + local resolver)
|
||
2. NTP (time synchronization)
|
||
3. DHCP + IPAM (first servers get IPs)
|
||
4. LDAP / IAM (users, groups, access rights)
|
||
5. PKI (certificates for encryption)
|
||
6. Configuration management (Ansible, Puppet)
|
||
7. Monitoring + logging (see what's happening)
|
||
8. Container registry / Package repo (docker registry, apt/yum mirror)
|
||
9. Load balancer (for services)
|
||
10. Storage backend (Ceph, NFS, SAN)
|
||
11. Orchestration (Kubernetes, OpenStack)
|
||
```
|
||
|
||
## OpenStack in the datacenter
|
||
|
||
OpenStack brings a software abstraction layer to DC enabling multi-tenancy and self-service:
|
||
|
||
### Control plane architecture
|
||
|
||
- **Controller nodes** — management services (Keystone, Nova API, Neutron API, Horizon, RabbitMQ, DB)
|
||
- **Compute nodes** — hypervisor (KVM), Nova Compute, Neutron agent
|
||
- **Storage nodes** — Ceph OSD, Cinder volumes, Swift object storage
|
||
- **Network nodes** — Neutron L3 router, DHCP agent, DVR
|
||
|
||
### Requirements for DC infrastructure
|
||
|
||
| Component | Requirement |
|
||
|-----------|-------------|
|
||
| **Controller** | 3-5 node HA cluster, 16+ vCPU, 32+ GB RAM, SSD |
|
||
| **Compute** | Dense performance per rack (GPU, high-core), NUMA-aware design |
|
||
| **Storage (Ceph)** | 10-25 GbE networking, NVMe/SSD OSD, 3+ replica |
|
||
| **Network** | 25/100 GbE spine-leaf, L3 BGP underlay, VXLAN overlay |
|
||
| **Rack power** | 10-30 kW/rack for GPU compute |
|
||
|
||
### Use cases
|
||
|
||
- Private cloud for enterprise (multi-tenant, self-service Horizon)
|
||
- NFVI for telco (DPDK, SR-IOV, low-latency)
|
||
- Academic / HPC clusters (Ironic, Cyborg, Manila)
|
||
- Government / regulated environments (on-prem, audit trail)
|
||
|
||
*Last revision: 2026-06-03*
|