GB300 pod setup (hypothetical, hardware agnostic) – Total TDP ≈ 2.42mW / Max power ≈ 2.78mW

Here’s a max‑GB300, 18‑rack airtime network that preserves a proper leaf–spine fabric, shared storage, and OOB/timing, while squeezing in the most GB300 GPUs possible.

  • Total racks: 18
  • GB300 accelerator racks: 16 (R3–R18)
  • Support racks:
    • R1 = Fabric + Edge + OOB + Timing (collapsed into one rack)
    • R2 = Shared Storage (Ceph + NVMe‑oF)
  • Accelerator density: 8× 4U servers per GB300 rack, each with 8× GB300 → 64 GB300/rack
  • Total GB300 GPUs: 16 × 64 = 1,024

Fabric & overlay (brief)

  • Leaf/ToR: Dual PTP‑aware 400 GbE ToRs per rack (MLAG/vPC), plus 1× OOB switch.
  • Spine: 4× 64×400 GbE non‑blocking (in R1).
  • Overlay: EVPN‑VXLAN over iBGP underlay (ECMP).
  • Timing: GNSS‑backed PTP grandmasters (R1) → boundary clocks (R1) → PTP‑aware ToRs → NIC HW timestamp on DUs/AI servers.
  • Storage: NVMe‑oF over RoCE; Ceph for block/object.

Power breakdown (assumptions shown)

Relevant assumptions:

  • GB300 accelerator rack (R3–R18): 150 kW DC typical per rack (includes servers + ToRs + OOB).
  • Max draw estimate: 1.15× typical (common industry planning factor).
  • PSU+distribution efficiency for AC @ wall: 92%.
  • Recommended breaker headroom: +10%.

Per‑rack power (DC → AC → breaker)

  • GB300 rack (each R3–R18):
    • DC typical: 150.0 kW → AC typical: 163.0 kW → Breaker: 179.3 kW
    • DC max est.: 172.5 kW → AC max: 187.5 kW → Breaker: 206.3 kW
  • R1 (Fabric + Edge + OOB + Timing): (itemized typical DC)
    • 4× spines ≈ 4.0 kW, 2× edge routers ≈ 4.0 kW,
       2× PTP BC ≈ 0.16 kW, RR 0.15 kW, L4/L7 0.30 kW,
       packet broker 0.30 kW, netmgmt agg 0.15 kW,
       2× OOB cores 0.30 kW, 2× consoles 0.14 kW,
       2× PTP GMs 0.18 kW, GNSS 0.03 kW, NTP 0.05 kW, KVM 0.06 kW
    • DC typical: 9.8 kW → AC typical: 10.7 kW → Breaker: 11.8 kW
    • DC max est.: 11.3 kW → AC max: 12.3 kW → Breaker: 13.5 kW
  • R2 (Storage): (typical DC)
    • 3× Ceph nodes 2.1 kW, 2× NVMe‑oF targets 1.4 kW, 2× JBOD 1.2 kW,
       2× ToR 1.8 kW, OOB 0.10 kW
    • DC typical: 6.6 kW → AC typical: 7.2 kW → Breaker: 7.9 kW
    • DC max est.: 7.6 kW → AC max: 8.3 kW → Breaker: 9.1 kW

Site totals (all 18 racks)

  • DC typical: 2,416.4 kW
    • (16× GB300 racks = 2,400.0 kW + R1 9.8 kW + R2 6.6 kW)
  • DC max (est.): 2,778.9 kW
    • (16× 172.5 kW + R1 11.3 kW + R2 7.6 kW)
  • AC @ wall (92% eff):
    • Typical: 2,626.5 kW
    • Max (est.): 3,020.5 kW
  • Recommended breaker capacity (+10%):
    • Typical: 2,889.2 kW
    • Max (est.): 3,322.6 kW