Sākums Izpētīt Palīdzība
Reģistrēties Pierakstīties
jeremy
/
HON95-wiki
spogulis no https://github.com/HON95/wiki
1
0
Atdalīts 0
Faili Problēmas 0 Vikivietne
Koks: 5432d54493
Atzari Tagi
HON95-wiki
/
hpc
/
betzy.md

betzy.md 5.5 KB
Vēsture Neapstrādāts

title: Betzy (Supercomputer) breadcrumbs:

  • title: High-Performance Computing (HPC) header_image: ../files/betzy.jpg --- {% include header.md %}

Norway's most powerful supercomputer as of its inauguration in late 2020. Placed #55 on the 2020 June TOP500. Managed by UNINETT Sigma2 and partially NTNU Trondheim.

Resources

History

  • 2018: Procurement started. [13]
  • June 2020: Placed #55 on the TOP500 list. [12]
  • 7 December 2020: Inauguration. [11]
  • April 2021: Four new X2415 blades (A100) and 5.3PB more storage (from 2.5PB to 7.8PB). [10]

Specifications

A mix of general XH2000 specifications and specific Betzy specifications.

  • Betzy overall specifications [1][2][5][9][10]:
    • System: Atos BullSequana XH2000 with X2410 (AMD EPYC) and X2415 (NVIDIA A100) blades.
    • OS: RHEL
    • Compute nodes: 1344 X2410 (AMD EPYC) (448 blades) + 4 X2415 (NVIDIA A100)
    • CPUs total (excluding A100 nodes): 2688 CPUs, 172032 cores
    • Memory: 336TiB total (excluding A100 nodes)
    • Storage: 7.8PB (2.5PB before 2021 upgrade), DNN powered, Lustre, 51GB/s bandwidth, 500k+ metadata OPS (before 2021 upgrade)
    • Interconnect topology: DragonFly+ topology
    • Queueing system: Slurm
    • Footprint: 14.78m2 (before 2021 upgrade)
    • Power: 952kW, 95% of heat captured to water (before 2021 upgrade)
    • Cooling: Liquid cooled
  • CPU node specifications (CPU nodes) [1][2][4]:
    • AMD EPYC Rome 7742
      • 64 cores, 128 threads (per CPU)
      • Clock: 2.25GHz base, 3.4GHz max boost
      • Memory: DDR4, 8 channels, 3200MHz, 204.8GB/s per-socket BW
      • PCIe: 4.0, x128
    • Memory: 256GiB, split into 8 NUMA nodes
    • Storage: 3x SATA or NVMe M.2 drives
    • NIC: InfiniBand HDR 100
  • CPU node specifications (GPU nodes) [0][14]:
    • CPUs: 2x AMD EPYC Rome 7452
      • 32 cores, 64 threads (per CPU)
      • Clock: 2.35GHz base, 3.35GHz max boost
      • Memory: DDR4, 8 channels, 3200MHz, 204.8GB/s per-socket BW
      • PCIe: 4.0, x128
  • Blade specifications [9]:
    • Betzy uses mainly X2410 blades (AMD), but also 4x X2415 blades (A100) (after the 2021 upgrade) [10].
    • Size: 1U
    • Cooling: Fanless, active liquid cooling.
    • All blades types (both used and not used):
      • X2410: 3x nodes (side-by-side) with 2x AMD EPYC Rome/Milan CPUs (6 CPUs total).
      • X2415: 1x node with 2x AMD EPYC Rome/Milan CPUs and 4x Nvidia A100 SXM4 GPUs.
      • X1120: 3x nodes (side-by-side) with 2x Intel Xeon CPUs (6 CPUs total).
      • X1125: 1x node with 2x Intel CPUs and 4x Nvidia V100 SXM2 GPUs.
  • Cabinet specifications (general, not Betzy specific) [8][9]:
    • Number of blades: 4-20 in front, 4-12 in back
    • Management switches:
      • Up to 2.
      • Up to 48 1Gb/s or 10Gb/s ports.
    • Interconnect switches:
    • Up to 10.
    • Infiniband HDR100: 80 ports, 100Gb/s (Betzy)
    • Alternative technologies:
      • Bull eXascale Interconnect (BXI): 48 ports, 100Gb/s
      • High-speed Ethernet: Up to 48 ports, up to 100Gb/s
    • Supported interconnect topologies:
      • DragonFly+
      • Full Fat Tree
    • PSU: 6x 15kW shelves
    • Power input: 3x 63A 3-phase 400V (for EU)
    • Cooling:
      • Direct Liquid Cooling (DLC)
      • Hydraulic chassis (HYC)
      • Primary (external) loop connected to customer water loop.
      • Secondary (internal) loop connected to blades, management switches, interconnect switches and PSUs.

References

{% include footer.md %}