GEFION

Danish Centre for AI innovation AS · DDN1 · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The Danish Centre for AI Innovation (DCAI) was established to accelerate artificial intelligence research and innovation across various organizations and industries. A key component of DCAI is Gefion, Denmark’s first AI supercomputer, which provides researchers and enterprises with access to advanced computing resources optimized for large-scale AI projects.

DCAI fosters collaboration between academia and industry, providing access to NVIDIA’s software platforms, training, and expertise. This collaborative environment aims to drive innovation and develop solutions to some of the world’s most pressing challenges.

In summary, DCAI’s systems, particularly the Gefion supercomputer, are designed to empower researchers and organizations in Denmark to leverage advanced AI capabilities, promoting innovation and addressing critical societal issues.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The system is on its final phase of deployment. The goal is to be ready for utilization in November/December 2024

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

Two file systems have been deployed on the DCAI. The larger file system has undergone benchmarking, while the smaller one has been deployed but is not suitable for benchmarking.

This storage solution utilizes the DDN NV400X2, a fully NVMe appliance hosting four virtual machines. The benchmarked file system comprises 12 appliances, with two appliances (eight virtual machines) dedicated to MDS, and ten appliances (forty virtual machines) for additional tasks. The NV400X2 is built on DDN Exascaler, a commercial solution embedded within the appliance and not available for separate download or purchase.

https://www.ddn.com/products/ai400x2-turbo/

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

For the 10 node challenge, a total of 128 NVIDIA DGX H100 was used (96 MPI ranks per node).

io500.sh wrapper has been changed as below.

io500_mpiargs="--bind-to core:overload-allowed"

The tunings utilized are:

"osc.*.max_dirty_mb" = "2000"
"osc.*.checksums" = "0"
"osc.*.short_io_bytes" = "65536"
"osc.*.max_pages_per_rpc" = "16M"
"ldlm.namespaces.*.lru_size" = "0"
"ldlm.namespaces.*.lru_max_age" = "1000"
"mdc.*.max_rpcs_in_flight" = "128"
"llite.*.max_cached_mb" = "65536"
"obdfilter.*.brw_size" = "16"

DNE3 is by default enable, so automatically space balancing accross all MDTs.

Stripe settings below were integrated into the io500.sh wrapper.

mkdir /lustre/gefion1/ddn/io500/datafiles/io500/mdtest-easy -p
  mkdir /lustre/gefion1/ddn/io500/datafiles/io500/mdtest-hard -p
  mkdir /lustre/gefion1/ddn/io500/datafiles/io500/ior-hard -p

ssh root@x.x.x.x "lfs setstripe -C 1280 /lustre/gefion1/client/ddn/io500/datafiles/io500/ior-hard"
  ssh root@x.x.x.x "lfs setdirstripe -D -c -1 /lustre/gefion1/client/ddn/io500/datafiles/io500/mdtest-hard"
  ssh root@x.x.x.x "lfs setdirstripe -D --max-inherit-rr 2 /lustre/gefion1/client/ddn/io500/datafiles/io500/mdtest-easy"

** NOTES **
For the form submitted, the specs under MDS and OSS are "Per Virtual Machine". Each DDN NV400X2 appliance has four embedded virtual machines that can be used for either MDS, OSS or hybrid function. In case of DCAI the EXAscaler environment has 2 dedicated NV400X2 for Metadata only, thus 8 (eight) VMs dedicated to Metadata Servers, and 10 (Ten) NV400X2 for OSS, totalizing 40 Virtual Machines for OSS. In total the system has 48 Virtual Machines.

In each description (MDS and OSS) we added the Storage Media and Interconnect details. We consider that question refers to the configuration of each server (in our case, each VM). 

In the case of MDS, we have a total of 12 NVME devices on each appliance. These 12 devices are divided into two groups of 6 (pool). And each Pool is shared accross 2 Virtual Machines. So, in the form we specified 6 NVME devices per VM, but one thing to remember is the fact that we are sharing these 6 devices into two Virtual Disks. 

Same case for OSSs. Each OSS appliance has 24 NVME devices split into 2 pools of 12. Each pool is split into 4 Virtual Disks, and each 2 VDs are assigned primarily to one VM. So, in the form we filled up 12 disks per VM, so total of 24 NVMEs per appliance, being  each 12 shared across 2 VMs. 

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

No single point of Failures on this system.

Redundant power supplies N+N

Redundant fans and internal components N+1

Dual Controllers hosting 2 Virtual Machines each. In case of one controller dies, the other VMs will take over the storage resources that are shared accross all the VMs. THe system can still running even if 3 VMs are off since a single VM are able to take over the storage resources and still serving data (with performance reduction).

Each appliance has 24 NVMe slots that are dual initialized. All the devices are simultaneously available to the two controllers and its respective Virtual Machines.

The storage controllers utilized Declustered Parity Raid for data protection using RAID6 schemas. All MDTs and OSTs are RAID6 identically configured, varying only the defined capacity for each type of target (Virtual Disks), that are presented individually to the VMs. 

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

IO500 was executed using 128 nodes directly (No containers).

SLURM batch jobs was submitted invoking the resources needed (128 nodes) and 96 processors per node. Each DGX mounts lustre filesystem in 1 mount-point 1 per  node where 96 MPI threads are executed per client node.

During the IO500 the entire cluster was dedicated to the benchmark and no other application or user was submitting any job.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

No client side cached used. Lustre provides client cache mechanism that utilizes RAM and not persistent storage.
The option of utilization of PCC (Persistent Client Caching) was disabled, thus not relying on any storage media. 

Onoly client data/metadata caching (linux page cache) has been used since it's integral part of Lustre file system.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The submitted system is a standalone system. EXAScaler is the only filesystem configured, with no burstbuffer or any other intermediate acceleration layer. 

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

Benchmarks were executed several times until a suitable result was achieved.

The score is validated using io500-verify.sh script. Output below

$ ./io500-verify ./results/2024.11.10-15.38.15/config-ddn-large.ini ./results/2024.11.10-15.38.15/result.txt
version         = io500-sc24_v1
result file ver = io500-sc24_v1
[run]
config-hash     = F19B4C17
score-hash      = 2B1CC036

[OK]