Helma

Erlangen National High Performance Computing Center · MEGWARE hnvme · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The cluster and storage system are hosted at the Erlangen National High Performance Computing Center (NHR@FAU). The storage system, known as hnvme, is an integral component of the Helma cluster, which is primarily dedicated to cutting-edge AI research. In addition to supporting artificial intelligence workloads, the cluster also provides the high computational performance required for a wide range of demanding simulation projects across diverse scientific disciplines.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The system was deployed in April 2025 and will remain available to all Helma users for a minimum of five years. In addition to serving the Helma cluster directly, the storage system is also remotely mounted on other NHR@FAU clusters, enabling broader accessibility and integration across the computing infrastructure.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The software stack includes both open-source and proprietary components:

All software components are publicly available for download or purchase as applicable.

The storage cluster comprises 10 Celestica SC6100 HA All-Flash storage controllers. Each controller includes:

  • 2x AMD EPYC Genoa single-socket servers
  • 24x dual-port NVMe drives (PCIe Gen5)
  • 4× InfiniBand NDR400 links
  • 2x Ethernet 100Gb

To maximize performance, each NVMe drive is split into two namespaces. Each server exports:

  • 1x MDT (Metadata Target), configured as xiRAID10 across 4 disks
  • 2x OSTs (Object Storage Targets), each configured as xiRAID6 across 10 disks

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

C/P-States were disabled in BIOS.
OS, xiRAID, Lustre and Pacemaker/Corosync were installed and configured on all servers.

xiRAID6 on OSTs used 128k strip size. Each xiRAID was pinned to a dedicated subset of cores.
Ldiskfs on OSTs was created with: -E stride=32,stripe_width=256

Afterwards, the following Lustre tunings were applied:

Serverside (persistent) Lustre settings

lctl set_param -P obdfilter.*.brw_size=16
lctl set_param -P obdfilter.*.precreate_batch=1024
lctl set_param -P obdfilter.*.osc.max_pages_per_rpc=4096
lctl set_param -P osc.*.max_pages_per_rpc=4096
lctl set_param -P mdc.*.max_rpcs_in_flight=128
lctl set_param -P osc.*.max_rpcs_in_flight=128
lctl set_param -P mdc.*.max_mod_rpcs_in_flight=127
lctl set_param -P llite.*.checksum_pages=0
lctl set_param -P llite.*.checksums=0
lctl set_param -P osc.*.checksums=0
lctl set_param -P mdc.*.checksums=0
lctl set_param -P mdt.*.max_mod_rpcs_in_flight=128

Clientside Lustre settings

lctl set_param ldlm.namespaces.*.lru_size=12800
lctl set_param mdc.*.max_rpcs_in_flight=128
lctl set_param mdc.*.max_mod_rpcs_in_flight=127
lctl set_param osc.*.max_rpcs_in_flight=128
lctl set_param llite.*.max_read_ahead_mb=2048
lctl set_param osc.*.short_io_bytes 65536
lctl set_param ldlm.namespaces.*.lru_max_age=1000

Lnet credits were slightly increased:

peer_credits: 16
credits: 512
peercredits_hiw: 8

For Lustre striping etc. check the submitted io500.sh.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Power
The system has power level redundancy (redundant PSUs & USV PDUs).

Networking

Each storage server has 2 InfiniBand HCAs for performance and redundancy. Each storage server is connected to two different InfiniBand switches.

Inter storage data and metadata server
There are 10 controllers with 24 dual-port NVMe drives. Each controller has two storage servers accessing all 24 NVMe drives. The NVMe drives are splitted into two namespaces for maximum performance. Both servers are configured in a active-active Pacemaker/Corosync setup. If one server in a controller fails, the MDT and OST devices fail over to the other server.

Intra storage data and metadata server storage media
For storage redundancy Xinnor xiRAID Classic was used (4 disk RAID10 for MDTs and 10 disk RAID6 for OSTs).

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The IO500 benchmark was executed using OpenMPI 5.0.5 across 186 GPU nodes, with 100 MPI processes per node. The benchmark was launched directly via mpirun, without the use of a workload manager or job scheduler. To ensure clean and consistent results, all files were deleted between runs, and no additional setup or tuning steps (besides the described ones) were performed prior to execution. The system was fully dedicated to running the benchmark during the testing period, ensuring exclusive access to all resources.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

Besides native Lustre client cache no additional cache was used.
All ior-easy tests use direct I/O to further prevent any caching.

Lustre server side caching was disabled (by default because flash based storage is detected):

lctl set_param osd-ldiskfs.*.read_cache_enable=0
lctl set_param osd-ldiskfs.*.writethrough_cache_enable=0

Xinnor xiRAID Classic itself does not cache.
Test data size was way above available server and client memory.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The system functions as a stand-alone storage solution and serves as the primary parallel file system for the Helma cluster. Additionally, it is exported to other clusters within NHR@FAU via 100 Gb Ethernet.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was executed multiple times, consistently producing similar results across runs.
The results are below the physical capabilites of the deployed hardware.
For aligned workloads, direct I/O was employed to bypass the page cache, ensuring that data was written directly to persistent storage. No additional caching mechanisms were used. Moreover, the test data size significantly exceeded the combined memory capacity of both servers and clients, eliminating the possibility of results being influenced by memory caching.