IO500 - MN5 GPP - Reproducibility

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

MareNostrum 5 is a pre-exascale EuroHPC supercomputer hosted at BSC-CNS. The system is supplied by Bull SAS combining Bull Sequana XH3000 and Lenovo ThinkSystem architectures and it has a total peak computational power of 314PFlops. The system will provide 4 partitions with different technical characteristics that jointly can fulfil the requirements of any HPC user.

The MareNostrum 5 partitions are divided as follows :

General Purpose partition based on Intel Sapphire Rapids, 6480 nodes, including 72 HBM nodes, 45,9 PFlops peak

Accelerated Partition based on Intel Sapphire Rapids and Nvidia Hopper GPUs, 1120 nodes with 4 Hopper GPUs each one, 260 PFlops peak

General Purpose - Next Generation Partition, , based on Nvidia GRACE CPU

Accelerated - Next Generation Partition , not fully defined. more information in the following months.

System Overview

The 2 main partitions of MareNostrum 5 have these technical characteristics :

MareNostrum 5 GPP (General Purpose Partition)

The machine has 6408 nodes based in Intel Sapphire rapids. Each is configures as:

2x Intel Sapphire Rapids 8480+ at 2Ghz and 56c each (112 cores per node)

256 GB of Main memory, using DDR5 (with 216 nodes with 1024GB)

960GB on NVMe storage (/scratch)

1x NDR200 shared by 2 nodes (SharedIO) (BW per node 100Gb/s)

In addition to the 6408 standard nodes, the machine has 72 HBM nodes based in Intel Sapphire Rapids 03H-LC with 112 cores per node at 1.9Ghz and 128GB HBM memory. This small sub-system will provide a high memory BW of 2TB/s per node.

The full machine provides a Peak Performance of 45.9 PFlops.

The network topology used is fat-tree , with islands with full fat tree without contention of 2160 nodes, and with a contention between islands of 2/3.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

Marenostrum 5 is available for users throughout Europe, via standard eurohpc request

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The GekkoFS burst buffer file system is available to any user. It can aggregate the local I/O capacity and performance of each compute node in an HPC cluster to produce a high-performance storage space that can be accessed in a distributed manner.

This storage space allows HPC applications and simulations to run in isolation from each other with regards to I/O, which reduces interferences and improves performance.

GekkoFS can be used by non-privileged users within their compute job without assistance from administrators accelerating their I/O.

GekkoFS is fully open-source with the latest release v0.9.6, using as tag isc26

GekkoFS Git Repository

For this test, we installed it widely in MN5 GPP and ACC partitions, so users can use it with module load GekkoFS/master-0.9.6

Specific optimizations are now enabled via environment variables so they are visible in the job submission file.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

Commands used to set striping information (either for the entire system or for particular directories)
File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

GekkoFS was using 125 server nodes and 10 client nodes (overlapping). Each client and node has one NIC available with a total of 10 client endpoints. Each server node offers 1 NVMe SSD. The necessary software dependencies were compiled beforehand by a non-privileged user. The code used is from the master without modifications.

Clients and daemons were not pinned, and they were executed in a standard way in Release mode without any kind of log, using the normal Gekkofs/master-0.9.6 module file.

The only particularity, is that we ask sbatch to minimize the number of switches between nodes (minimum is 2)

Syscall Intercept backend was used, and the optimizations shown in the job submission file activated.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

Power (For example, can power failure of a node or rack bring down the storage system?)
Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

GekkoFS does support replication (in the main release) and erasure encoding (not in the main release). Replication was not enabled for IO500.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The benchmark was run using a slurm job. GekkoFS dependencies and the file system itself were built prior to execution for ALL MN5 users via a module file. The file system was started before running the benchmark and contained no data. The benchmark was run during production operation of MN5. However, all used IO500 nodes were exclusive to the user. The file system was only accessed by the IO500 benchmark and we asked sbatch to reduce the number of switches to 2 (minimum possible for 125 nodes).

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

Client data/metadata caching (in Linux page cache or in other memory cache)
Client side NVMe read-only data cache
Storage server metadata/data caching in RAM
Storage controller caching
RAID card caching

GekkoFS Servers use the existing node-local file system (EXT4 in our case) and therefore the Linux page cache was used.

All metadata and data operations were performed synchronously from client to server without any caching or batching involved. Two GekkoFS cache mechanisms were enabled:

Write size cache: This mode only lets the client periodically update the file size to the server (set to 1000 write operations) or when fsync() or close() is called.

Readdir cache: On opening a directory, the directory entries and their metadata are transferred and stored on the client until the directory is closed. This avoids consecutive stat RPC during `ls`.

Find operation filtering is done in the user side, in this occasion the filtering is done in the server side (options --server-side and -C)

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

GekkoFS servers use existing node-local storage and the corresponding file system. It was therefore the primary storage for IO500.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

Did you run the benchmark multiple times and get similar scores?
Did you validate the score is below the physical capabilities of the deployed hardware?
How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was run several times (15+) with consistent results that were within expectations for the hardware available.

We repeated the test because sometimes the selection of nodes, selected SSDs with low performance (wear leveling). We also tuned the number of client processes to provide more consistent results, but didn't found a direct relation between ior-easy-write and them. ior-easy-write was the only indicator that the test will go up to 600 points or not, and we suspect wear leveling or NVMe low performance.

The results were close to the physical capabilities of the deployed hardware both concerning storage and network performance and a big improvement with a 400-->125 server nodes from the last code iteration.

Because all operations in GekkoFS are synchronous, all data was written to node-local storage on the server before the client received the result. This also applied to metadata.