Gautschi

Purdue University · DDN /scratch/gautschi · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

Gautschi is a Community Cluster optimized for communities running traditional, tightly-coupled science and engineering applications. The Gautschi Community Cluster is equipped with both CPU and GPU compute nodes, each designed for specific computational tasks. The cluster includes Dell CPU compute nodes that feature dual 96-core AMD Epyc "Genoa" processors, providing 192 cores per node and 384 GB of memory. GPU compute nodes come with two 56-core Intel Xeon Platinum 8480+ processors, a total of 1031 GB of CPU memory, and eight NVIDIA H100 GPUs, each boasting 80 GB of memory.  All compute nodes have 400 Gbps NDR Infiniband interconnect and service through 2030.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

Gautschi was built in November 2024, entered production in February 2025(?) and will be retired in 2030. It serves Purdue University's faculty and research groups from every School at Purdue. 

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The Gautechi scratch storage system software is DDN EXAScaler, which is a commercially-supported product built on the open-source Lustre filesystem.  The product webpage can be found here: https://www.ddn.com/products/lustre-file-system-exascaler/ .  I believe anyone can purchase this software and support.  Lustre itself is available as open-source software here: https://www.lustre.org/

Gautschi scratch is 3 SFA400NV2XE with 2 controllers each. On this hardware is 12 VMs that are all mds/oss nodes. Each target is highly available. Each server has 4x50 HDR Infiniband links for 200Gib/s connectivity.

Gautschi makes use of Lustre components such as project quota and distributed namespace. 

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

I have submitted the ini file used in the benchmarking along with the results tarball and stdout/stderr. 

The benchmark was run as close to stock as possible. No containers, lipe_scan scripts, or multiple mountpoints were used in these runs. openmpi 5.0.5 was used with gcc 14.1.0

On clients, the following non-default settings were applied: 

lctl set_param osc.*.max_rpcs_in_flight=32
lctl set_param osc.*.max_dirty_mb=256
lctl set_param osc.*.checksums=0

for the benchmarking directories the script uses to place data: 
  990  2025-11-06 00:15:50 lfs setdirstripe -D -c -1 real-test
  991  2025-11-06 00:15:52 lfs setstripe -c 1 real-test
  992  2025-11-06 00:16:13 lfs setdirstripe -D -c 1 -i -1 -X -1 --max-inherit-rr 3 real-test/mdtest-easy
  993  2025-11-06 00:16:16 lfs setstripe -C 96 -c -1 real-test/ior-hard
  994  2025-11-06 00:16:19 lfs setdirstripe -c -1 -D real-test/mdtest-hard



Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

All systems are redundant from the power having split PDUs and being on separate power feeds. The storage uses distributed RAID provided be DDN SFA OS to protect against disk failure. The whole group can lose several disks overtime without data loss due to built-in spare capacity.

DDN Exascaler provides failover resiliency for individual server components to ensure metadata and data services remain online 

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The io500 benchmark was run interactively on 10 client nodes marked for reservation while the rest of the system was in production though the benchmark was run via SLURM submission and should produce similar results. The system was not dedicated to running the benchmark as numerous other jobs were running on the system simultaneously.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

For ior-easy tests, Direct IO was used, so no client page cache was in use.  The rest of the tests used regular POSIX I/O, so Linux page cache could be used for those. Please see the pjc.ini file for verification. 

There is no caching storage on the clients

The storage servers have a write-back cache and battery backed cache at the SFA OS layer 

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The system is a stand-alone storage system.  It is not a burst buffer layered on primary storage.  All clients mount the storage as Lustre via nfs.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was run many times over 2 weeks with little variation in score.

The vendor provided guidance to the maximum throughput capabilities of the system, and I validated the score was below these figures, as expected.  In monitoring, I could see the throughput capability could be reached, but not exceeded.

The storage system is a POSIX-compliant, cache-coherent parallel filesystem with global lock management and flock enabled.  I did not verify persistence personally, but the Lustre filesystem is known to comply with io500 rules regarding persistence.