FrightPro SR800 S4

Telecommunications Technology Association · Gluesys ExaStor · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The client nodes and the storage system are hosted at the HPC Innovation Hub at TTA. The storage system, ExaStor F824 by Gluesys, is designed for high capacity and high computational performance to support AI workloads. It is deployed for production use and is currently undergoing a research evaluation to further improve its performance.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The system was deployed in September 2025 and will remain available for a minimum five years at the HPC Innovation Hub at TTA.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The software stack includes both open-source and proprietary components:

All software components are publicly available for download or purchase as applicable.

The storage consists of one-node which includes:

  • 2x Intel(R) Xeon(R) Gold 5416S
  • 4x 64GB DDR5 Memory
  • 16x dual-port NVMe drives (PCIe Gen5)
    • 1x MGT (Management Target)
    • 2x MDT (Metadata Target)
    • 13x OSTs (Object Storage Targets)
  • 2× InfiniBand 100Gbps links

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

The following Lustre tunings were applied:

Serverside (persistent) Lustre settings

lctl set_param -P obdfilter.*.brw_size=16
lctl set_param -P obdfilter.*.precreate_batch=1024
lctl set_param -P obdfilter.*.osc.max_pages_per_rpc=4096
lctl set_param -P osc.*.max_pages_per_rpc=4096
lctl set_param -P mdc.*.max_rpcs_in_flight=128
lctl set_param -P osc.*.max_rpcs_in_flight=128
lctl set_param -P mdc.*.max_mod_rpcs_in_flight=127
lctl set_param -P llite.*.checksum_pages=0
lctl set_param -P llite.*.checksums=0
lctl set_param -P osc.*.checksums=0
lctl set_param -P mdc.*.checksums=0

Clientside Lustre settings

lctl set_param ldlm.namespaces.*.lru_size=12800
lctl set_param mdc.*.max_rpcs_in_flight=128
lctl set_param mdc.*.max_mod_rpcs_in_flight=127
lctl set_param osc.*.max_rpcs_in_flight=128
lctl set_param llite.*.max_read_ahead_mb=2048
lctl set_param osc.*.short_io_bytes=65536
lctl set_param ldlm.namespaces.*.lru_max_age=1000

zfs tuning is also applied:

zfs set atime=off MGSpool
zfs set checksum=off MGSpool
zfs set compression=off MGSpool
zfs set xattr=sa MGSpool
zfs set redundant_metadata=most MGSpool

zfs set atime=off MDT${i}pool
zfs set checksum=off MDT${i}pool
zfs set compression=off MDT${i}pool
zfs set xattr=sa MDT${i}pool
zfs set redundant_metadata=most MDT${i}pool

zfs set atime=off OST${i}pool
zfs set checksum=off OST${i}pool
zfs set recordsize=1M OST${i}pool
zfs set compression=off OST${i}pool
zfs set xattr=sa OST${i}pool

For Lustre striping etc. check the submitted io500.sh.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Power
The system has power level redundancy (redundant PSUs).

Networking

Each storage server has 2 InfiniBand HCAs for performance and redundancy. 

Inter storage data and metadata server

We only test one storage server with 1 MGT NVMe drive, 2 MDT NVMe drives and 13 OST NVMe drives. 

Intra storage data and metadata server storage media
No RAID is used for the performance test.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

We simply used the command "sh io500.sh config-tta.ini" and did not use any batch scheduler.

The IO500 benchmark was executed using mpich 4.1.1 across 6 client nodes, with 52 MPI processes per node.

During the IO500 benchmark execution the system was entirely dedicated to running the benchmark.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

Since we use ZFS as the backend filesystem, the storage server caches some metadata and data in RAM (via the ZFS ARC).

Apart from that, we did not enable any additional caching for the IO500 runs.

For the clients, native Lustre client cache was used.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The submitted system is mainly for a stand-alone storage system, a primary storage itself.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was run multiple times and yielded consistent results across runs. The observed performance is within the physical capabilities of the deployed hardware. We also verified that data was durably written to storage and that the entire procedure adheres to the IO500 rules.