KISTI-6 HANGANG
Korea Institute of Science and Technology Information · HPE Cray /scratch · Fully Reproducible
System Purpose
Question asked
Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.
The submitted system is HANGANG, KISTI's sixth-generation national supercomputer, operated by the KISTI National Supercomputing Center(www.ksc.re.kr). The storage system under test is the production /scratch Lustre parallel file system, which supports large-scale scientific simulations, AI workloads, and data-intensive research conducted by academic and industrial users across Korea.
The /scratch file system provides high-performance shared storage for computational workloads running on NVIDIA GH200 Grace Hopper-based compute nodes. It accommodates application input datasets, checkpoint files, intermediate results, and simulation or AI output data. Users typically analyze these results or transfer them to longer-term storage as needed.
The IO500 benchmark was executed directly on the same production /scratch file system used by regular HPC users, rather than on a dedicated benchmark-only storage environment. All benchmark runs were conducted using standard user accounts and the production Slurm batch scheduling system, under normal shared-system operating conditions.
Availability
Question asked
Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.
Please describe the availability of the system to users and who are its set of most regular users.
The HANGANG system and its ClusterStor /scratch Lustre file system completed final installation in October 2026. Following a beta service period, official production service is scheduled to begin in December 2026. It is a permanent on-premises HPC system serving academic and industrial researchers across Korea. The IO500 benchmarks were conducted on October 20, 2026 (10 Node Challenge) and October 23, 2026 (Production).
Storage System Software
Question asked
How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.
Can anyone download/purchase this software?
List either product webpage or open-source repo of the exact software used in IO500.
Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).
The storage system is a commercially available HPE Cray ClusterStor solution based on the open-source Lustre parallel file system. It is deployed as the shared /scratch file system of KISTI's HANGANG supercomputer.
The system serves approved academic, government, and industrial researchers in Korea through KISTI's national supercomputing allocation program (2026 National Supercomputer No. 6 Innovation Support Program). The /scratch file system is accessible from 2,084 NVIDIA GH200 and 800 CPU compute nodes through Slurm-managed jobs.
For this IO500 submission, only NVIDIA GH200 compute nodes were used, and all benchmark data was placed on the NVMe flash OST pool (272 OSTs). The HDD OST pool was not used for benchmark data.
During IO500 benchmarking, the file system remained in shared production operation, with large-scale tests scheduled during periods of relatively low user activity.
The ClusterStor storage system and its software are commercially available from HPE, and the Lustre client and server source code is publicly available from the Lustre community repository. The IO500 benchmark itself was the unmodified io500-sc26_v1 release (git tag io500-sc26_v1; IOR, mdtest, and pfind unmodified).
Software references:
-
HPE Cray ClusterStor: https://www.hpe.com/us/en/compute/hpc/storage.html
-
Lustre (open source): https://github.com/lustre/lustre-release
Runtime Environment
Question asked
State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.
NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.
Several examples include:
- Commands used to set striping information (either for the entire system or for particular directories)
- File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
- Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
We provide the scripts, configuration files, and documentation necessary to reproduce our benchmark environment and attempt to achieve a similar IO500 score. These include the official-template io500.sh, the exact config.ini, Slurm batch scripts, environment and case files, submission commands, job logs, IO500 stdout, Slurm allocation records, and io500-verify output. All files are included in the results tarball, at the top level or under job-extra/. The IO500, IOR, mdtest, and pfind source codes were not modified.
1. Directory Layout and Striping
The benchmark base directory was striped across all 24 MDTs using CRUSH hashing. Directory inheritance was configured to distribute MDTEST Easy rank directories across MDTs, while MDTEST Hard used a shared 24-way striped directory.
-
IOR Easy: 1 Flash OST per file, 16 MiB stripe size.
-
IOR Hard (Production): 1,088 stripes across 272 Flash OSTs (4-way overstriping), 4 MiB stripe size.
-
IOR Hard (10 Node Challenge): All Flash OSTs, 1 MiB stripe size, MPI-IO collective I/O.
-
MDTEST Easy/Hard: 64 KiB Data-on-MDT (DoM) layout.
2. Lustre Client Tuning
The following settings were applied to all benchmark client nodes:
-
MDC
max_rpcs_in_flight = 128 -
MDC
max_mod_rpcs_in_flight = 63 -
OSC
max_rpcs_in_flight = 256 -
OSC
max_dirty_mb: System default, not explicitly modified.
For the 10 Node Challenge, the MDC lock LRU size was set to 10,000, and eight Lustre client mount instances per node were used through a mount-namespace wrapper (mmwrap-v2.sh).
3. MDS/OSS Server Configuration
Server-side tuning details are pending confirmation from the storage administrator. Any non-default MDS/OSS parameters will be documented if applicable.
4. IO500 Configuration
Both runs used a 300-second stonewall time with cache dropping disabled.
Production:
-
IOR Easy: POSIX with O_DIRECT, 64 MiB transfer size.
-
IOR Hard: POSIX, segment count 170,000.
-
MDTEST Easy: 90,000 files per configured MDTEST iteration.
-
MDTEST Hard: 18,000 files per configured MDTEST iteration.
-
Pfind: Single-directory access parallelization using hashing enabled.
10 Node Challenge:
-
IOR Easy: Same as Production.
-
IOR Hard: MPI-IO with collective I/O enabled.
-
Pfind: Single-directory access parallelization using hashing enabled.
-
Additional MPICH MPI-IO hints were applied as documented in the included environment files.
5. Job Execution
-
Production: 1,632 nodes, 4 MPI ranks per node (6,528 ranks).
-
10 Node Challenge: 10 nodes, 192 MPI ranks per node (1,920 ranks), using
mmwrap-v2.sh. -
MPI environment:
FI_CXI_RX_MATCH_MODE=software,FI_CXI_RDZV_PROTO=alt_read, andMPICH_OFI_NIC_POLICY=NUMA.
The complete execution commands, Lustre layout settings, environment variables, and supporting scripts are included in the submitted results archive.
Fault Tolerance Mechanisms
Question asked
Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.
- Power (For example, can power failure of a node or rack bring down the storage system?)
- Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
- Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
- Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)
Please list any additional information needed to determine whether this system has a single point of failure.
Single Point of Failure: No (subject to final configuration verification)
Power: ClusterStor servers and enclosures have redundant power supplies connected to independent power distribution circuits. Server failures are handled by HA partner nodes.
Networking: HPE Slingshot networking and Lustre LNet Multi-Rail provide redundant communication paths. Compute nodes have four Slingshot NICs, and storage servers use redundant network connections.
Storage Server HA: OSS and MDS servers operate in HA pairs with shared storage. Upon server failure, the surviving partner takes over the affected Lustre targets, allowing clients to reconnect and resume I/O. The system includes 68 SSUs and 24 MDTs.
Storage Media Protection: Flash OSTs, HDD OSTs, and MDTs use RAID-based protection. Shared storage connectivity enables HA failover, while RAID protects against individual drive failures.
Additional Information: Exact RAID layouts, power failure domains, storage-server network redundancy, and MDS/MGS failover configurations require final verification.
Execution
Question asked
Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.
During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?
The IO500 benchmark was executed as a Slurm batch job (sbatch) under a regular user account. The job script verified Lustre mounts, client tunables, and MPI/network health on all allocated nodes, then prepared the run directory using the preconfigured DNE/DoM layout.
The unmodified IO500 benchmark was launched using srun:
-
Production: 1,632 nodes × 4 MPI ranks (6,528 ranks)
-
10 Node Challenge: 10 nodes × 192 MPI ranks (1,920 ranks)
After execution, the results were validated using io500-verify.
The /scratch file system remained shared with other users (exclusive_cluster_usage = No). Benchmark runs were scheduled during periods of relatively low system activity to minimize interference.
Caching
Question asked
Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).
A few examples would include:
- Client data/metadata caching (in Linux page cache or in other memory cache)
- Client side NVMe read-only data cache
- Storage server metadata/data caching in RAM
- Storage controller caching
- RAID card caching
Client: Standard Linux page cache and Lustre client-side data/metadata caching were used. IOR Easy employed O_DIRECT to bypass the client data page cache. IOR read workloads used task reordering (-C) to read data written by other nodes, reducing cache effects. Cache dropping was disabled (drop-caches = FALSE). For the 10 Node Challenge, the MDC lock LRU size was set to 10,000.
Client-side SSD/NVMe Cache: None. Compute nodes are diskless and do not use local SSD/NVMe caching.
Storage Servers: Standard Lustre/ldiskfs server-side RAM caching and filesystem journaling were used. No additional benchmark-specific server caching was configured.
Storage Controller/RAID Cache: No benchmark-specific cache tuning was applied; hardware cache settings remained at their production defaults, subject to configuration verification.
Data Source
Question asked
Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.
The submitted /scratch Lustre file system is a stand-alone primary storage system, not a burst buffer or caching layer in front of another storage system. It directly stores application input data, checkpoints, intermediate results, and output data for HPC and AI workloads.
Trust
Question asked
Please describe any steps taken to ensure that the results are trustworthy.
- Did you run the benchmark multiple times and get similar scores?
- Did you validate the score is below the physical capabilities of the deployed hardware?
- How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?
Reproducibility: The benchmark was executed multiple times with comparable results. Production scores were 1059.71, 1124.12, and 1155.89; 10 Node Challenge scores were 311.00 and 346.03. All runs passed io500-verify, using a 300-second stonewall time.
Physical Limits: The measured peak bandwidth of approximately 9.5 TiB/s (IOR Easy Read) was below the aggregate theoretical client network bandwidth (1,632 nodes × 4 × 400 Gb/s). Storage-side hardware bandwidth limits require separate verification.
Data Persistence: The unmodified IO500 benchmark was used with its standard synchronization mechanisms, including IOR fsync before close (-e) and MDTEST synchronization (-Y). IOR Easy additionally used O_DIRECT. The Lustre file system uses RAID-protected persistent storage. No benchmark source code was modified to bypass synchronization or durability requirements.