AI400X3M
DDN · DDN AI400X3M · Fully Reproducible
System Purpose
Question asked
Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.
A benchmarking and system-test environment for DDN EXAScaler (Lustre) storage with Kubernetes clients.
It is used to evaluate and tune storage performance for AI and HPC workloads, and to validate container-based access to EXAScaler through the DDN EXA CSI driver.
Availability
Question asked
Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.
Please describe the availability of the system to users and who are its set of most regular users.
Deployed 2026/01 as an internal test system. Not available to external users; the regular users are DDN engineering staff (benchmarking, QA, development).
Storage System Software
Question asked
How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.
Can anyone download/purchase this software?
List either product webpage or open-source repo of the exact software used in IO500.
Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).
- Availability: commercially available from DDN.
- Can anyone purchase it: yes.
- Software: DDN EXAScaler 6.3.9.
- Product page (AI400X3M): https://www.ddn.com/products/ai400x3m/
Runtime Environment
Question asked
State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.
NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.
Several examples include:
- Commands used to set striping information (either for the entire system or for particular directories)
- File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
- Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
Kubernetes environment
All IO500 clients ran inside a dedicated 10-node Kubernetes cluster (v1.31). The control plane runs on one of
the 10 nodes; no other nodes belong to the cluster.
- Job: a Kubeflow MPI Operator MPIJob (v2beta1) with 80 worker pods (8 per node) and 8 MPI ranks per pod, so 640 ranks. The launcher pod runs on one of the same 10 nodes. MPICH 5.0.2 (ch4:ucx) over RoCE; ranks are started over ssh into the worker pods, and the pods use host networking.
- CPU/memory: Guaranteed QoS pods with the kubelet static CPU manager. Each worker gets 15 dedicated CPUs within one socket (4 pods per socket) and 40 GiB. The launcher verifies rank placement before io500 starts.
- Storage access: through the DDN EXA CSI driver. Each worker pod has its own PersistentVolumeClaim, bound to a pre-provisioned static PersistentVolume; each PersistentVolume is a separate Lustre client mount of the same IO500 directory (no bind mount), so every pod uses its own Lustre client (8 per node). The client tuning parameters are part of the PersistentVolume definition and are applied by the CSI driver when it mounts the volume.
- RDMA: NICs are exposed to pods through the RDMA shared device plugin.
- Cluster membership (kubectl get nodes, 2026-11-06 05:57 UTC):
NAME STATUS ROLES AGE VERSION
k8n01 Ready control-plane 26d v1.31.14
k8n02 Ready <none> 26d v1.31.14
k8n03 Ready <none> 26d v1.31.14
k8n04 Ready <none> 26d v1.31.14
k8n05 Ready <none> 26d v1.31.14
k8n06 Ready <none> 26d v1.31.14
k8n07 Ready <none> 26d v1.31.14
k8n08 Ready <none> 26d v1.31.14
k8n09 Ready <none> 26d v1.31.14
k8n10 Ready <none> 26d v1.31.14
- Evidence of placement: the io500 tarball contains rankmap.txt (rank → host → CPU socket; 64 ranks on each of the 10 nodes). Note: IOR and mdtest report "640 nodes" because each pod looks like a separate host to them.
Scripts and configuration
All scripts and configuration needed to reproduce the run are provided [CONFIRM: io500.tgz / extraCodes / GitHub link]:
Kubernetes manifests (MPIJob, PersistentVolumes), io500.sh (including setup() with all striping commands),
io500.ini, ROMIO hints, the container image recipe and the run script.
Non-default parameters
Servers (EXAScaler 6.3.9, 4 instances)
- Kernel cmdline: spec_rstack_overflow=ibpb
- libcfs cpu_npartitions=10
- mdt.*.identity_upcall=NONE; mdt.*.enable_remote_dir_gid=-1
- EXAScaler deployment settings: obdfilter.*.brw_size=16 (MiB), quota enforcement enabled (ugp), ldiskfs targets mounted with online discard (fstrim)
Clients (10 Kubernetes nodes)
- Per-mount parameters (from the PersistentVolume, applied by the CSI driver): osc.*.max_rpcs_in_flight=32 osc.*.checksums=0 ldlm.namespaces.*.lru_max_age=5000 llite.*.max_read_ahead_mb=2048 llite.*.max_read_ahead_per_file_mb=128 llite.*.max_read_ahead_whole_mb=16 llite.*.max_cached_mb=4096
- LNet: multi-rail over 2 RoCE NICs
- Hosts: tuned hpc-compute profile, performance governor (all-core ~2.8 GHz), uncore frequency fixed at 2.3 GHz
- RoCE: MTU 9000, DSCP trust, PFC on priority 3, ECN on priority 3
MPI / ROMIO
- MPICH 5.0.2 (ch4:ucx) with ROMIO patches (ROMIO_PTHREADIO, Lustre background write thread)
- mpiexec -n 640 -ppn 1, UCX_TLS=rc,self, UCX_IB_GID_INDEX=3, two NICs (UCX_MAX_RNDV_RAILS=2, UCX_MAX_EAGER_RAILS=2, UCX_MAX_RNDV_LANES=2), ROMIO_LUSTRE_CO_RATIO=4, ROMIO_PTHREADIO=1
- ROMIO hints (ior-hard): cb_nodes 320, cb_config_list *:32, romio_cb_write enable, romio_ds_write disable, romio_ds_read disable, romio_lustre_ds_in_coll disable, direct_write true
Fault Tolerance Mechanisms
Question asked
Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.
- Power (For example, can power failure of a node or rack bring down the storage system?)
- Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
- Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
- Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)
Please list any additional information needed to determine whether this system has a single point of failure.
- Power: redundant PSUs in the appliance enclosure, connected to dual PDUs/power feeds.
- Server: the 4 server instances form one Pacemaker/Corosync HA group (with STONITH). Every instance sees every MDT/OST volume, so any instance can take over a failed instance's targets. The two SFA controllers are active-active; each volume has a home controller and fails over to the other.
- Media: RAID6 (tolerates 2 drive failures per RAID set) in declustered pools with distributed spare capacity and multi-rebuild enabled.
- Single point of failure: none for single-component failures (controller, server instance, NIC, PSU, drive). The system is a single appliance enclosure.
Execution
Question asked
Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.
During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?
The benchmark ran as a Kubernetes MPIJob on the dedicated 10-node Kubernetes cluster described above; no batch
scheduler (e.g. SLURM) was involved.
The system was dedicated to the benchmark; no other jobs ran on the cluster or the storage system.
Caching
Question asked
Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).
A few examples would include:
- Client data/metadata caching (in Linux page cache or in other memory cache)
- Client side NVMe read-only data cache
- Storage server metadata/data caching in RAM
- Storage controller caching
- RAID card caching
- Client: the Lustre client uses the Linux page cache for buffered I/O (default behaviour). ior-easy uses O_DIRECT; ior-hard uses ROMIO collective I/O with direct writes.
- Server: Linux page cache and ldiskfs metadata caching in RAM.
- SFA controller cache: write-back, mirrored between the two controllers.
- No client-side NVMe or other caching layers.
Data Source
Question asked
Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.
Stand-alone primary storage. The EXAScaler file system on the AI400X3M is the source of truth for all data;
it is not a burst buffer or cache in front of another storage system.
Trust
Question asked
Please describe any steps taken to ensure that the results are trustworthy.
- Did you run the benchmark multiple times and get similar scores?
- Did you validate the score is below the physical capabilities of the deployed hardware?
- How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?
- Multiple runs: full runs with the identical procedure. Runs 1-5 are within 1% of each other.
- Below hardware limits: the results are below the aggregate bandwidth of the 4 x ConnectX-8 400 Gb/s server network links and of the 20 x Phison X200 NVMe drives.
- Persistence: IOR fsyncs files inside the timed phases, and Lustre does not complete a sync until the servers have committed it. The SFA acknowledges a write once it is mirrored in both controllers' cache, which is protected against power loss.