Blue Vela Storage Scale System 6000 - shared
IBM · IBM ess6000-1 · Fully Reproducible
System Purpose
Question asked
Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.
The IBM Blue Vela system is an advanced, on-premises solution tailored for training large language models (LLMs) and other foundational models designed and run by IBM Research for internal workloads.
This system is particularly well-suited for a range of AI and HPC workloads, including but not limited to training and fine-tuning models across numerous modalities, such as language, speech, time series, document understanding, and multimodal workloads; developing agentic capabilities; benchmarking; system testing; and systems research. By providing a robust infrastructure, Blue Vela facilitates the development and training of large, complex models, enabling researchers and organizations to derive insights across various domains.
IBM can leverage data generated by workloads running on Blue Vela for real-time analysis, model refinement, and performance benchmarking. This data is crucial for advancing research initiatives, optimizing operational processes, and supporting strategic decision-making across diverse fields. Ultimately, the system empowers Researchers and Developers to harness cutting-edge computational capabilities for innovative and impactful applications.
Availability
Question asked
Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.
Please describe the availability of the system to users and who are its set of most regular users.
Blue Vela is a production system run by IBM Research for internal workloads, comprising 768 H100-based Dell XE9680s. The primary workload for Blue Vela is internal AI model development and testing, along with a growing quantum HPC workload used by Researchers, AI and Data Engineers, and Software Developers. Compute is accessed via an LSF job scheduler for mission-critical AI workloads.
While the system is comprised of 768 compute nodes, the io500 run was performed on an allocation of 32 nodes made available for MPI job access outside LSF, and additional tuning was performed in collaboration between IBM Storage Scale Development and IBM Research.
During the io500 run submitted, only 2.6% of the cluster's compute capacity, the remaining ~97% was fully occupied by various IBM Research workloads. Overall, the system serves a diverse Research user base, which is critical to driving innovation and efficiency in AI and HPC projects.
Storage System Software
Question asked
How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.
Can anyone download/purchase this software?
List either product webpage or open-source repo of the exact software used in IO500.
Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).
IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.
Anyone can purchase the software or download a limited use version (it is available commercially).
The product webpage is https://www.ibm.com/products/storage-scale
Regarding the specific storage deployment, the Blue Vela uses four building blocks of the IBM Storage Scale System 6000 solution, which is described here:
https://www.ibm.com/downloads/documents/us-en/10a99803f5afd8d6
Runtime Environment
Question asked
State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.
NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.
Several examples include:
- Commands used to set striping information (either for the entire system or for particular directories)
- File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
- Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
1. The submitted io500-login0-2026.11.05-12.14.19.tgz archive has mmlsconfig and mmlscluster data from both the storage and client clusters.
2 The PCI tuning option PCI_WR_ORDERING has been enabled on all nodes as follows:
Find the MST device name of each adapter, e.g.:
mst start
mst status
/dev/mst/mt4123_pciconf0
Enable PC_WR_ORDERING for each such device, e.g.:
mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1
<reboot for the change to take effect>
3. The file system block is 16 MB (file system block size is a choice based on the use case)
4. The GNR erasure encoded pool uses 8+2P encoding, which is a choice based on the use case (8+2P is the most common erasure encoding option used for the storage configuration).
5. The file system created for the io500 run preallocates 500M inodes (--mmcrfs --inode-limit 1500M:500M) out of spare NVMe drive space that was not in use for the production file system. There are no dedicated resources (e.g. servers, or IB adapter links) used on the servers – all server resources are shared with production workloads running under LSF.
Fault Tolerance Mechanisms
Question asked
Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.
- Power (For example, can power failure of a node or rack bring down the storage system?)
- Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
- Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
- Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)
Please list any additional information needed to determine whether this system has a single point of failure.
IBM Storage Scale servers are designed with power redundancy in mind.
The network switch is a single point of failure (not-dual redundant switches)
The storage subsystem employs IBM Storage Scale RAID to use 8+2 erasure encoding and, for some metadata, replication to protect file system data from hardware component failures.
Execution
Question asked
Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.
During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?
The io500 run ran concurrently with production workloads on the system, using the same production file system for logging, system libraries, mounted on all the storage cluster nodes and all the available H100-based Dell XE9680 clients.
A SLURM reservation was placed for 32 Dell XE9680 clients, so they were drained from the regular LSF production workloads. Only 20 of the Dell XE9680 clients are used for the io500 run.
Additional configuration options were applied (the above Runtime Environment section captures the mmlsconfig output for the io500 node class client nodes). The clients ran io500's dataset against a newly created file system that was dedicated for this work, but using IBM Storage Scale System 6000 NVMe drives used for the production file system, using spare space on the NVMe drives that was not allocated to the production file system. The purpose of using a separate file system for io500's datasets was to to allow us to experiment with file system configuration options. There are no dedicated resources on the servers and the only file system level tuning that was done for io500 was to preallocate 500M inodes before running io500.
Caching
Question asked
Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).
A few examples would include:
- Client data/metadata caching (in Linux page cache or in other memory cache)
- Client side NVMe read-only data cache
- Storage server metadata/data caching in RAM
- Storage controller caching
- RAID card caching
On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching. The effect of this caching provides similar caching to that obtained using solid state drive caching systems, and the NVMe drives on the servers also provide solid state caching.
Data Source
Question asked
Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.
There is no burst buffer used – it is the primary storage itself. Caching on the server-side page pool is analogous to how disk drive caching works.
Trust
Question asked
Please describe any steps taken to ensure that the results are trustworthy.
- Did you run the benchmark multiple times and get similar scores?
- Did you validate the score is below the physical capabilities of the deployed hardware?
- How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?
Similar scores have been achieved in several other runs of the benchmark on the same system.
We have validated that the scores achieved are within the physical capabilities of the system.