Blue Vela Storage Scale System 6000 - shared

IBM · IBM ess6000-1 · Fully Reproducible

Files
Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The IBM Blue Vela system is an advanced, on-premises solution tailored for training large language models (LLMs) and other foundational models designed and run by IBM Research for internal workloads.

This system is particularly well-suited for a range of AI and HPC workloads, including but not limited to training and fine-tuning models across numerous modalities, such as language, speech, time series, document understanding, and multimodal workloads; developing agentic capabilities; benchmarking; system testing; and systems research. By providing a robust infrastructure, Blue Vela facilitates the development and training of large, complex models, enabling researchers and organizations to derive insights across various domains.

IBM can leverage data generated by workloads running on Blue Vela for real-time analysis, model refinement, and performance benchmarking. This data is crucial for advancing research initiatives, optimizing operational processes, and supporting strategic decision-making across diverse fields. Ultimately, the system empowers Researchers and Developers to harness cutting-edge computational capabilities for innovative and impactful applications.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

Blue Vela is a production system run by IBM Research for internal workloads, comprising 768 H100-based Dell xe9680s. The primary workload for Blue Vela is internal AI model development and testing, along with a growing quantum HPC workload used by Researchers, AI and Data Engineers, and Software Developers. . Compute is accessed via an LSF job scheduler for mission-critical AI workloads. While the system is comprised of 768 compute nodes, the io500 run was performed on an allocation of 20 nodes made available for MPI job access outside LSF, and additional tuning was performed in collaboration between IBM Storage Scale Development and IBM Research. During the io500 runs, which occupied 2.6% of the cluster's compute capacity, the remaining 97% was fully occupied by various IBM Research workloads. Overall, the system serves a diverse Research user base, which is critical to driving innovation and efficiency in AI and HPC projects.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.

Anyone can purchase the software or download a limited use version (it is available commercially).

The product webpage is https://www.ibm.com/products/storage-scale

Regarding the specific storage deployment, this submission uses two building blocks of the IBM Storage Scale 3500 solution, which is described here:
https://www.ibm.com/downloads/documents/us-en/10a99803f5afd8d6

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

1. The mmlsconfig info is below

Configuration data for cluster dfw_ess6000.bluevela.rmf.ibm.com:

----------------------------------------------------------------

clusterName dfw_ess6000.bluevela.rmf.ibm.com

clusterId 9941035806433427289

dmapiFileHandleSize 32

tscCmdAllowRemoteConnections no

ccrEnabled yes

sdrNotifyAuthEnabled yes

maxblocksize 16m

[s6k_ess6000io1_ib_ess6000io2_ib,s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

nsdRAIDTracks 320K

nspdQueues 240

[ess3500_ess3500a1_ib_ess3500b1_ib,s6k_ess6000io1_ib_ess6000io2_ib,s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

nspdBufferMemPerQueue 24m

nspdThreadsPerQueue 2

nsdMinWorkerThreads 3842

nsdMaxWorkerThreads 3842

nsdSmallThreadRatio 1

nsdRAIDEventLogToConsole all

nsdRAIDBlockDeviceMaxSectorsKB 0

nsdRAIDBlockDeviceNrRequests 0

nsdRAIDBlockDeviceQueueDepth 0

nsdRAIDBlockDeviceScheduler off

nsdRAIDSmallThreadRatio 2

nsdRAIDDefaultGeneratedFD no

nsdRAIDMasterBufferPoolSize 2G

nsdRAIDReconstructAggressiveness 0

nsdRAIDThreadsPerQueue 16

nsdRAIDSSDPerformanceShortTimeConstant 2500000

nsdRAIDDiskDiagTimeout 130

[common]

numaMemoryInterleave no

[ess3500_ess3500a1_ib_ess3500b1_ib,s6k_ess6000io1_ib_ess6000io2_ib,s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

numaMemoryInterleave yes

panicOnIOHang yes

pitWorkerThreadsPerNode 32

prefetchPct 50

maxFilesToCache 128k

maxStatCache 128k

nsdRAIDClientOnlyChecksum yes

pagepoolMaxPhysMemPct 90

[common]

mmhealth-events-ignore ccr_comm_dir_warn

pagepool 1G

[s6k_ess6000io1_ib_ess6000io2_ib]

pagepool 486294451814

[ems2-vm1-ib,utility]

pagepool 32G

[common]

workerThreads 48

[ess3500_ess3500a1_ib_ess3500b1_ib,s6k_ess6000io1_ib_ess6000io2_ib,s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

workerThreads 1024

[utility]

workerThreads 256

[common]

ignorePrefetchLUNCount no

[ess3500_ess3500a1_ib_ess3500b1_ib,s6k_ess6000io1_ib_ess6000io2_ib,s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

ignorePrefetchLUNCount yes

[utility]

ignorePrefetchLUNCount no

[s6k_ess6000io5_ib_ess6000io6_ib]

pagepool 486357658828

[ess3500_ess3500a1_ib_ess3500b1_ib,s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

nsdRAIDEnableSpaceMonitor 1

[s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

nsdRAIDFastWriteFSMetadataLimit 256k

[s6k_ess6000io3_ib_ess6000io4_ib]

pagepool 486351438643

[s6k_ess6000io3_ib_ess6000io4_ib,s6k_ess6000io5_ib_ess6000io6_ib,s6k_ess6000io7_ib_ess6000io8_ib]

nsdRAIDMaxPdiskQueueDepth default:0;hdd:128;ssd:0;nvme:0;nvdimm:0;nvram:0

[s6k_ess6000io3_ib_ess6000io4_ib]

nsdRAIDPagepoolBaseline 486351438643

[common]

autoload no

[ems2-vm1-ib]

autoload yes

[CALLHOME_SERVERS,GUI_MGMT_SERVERS,GUI_SERVERS,s6k_x86_64,utility]

autoBuildGPL yes

[s6k_ess6000io5_ib_ess6000io6_ib]

nsdRAIDPagepoolBaseline 486357658828

[s6k_ess6000io7_ib_ess6000io8_ib]

pagepool 486351443558

nsdRAIDPagepoolBaseline 486351443558

[common]

cipherList AUTHONLY

[ess3500a1-ib,ess3500b1-ib,ess6000io1-ib,ess6000io2-ib,ess6000io3-ib,ess6000io4-ib,ess6000io5-ib,ess6000io6-ib,ess6000io7-ib,ess6000io8-ib,utility]

verbsRdma enable

[common]

verbsRdmaCm disable

[ess3500a1-ib,ess3500b1-ib,ess6000io1-ib,ess6000io2-ib,ess6000io3-ib,ess6000io4-ib,ess6000io5-ib,ess6000io6-ib,ess6000io7-ib,ess6000io8-ib,utility]

verbsRdmaSend yes

[utility]

verbsPorts mlx5_0/1 mlx5_1/1

[ess6000io1-ib,ess6000io2-ib,ess6000io3-ib,ess6000io4-ib]

verbsPorts mlx5_0/1 mlx5_1/1 mlx5_2/1 mlx5_3/1

[ess6000io5-ib,ess6000io6-ib]

verbsPorts  mlx5_0/1 mlx5_1/1 mlx5_2/1 mlx5_3/1

[ess6000io7-ib]

verbsPorts mlx5_0 mlx5_1 mlx5_2 mlx5_3 mlx5_4 mlx5_5 mlx5_6 mlx5_7

[ess3500a1-ib,ess3500b1-ib,ess6000io8-ib]

verbsPorts mlx5_0 mlx5_1 mlx5_2 mlx5_3

[ess3500_ess3500a1_ib_ess3500b1_ib]

pagepool 324215793254

nsdRAIDTracks 128K

nspdQueues 120

nsdRAIDEnableRGCMRebalanceWeight yes

backgroundSpaceReclaim Good: 1 10% 60 1% Info: 0 6% 30 1% Warning: 0 3% 10 0% Critical: 0 2% 5 0% OOS: 0 1% 1 0%

nsdRAIDPagepoolBaseline 324215793254

[common]

minReleaseLevel 5.2.2.0

maxMBpS 100000

autoCompactDir no

dataShipServerBufferPct 50

fsyncIsGlobal no

maxBufferDescs 10000000

nBucketGroups 1024

preferDesignatedMnode yes

proactiveReconnect no

adminMode central

File systems in cluster dfw_ess6000.bluevela.rmf.ibm.com:

---------------------------------------------------------

/dev/ess6000-1

Client cluster:
(base) [jlewars@login4 io500]$ mmlsconfig

Configuration data for cluster dfw.bluevela.rmf.ibm.com:

--------------------------------------------------------

clusterName dfw.bluevela.rmf.ibm.com

clusterId 14605646151343901334

dmapiFileHandleSize 32

minReleaseLevel 5.1.9.0

tscCmdAllowRemoteConnections no

ccrEnabled yes

sdrNotifyAuthEnabled yes

[compute,utility]

verbsRdma enable

verbsRdmaSend yes

[common]

numaMemoryInterleave yes

ignorePrefetchLUNCount yes

workerThreads 1024

maxFilesToCache 128K

maxStatCache 128K

maxblocksize 16M

verbsRdmaCm disable

proactiveReconnect no

cipherList AUTHONLY

maxMBpS 100000

prefetchLargeBlockThreshold 4M

[io500]

dataShipClientBufferPct 50

dataShipClientBuffersPerServer 50

fsyncIsGlobal no

maxBufferDescs 10000000

nBucketGroups 1024

preferDesignatedMnode yes

prefetchPct 50

autoCompactDir no

disablePrefetchThreadRebalance yes

[common]

autoload no

verbsPorts mlx5_1/1 mlx5_6/1

pagepool 32G

[io500]

pagepool 1024G

[common]

adminMode central

File systems in cluster dfw.bluevela.rmf.ibm.com:

-------------------------------------------------

(none)


2.Using the ‘mmchnode’ all nodes in the storage cluster have been defined as ‘manager’ role nodes as shown in ‘mmlscluster’ data. 

Storage cluster mmlscluster below:

GPFS cluster information

========================

GPFS cluster name:         dfw_ess6000.bluevela.rmf.ibm.com

GPFS cluster id:           9941035806433427289

GPFS UID domain:           dfw_ess6000.bluevela.rmf.ibm.com

Remote shell command:      /usr/bin/ssh

Remote file copy command:  /usr/bin/scp

Repository type:           CCR

Node  Daemon node name                      IP address     Admin node name                       Designation

--------------------------------------------------------------------------------------------------------------

1   ess6000io1-ib.bluevela.rmf.ibm.com    10.127.100.5   ess6000io1-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

2   ess6000io2-ib.bluevela.rmf.ibm.com    10.127.100.6   ess6000io2-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

3   ess6000io3-ib.bluevela.rmf.ibm.com    10.127.100.7   ess6000io3-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

4   ess6000io4-ib.bluevela.rmf.ibm.com    10.127.100.8   ess6000io4-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

5   ems2-vm1-ib.bluevela.rmf.ibm.com      10.127.100.2   ems2-vm1-ib.bluevela.rmf.ibm.com      quorum-perfmon

6   mgmt-r03-m3-ib1.bluevela.rmf.ibm.com  10.127.50.3    mgmt-r03-m3-ib1.bluevela.rmf.ibm.com  perfmon

7   mgmt-r04-m3-ib1.bluevela.rmf.ibm.com  10.127.50.9    mgmt-r04-m3-ib1.bluevela.rmf.ibm.com  perfmon

8   ess6000io5-ib.bluevela.rmf.ibm.com    10.127.100.9   ess6000io5-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

9   ess6000io6-ib.bluevela.rmf.ibm.com    10.127.100.10  ess6000io6-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

10   ess6000io7-ib.bluevela.rmf.ibm.com    10.127.100.11  ess6000io7-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

11   ess6000io8-ib.bluevela.rmf.ibm.com    10.127.100.12  ess6000io8-ib.bluevela.rmf.ibm.com    quorum-manager-perfmon

12   mgmt-r02-m1-ib1.bluevela.rmf.ibm.com  10.127.49.1    mgmt-r02-m1-ib1.bluevela.rmf.ibm.com  perfmon

13   ess3500a1-ib.bluevela.rmf.ibm.com     10.127.100.3   ess3500a1-ib.bluevela.rmf.ibm.com     manager

14   ess3500b1-ib.bluevela.rmf.ibm.com     10.127.100.4   ess3500b1-ib.bluevela.rmf.ibm.com     manager

15   mgmt-r02-m4-ib1.bluevela.rmf.ibm.com  10.127.49.4    mgmt-r02-m4-ib1.bluevela.rmf.ibm.com  perfmon

16   mgmt-r02-m5-ib1.bluevela.rmf.ibm.com  10.127.49.5    mgmt-r02-m5-ib1.bluevela.rmf.ibm.com  perfmon

3.    The PCI tuning option PCI_WR_ORDERING has been enabled on all nodes as follows:

Find the MST device name of each adapter, e.g.:

mst start
mst status
/dev/mst/mt4123_pciconf0

Enable PC_WR_ORDERING for each such device, e.g.:

mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1 

<reboot for the change to take effect>

4.    The file system block is 16 MB (file system block size is a choice based on the use case)

5.    The GNR erasure encoded pool uses 8+2P encoding, which is a choice based on the use case (8+2P is the most common erasure encoding option used for the storage configuration). 

6.    The file system created for the io500 run preallocates 500M inodes (--mmcrfs --inode-limit 500M:500M) out of spare NVMe drive space that was not in use for the production file system.  There are no dedicated resources (e.g. servers, or IB adapter links) used on the servers – all resources are shared with production workloads running under LSF.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

IBM Storage Scale servers are designed with power redundancy in mind.


The network switch is a single point of failure (not-dual redundant switches).

We currently don’t have additional suggestions or comments on this reproducibility questionnaire.

The storage subsystem employs IBM Storage Scale RAID to use 8+2 erasure encoding and, for some metadata, replication to protect file system data from hardware component failures.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The io500 run ran concurrently with production workloads on the system, using the same production file system for logging, system libraries, mounted on all the storage cluster nodes and all of the available 768 H100-based Dell XE9680 clients.

A SLURM reservation was placed for 32 Dell XE9680 clients so they were drained from the regular LSF production workloads.  Only 20 of the Dell XE9680 clients are used for the io500 run.
 
Additional configuration options were applied (the above Runtime Environment section captures the mmlsconfig output for the io500 node class client nodes).  The clients ran io500's dataset against a newly created file system that was dedicated for this work, but using IBM Storage Scale System 6000 NVMe drives used for the production file system, using spare space on the NVMe drives that was not allocated to the production file system.  The purpose of using a separate file system for io500's datasets was to allow us to experiment with file system configuration options.  There are no dedicated resources on the servers and the only file system level tuning that was done for io500 was to preallocate 500M inodes before running io500.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching.  The effect of this caching provides similar caching to that obtained using solid state drive caching systems, and the NVMe drives on the servers also provide solid state caching.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

There is no burst buffer.  Caching on the server-side page pool is analogous to how disk drive caching works.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

Very similar scores have been achieved in other runs of the benchmark on the same system.

We have validated that the scores achieved are within the physical capabilities of the system.