DACH Storage Scale System 6000 - 1 BB HDAT preview
IBM Ehningen System · IBM bb7nvme HDAT configuration · Fully Reproducible
System Purpose
Question asked
Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.
These resources are always in use and generally in the same cluster format although the Storage Scale servers may be split into separate building blocks and the client nodes may mount other file systems for different storage servers.
If there’s a need to do a larger config change, like split the servers into separate building blocks, the timeframe to redeploy this cluster in a different configuration is roughly 2-3 days.
The system is available for team members that work on IBM Storage Scale, with the primary users being performance engineers.
Availability
Question asked
Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.
Please describe the availability of the system to users and who are its set of most regular users.
IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.
Anyone can purchase the software or download a limited use version (it is available commercially).
The IBM Strorage Scale product webpage is:
https://www.ibm.com/products/storage-scale
Regarding the specific storage deployment, this submission uses one building block of the IBM Storage Scale System 6000 solution
Details on the the IBM Storage Scale System 6000 are here:
https://www.ibm.com/downloads/documents/us-en/10a99803f5afd8d6
Storage System Software
Question asked
How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.
Can anyone download/purchase this software?
List either product webpage or open-source repo of the exact software used in IO500.
Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).
IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.
Anyone can purchase the Storage Scale software or download a limited use version (it is available commercially).
On top of G/A code, the Storage Scale System 6000 servers are running a development build based on the IBM Storage Scale 6.0.1 code stream, incoporating updates currently planned for the IBM Storage Scale 6.0.2 release (tentatively targeted for 4Q 2026). For the development build that incoporates updates currently planned for the IBM Storage Scale 6.0.2in question, IBM intends to keep a copy of the build available for the purposes of reproducing this io500 submission results.
The product webpage is https://www.ibm.com/products/storage-scale
Runtime Environment
Question asked
State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.
NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.
Several examples include:
- Commands used to set striping information (either for the entire system or for particular directories)
- File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
- Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
The Storage Scale System 6000 servers are running a development build based of the IBM Storage Scale 6.0.2 code stream, in advance of the 6.0.2 G/A. In addition, the Storage Scale System configuration uses Horizontal DAT, which is a prototype/ early version of changes that the IBM Storage Scale Development team are planning for release in the future. These changes are not firmly committed into a development plan yet but can be provided to customers purely for non-production io500 reproduction purposes.
The basic idea behind HDAT is to split all the drives in the Storage Scale System 6000 into two namespaces, one used for an erasure coded GNR pool, and another used for a high performance DAT (Data Acceleration Tier) pool.
In the case we need to reproduce our results, IBM intends to keep this code available for several months for reproduction purposes.
Details on the runtime environment are:
1. The configuration is captured in mmlsconfig below:
mmlsconfig
Configuration data for cluster storage.ess6000:
-----------------------------------------------
clusterName storage.ess6000
clusterId 8861791687003745391
dmapiFileHandleSize 32
tscCmdAllowRemoteConnections no
ccrEnabled yes
sdrNotifyAuthEnabled yes
maxblocksize 16m
[bb7,bb8,bb9]
nspdBufferMemPerQueue 24m
nsdRAIDEventLogToConsole all
nsdRAIDBlockDeviceMaxSectorsKB 0
nsdRAIDBlockDeviceNrRequests 0
nsdRAIDBlockDeviceQueueDepth 0
nsdRAIDBlockDeviceScheduler off
nsdRAIDSmallThreadRatio 2
nsdRAIDDefaultGeneratedFD no
nsdRAIDMasterBufferPoolSize 2G
nsdRAIDSSDPerformanceShortTimeConstant 2500000
nsdRAIDDiskDiagTimeout 130
numaMemoryInterleave yes
panicOnIOHang yes
nsdRAIDClientOnlyChecksum yes
nsdRAIDEnableSpaceMonitor 1
nsdRAIDFastWriteFSMetadataLimit 256k
nsdRAIDEnableMasterBufferRootLG yes
nsdRAIDEnableRGCMRebalanceWeight yes
[bb7]
nsdRAIDPagepoolBaseline 973399941120
[bb8]
nsdRAIDPagepoolBaseline 486345535488
[common]
verbsRdmaCm enable
mmhealth_gpfs_essconfigsyncmonitor_disable yes
mmhealth-gpfs-essconfigsyncmonitor_disable yes
cesSharedRoot /gpfs/bb7ces
disablePrefetchThreadRebalance yes
[bb7,bb8,bb9]
nsdRAIDReconstructAggressiveness 0
[bb8]
nsdRAIDDefaultIoTimeout 50000
[fscc-ess6000-2-a,bb7,bb8,bb9]
backgroundSpaceReclaim Good: 1 10% 60 1% Info: 0 6% 30 1% Warning: 0 3% 10 0% Critical: 0 2% 5 0% OOS: 0 1% 1 0%
[clients]
prefetchPartitions 2
[common]
fgdlTokenBatchAcquire yes
autoCompactDir no
dataShipAssertOnFailure yes
inodePrefetchThreshold 5
randomReadblockThreshold -1
[all16,bb7,bb8,bb9,gpu4,turin,ubuntu]
prefetchPct 50
[common]
subnets 10.200.0.0/16 10.200.0.0/client.all16 10.200.0.0/storage.ess3500
mtsecCheckEnabled 0
mmhealth-network-check_wr_ordering True
disablePersistExpelList yes
autoload no
[fscc-ess6000-2-a,fscc-ess6000-2-b]
autoload yes
[bb7,bb8]
nsdRAIDBufferPoolSizePct 80
[common]
maxMBpS 2048
[all16,bb7,bb8,bb9,gpu4,turin,ubuntu]
maxMBpS 50000
[bb7,bb8,bb9]
pagepoolMaxPhysMemPct 90
[all16,ubuntu]
proactiveReconnect no
[bb7,bb8,client]
logBufferCount 64
[all16,bb7,bb8,gpu4,turin,ubuntu]
fsyncIsGlobal no
[bb7]
maxDirtyBuffers 9664
[bb7,bb8,bb9]
nsdMinWorkerThreads 3842
nsdMaxWorkerThreads 3842
nsdSmallThreadRatio 1
pitWorkerThreadsPerNode 32
[common]
readReplicaPolicy local
[bb8]
allowFullblockRead 0
[bb9]
nsdRAIDThreadsPerQueue 16
ignorePrefetchLUNCount yes
nsdRAIDPagepoolBaseline 973394042880
[bb8]
verbsRdmasPerConnection 56
verbsRdmasPerConnectionOverride 56
[common]
cesCidrPool <IP=10.200.3.45><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.46><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.47><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.48><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.40><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.41><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.42><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.43><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.44><ATTRIBUTE=><GROUP=><PREFIX=>
[all16,bb8]
aioWorkerThreads 1024
[bb7,bb8]
verbsPorts mlx5_0/1/77 mlx5_1/1/78 mlx5_2/1/77 mlx5_3/1/78 mlx5_4/1/77 mlx5_5/1/78 mlx5_6/1/77 mlx5_7/1/78
[gpu4]
verbsPorts mlx5_0/1/77 mlx5_1/1/77 mlx5_2/1/77 mlx5_3/1/77 mlx5_0/1/78 mlx5_1/1/78 mlx5_2/1/78 mlx5_3/1/78
[all16]
verbsPorts mlx5_0/1/77 mlx5_0/1/78 mlx5_0/1/79 mlx5_1/1/77 mlx5_1/1/78 mlx5_0/1/79
[fscc-sr650-54,fscc-sr655v3-13]
verbsPorts mlx5_0/1/77 mlx5_0/1/78 mlx5_1/1/79
[ubuntu]
verbsPorts mlx5_0/1/77 mlx5_0/1/78
[bb9]
verbsPorts mlx5_0/1/77 mlx5_1/1/77 mlx5_2/1/77 mlx5_3/1/77 mlx5_0/1/78 mlx5_1/1/78 mlx5_2/1/78 mlx5_3/1/78
[turin]
verbsPorts mlx5_0/1/77 mlx5_0/1/78
[common]
verbsRdmaSend yes
[bb8]
nsdRAIDMaxPdiskQueueDepth 500000
[common]
maxTcpConnsPerNodeConn 2
[fscc-sr655v3-20]
maxTcpConnsPerNodeConn 3
tscPrioSmallMsgConnections 7
tscSmallMsgSizeThreshold 262144
tscSmallMsgTcpConnPct 34
[common]
preferDesignatedMnode yes
disableExpelHistory 1
[bb7]
pagepool 973399941120
[bb8]
pagepool 486345535488
[gpu4]
pagepool 50G
[fscc-sr675v3-1]
pagepool 100g
[bb9]
pagepool 973394042880
[clients,ubuntu]
pagepool 100G
[all16,bb8,ubuntu]
tscLostMessageRetry 3
[bb8]
verbsRecvBufferCount 64
[bb7,bb9]
nsdRAIDTracks 320K
[bb8]
nsdRAIDTracks 480K
[common]
mmapRangeLock yes
minReleaseLevel 6.0.1.0
workerThreads 1k
verbsNumRecvChannel 2
verbsNumSendChannel 2
[fscc-ess6000-2-a,bb8,clients]
nBucketGroups 1024
[gpu4]
nBucketGroups 1k
[bb7]
nBucketGroups 128
[bb7,bb8]
nsdRAIDDesiredThreadPct 100
[common]
nspdQueues 240
cipherList AUTHONLY
verbsRdmaRoCEToS 106
[all16,bb9,gpu4,turin,ubuntu]
dataShipClientBufferPct 50
dataShipClientBuffersPerServer 50
[bb7,bb8,bb9]
dataShipServerBufferPct 50
[common]
kernelLockType 0
[bb8,clients]
maxBufferDescs 10000000
[bb7]
maxBufferDescs 2M
[gpu4]
maxBufferDescs 1M
[bb9]
maxBufferDescs 10000000
[bb7,bb8,bb9]
nspdThreadsPerQueue 2
[bb8]
verbsRdmaEnableSendQueue no
[common]
verbsRdma enable
verbsRdmaUseIPv6LinkLocal yes
[test]
verbsRdmaUseIPv6LinkLocal no
[bb8]
ioHistorySize 0
[bb7]
nsdRAIDAllowTraditionalNSD no
[bb7,bb8,bb9,gpu4]
maxFilesToCache 128k
[ces]
maxFilesToCache 2M
[clients]
maxFilesToCache 131072
[turin]
maxFilesToCache 128k
[bb7,bb8,bb9,gpu4]
maxStatCache 128k
[ces]
maxStatCache 2M
[clients]
maxStatCache 131072
[turin]
maxStatCache 128k
[clients]
prefetchMaxThreadsPerOpenFile 12
[turin]
prefetchMaxThreadsPerOpenFile 40
[clients]
maxPrefetchInstDelay 160000
[gpu4]
maxPrefetchInstDelay 300
[common]
trace all 0
[bb8]
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3 fsckx 3 iolite 0
tracedevTimeFormat absolute
[common]
traceRecycle off
[bb8]
tracedevOverwriteBufferSize 2147483648
tracedevWriteMode overwrite
[fscc-ess6000-3-b,clients]
bdevCaching yes
[common]
swapdInstances 0
adminMode central
File systems in cluster storage.ess6000:
----------------------------------------
/dev/bb7nvme16m
/dev/bb8nvme16m
/dev/bb9nvme16m
/dev/nvme
Using the ‘mmchnode’ all nodes (clients and servers) have been defined as ‘manager’ role nodes as shown in ‘mmlscluster’ data below:
mmlscluster
GPFS cluster information
========================
GPFS cluster name: storage.ess6000
GPFS cluster id: 8861791687003745391
GPFS UID domain: storage.ess6000
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR
Node Daemon node name IP address Admin node name Designation
--------------------------------------------------------------------
1 fscc-ess6000-2-a 9.87.92.114 fscc-ess6000-2-a manager-perfmon
2 fscc-ess6000-2-b 9.87.92.116 fscc-ess6000-2-b manager-perfmon
3 fscc-ess6000-3-a 9.87.93.171 fscc-ess6000-3-a manager-perfmon
4 fscc-ess6000-3-b 9.87.93.173 fscc-ess6000-3-b manager-perfmon
5 fscc-un002vm 9.87.95.209 fscc-un002vm perfmon
7 fscc-sr655v3-18 9.87.92.137 fscc-sr655v3-18 manager-perfmon
8 fscc-sr655v3-19 9.87.92.193 fscc-sr655v3-19 manager-perfmon
9 fscc-sr655v3-20 9.87.92.195 fscc-sr655v3-20 manager-perfmon
10 fscc-sr655v3-21 9.87.92.197 fscc-sr655v3-21 manager-perfmon
11 fscc-sr655v3-22 9.87.92.199 fscc-sr655v3-22 manager-perfmon
12 fscc-sr655v3-23 9.87.92.201 fscc-sr655v3-23 manager-perfmon
13 fscc-sr655v3-24 9.87.92.203 fscc-sr655v3-24 manager-perfmon
14 fscc-sr655v3-25 9.87.92.236 fscc-sr655v3-25 manager-perfmon
15 fscc-sr655v3-26 9.87.92.250 fscc-sr655v3-26 manager-perfmon
16 fscc-sr655v3-27 9.87.92.75 fscc-sr655v3-27 manager-perfmon
17 fscc-sr655v3-28 9.87.92.77 fscc-sr655v3-28 manager-perfmon
18 fscc-sr655v3-29 9.87.93.164 fscc-sr655v3-29 manager-perfmon
19 fscc-sr655v3-30 9.87.93.204 fscc-sr655v3-30 manager-perfmon
20 fscc-sr655v3-31 9.87.93.31 fscc-sr655v3-31 manager-perfmon
21 fscc-sr655v3-32 9.87.93.68 fscc-sr655v3-32 manager-perfmon
22 fscc-sr655v3-1 9.87.92.26 fscc-sr655v3-1 manager-perfmon
23 fscc-sr655v3-2 9.87.92.28 fscc-sr655v3-2 manager-perfmon
24 fscc-sr655v3-3 9.87.92.30 fscc-sr655v3-3 manager-perfmon
25 fscc-sr655v3-4 9.87.92.32 fscc-sr655v3-4 manager-perfmon
26 fscc-sr655v3-5 9.87.92.34 fscc-sr655v3-5 manager-perfmon
27 fscc-sr655v3-6 9.87.92.46 fscc-sr655v3-6 manager-perfmon
28 fscc-sr655v3-7 9.87.92.48 fscc-sr655v3-7 manager-perfmon
29 fscc-sr655v3-8 9.87.92.50 fscc-sr655v3-8 manager-perfmon
30 fscc-sr655v3-9 9.87.92.52 fscc-sr655v3-9 manager-perfmon
31 fscc-sr655v3-10 9.87.92.55 fscc-sr655v3-10 manager-perfmon
32 fscc-sr655v3-11 9.87.92.57 fscc-sr655v3-11 manager-perfmon
33 fscc-sr655v3-12 9.87.92.59 fscc-sr655v3-12 manager-perfmon
34 fscc-sr655v3-13 9.87.92.61 fscc-sr655v3-13 manager-perfmon
35 fscc-sr655v3-14 9.87.94.201 fscc-sr655v3-14 manager-perfmon
36 fscc-sr655v3-15 9.87.94.203 fscc-sr655v3-15 manager-perfmon
37 fscc-sr655v3-16 9.87.94.205 fscc-sr655v3-16 manager-perfmon
40 fscc-sr650-54 9.87.95.80 fscc-sr650-54 quorum-perfmon
73 fscc-sr655v3-33 9.87.92.128 fscc-sr655v3-33 manager-perfmon
74 fscc-sr655v3-34 9.87.92.208 fscc-sr655v3-34 manager-perfmon
75 fscc-sr655v3-35 9.87.92.232 fscc-sr655v3-35 manager-perfmon
76 fscc-sr655v3-36 9.87.93.106 fscc-sr655v3-36 manager-perfmon
77 fscc-sr655v3-37 9.87.93.112 fscc-sr655v3-37 manager-perfmon
78 fscc-sr655v3-38 9.87.93.114 fscc-sr655v3-38 manager-perfmon
79 fscc-sr655v3-39 9.87.93.116 fscc-sr655v3-39 manager-perfmon
80 fscc-sr655v3-40 9.87.93.208 fscc-sr655v3-40 manager-perfmon
81 fscc-sr655v3-41 9.87.94.131 fscc-sr655v3-41 manager-perfmon
82 fscc-sr655v3-42 9.87.94.138 fscc-sr655v3-42 manager-perfmon
83 fscc-sr655v3-43 9.87.94.218 fscc-sr655v3-43 manager-perfmon
84 fscc-sr655v3-44 9.87.92.24 fscc-sr655v3-44 manager-perfmon
85 fscc-sr655v3-45 9.87.92.78 fscc-sr655v3-45 manager-perfmon
86 fscc-sr655v3-46 9.87.92.80 fscc-sr655v3-46 manager-perfmon
87 fscc-sr655v3-47 9.87.92.82 fscc-sr655v3-47 manager-perfmon
88 fscc-sr655v3-48 9.87.92.87 fscc-sr655v3-48 manager-perfmon
89 fscc-arm-07 9.87.95.123 fscc-arm-07
91 fscc-ess6000-4-a 9.87.92.188 fscc-ess6000-4-a manager-perfmon
92 fscc-ess6000-4-b 9.87.92.190 fscc-ess6000-4-b manager-perfmon
95 fscc-sr655v3-17 9.87.92.101 fscc-sr655v3-17 manager
2. In this version of the Storage Scale Server The PCI tuning option PCI_WR_ORDERING has been enabled in the ESS code.
The client nodes have manually enabled this setting using this process:
Find the MST device name of each adapter, e.g.:
mst start
mst status
/dev/mst/mt4123_pciconf0
Enable PC_WR_ORDERING for each such device, e.g.:
mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1
<reboot for the change
Fault Tolerance Mechanisms
Question asked
Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.
- Power (For example, can power failure of a node or rack bring down the storage system?)
- Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
- Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
- Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)
Please list any additional information needed to determine whether this system has a single point of failure.
IBM Storage Scale servers are designed with power redundancy in mind.
The network switch is a single point of failure (not-dual redundant switches) but Storage Scale supports such redundant switch configurations.
This run uses a storage subsystem employing IBM Storage Scale RAID with a 16+2P erasure encoding for data and metadata.
Execution
Question asked
Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.
During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?
./io500.sh config_withFGWS_PFIND_FGRS.bb7nvme16m.ini 36 > io500.out 2>&1
The entire system was dedicated to the io500 run.
Caching
Question asked
Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).
A few examples would include:
- Client data/metadata caching (in Linux page cache or in other memory cache)
- Client side NVMe read-only data cache
- Storage server metadata/data caching in RAM
- Storage controller caching
- RAID card caching
On the client side, the Storage Scale pagepool (in RAM) is used to cache data in the same manner that the Linux file cache is used to cache data with other file systems such as ext4. Data is not cached on NVMe drives on the clients as those client drives are not used in the I/O path by the Storage Scale file system.
The IBM Storage Scale DAT (Data Acceleration Tier) technology is used here but this is a replication feature, not caching.
On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching. The effect of this caching provides similar caching to that obtained using solid state drive caching systems, and the NVMe drives on the servers also provide solid state caching.
Data Source
Question asked
Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.
It depends on the definition of 'burst buffer' but, while the system includes two distinct storage pools (the erasure coded pool and the high performance NVMe-oF accessed DAT pool), both are fully integrated within the same coherent namespace. All writes are immediately visible across all nodes; there is no eventual-consistency or staging mechanism.
This design differs from “burst buffer” systems described in On the Role of Burst Buffers in Leadership-Class Storage Systems (MCS ANL, 2012), which define burst buffers as “low-capacity, high-throughput staging layers that absorb short-term I/O bursts”.
The DAT performance pool exists on the IBM Storage Scale System 6000 drives, and can hold large amounts of data indefinitely, but for protection against local disk failures, data replication or migration to the GNR pool is recommended. That's one reason the DAT pool has been referred to as a burst buffer when used with a single data replica only in the DAT pool.
Trust
Question asked
Please describe any steps taken to ensure that the results are trustworthy.
- Did you run the benchmark multiple times and get similar scores?
- Did you validate the score is below the physical capabilities of the deployed hardware?
- How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?
Very similar scores have been achieved in several other runs of the benchmark on the same system.
We have validated that the scores achieved are within the physical capabilities of the system.