DACH Storage Scale System 6000 - 1 BB HDAT preview

IBM Ehningen System · IBM bb7nvme HDAT configuration · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

These resources are always in use and generally in the same cluster format although the Storage Scale servers may be split into separate building blocks and the client nodes may mount other file systems for different storage servers. 

If there’s a need to do a larger config change, like split the servers into separate building blocks, the timeframe to redeploy this cluster in a different configuration is roughly 2-3 days.

The system is available for team members that work on IBM Storage Scale, with the primary users being performance engineers.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.

Anyone can purchase the software or download a limited use version (it is available commercially).

The IBM Strorage Scale product webpage is:
https://www.ibm.com/products/storage-scale


Regarding the specific storage deployment, this submission uses one building block of the IBM Storage Scale System 6000 solution

Details on the the IBM Storage Scale System 6000 are here:

https://www.ibm.com/downloads/documents/us-en/10a99803f5afd8d6

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available. 

Anyone can purchase the Storage Scale software or download a limited use version (it is available commercially).  

On top of G/A code, the Storage Scale System 6000 servers are running a development build based on the IBM Storage Scale 6.0.1 code stream, incoporating updates currently planned for the IBM Storage Scale 6.0.2 release (tentatively targeted for 4Q 2026).   For the development build that incoporates updates currently planned for the IBM Storage Scale 6.0.2in question, IBM intends to keep a copy of the build available for the purposes of reproducing this io500 submission results.

The product webpage is https://www.ibm.com/products/storage-scale

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

The Storage Scale System 6000 servers are running a development build based of the IBM Storage Scale 6.0.2 code stream, in advance of the 6.0.2 G/A.  In addition, the Storage Scale System configuration uses Horizontal DAT, which is a prototype/ early version of changes that the IBM Storage Scale Development team are planning for release in the future.  These changes are not firmly committed into a development plan yet but can be provided to customers purely for non-production io500 reproduction purposes. 

The basic idea behind HDAT is to split all the drives in the Storage Scale System 6000 into two namespaces, one used for an erasure coded GNR pool, and another used for a high performance DAT (Data Acceleration Tier) pool.

In the case we need to reproduce our results, IBM intends to keep this code available for several months for reproduction purposes.

Details on the runtime environment are:

1.    The configuration is captured in mmlsconfig below:

mmlsconfig

Configuration data for cluster storage.ess6000:

-----------------------------------------------

clusterName storage.ess6000

clusterId 8861791687003745391

dmapiFileHandleSize 32

tscCmdAllowRemoteConnections no

ccrEnabled yes

sdrNotifyAuthEnabled yes

maxblocksize 16m

[bb7,bb8,bb9]

nspdBufferMemPerQueue 24m

nsdRAIDEventLogToConsole all

nsdRAIDBlockDeviceMaxSectorsKB 0

nsdRAIDBlockDeviceNrRequests 0

nsdRAIDBlockDeviceQueueDepth 0

nsdRAIDBlockDeviceScheduler off

nsdRAIDSmallThreadRatio 2

nsdRAIDDefaultGeneratedFD no

nsdRAIDMasterBufferPoolSize 2G

nsdRAIDSSDPerformanceShortTimeConstant 2500000

nsdRAIDDiskDiagTimeout 130

numaMemoryInterleave yes

panicOnIOHang yes

nsdRAIDClientOnlyChecksum yes

nsdRAIDEnableSpaceMonitor 1

nsdRAIDFastWriteFSMetadataLimit 256k

nsdRAIDEnableMasterBufferRootLG yes

nsdRAIDEnableRGCMRebalanceWeight yes

[bb7]

nsdRAIDPagepoolBaseline 973399941120

[bb8]

nsdRAIDPagepoolBaseline 486345535488

[common]

verbsRdmaCm enable

mmhealth_gpfs_essconfigsyncmonitor_disable yes

mmhealth-gpfs-essconfigsyncmonitor_disable yes

cesSharedRoot /gpfs/bb7ces

disablePrefetchThreadRebalance yes

[bb7,bb8,bb9]

nsdRAIDReconstructAggressiveness 0

[bb8]

nsdRAIDDefaultIoTimeout 50000

[fscc-ess6000-2-a,bb7,bb8,bb9]

backgroundSpaceReclaim Good: 1 10% 60 1% Info: 0 6% 30 1% Warning: 0 3% 10 0% Critical: 0 2% 5 0% OOS: 0 1% 1 0%

[clients]

prefetchPartitions 2

[common]

fgdlTokenBatchAcquire yes

autoCompactDir no

dataShipAssertOnFailure yes

inodePrefetchThreshold 5

randomReadblockThreshold -1

[all16,bb7,bb8,bb9,gpu4,turin,ubuntu]

prefetchPct 50

[common]

subnets 10.200.0.0/16 10.200.0.0/client.all16 10.200.0.0/storage.ess3500

mtsecCheckEnabled 0

mmhealth-network-check_wr_ordering True

disablePersistExpelList yes

autoload no

[fscc-ess6000-2-a,fscc-ess6000-2-b]

autoload yes

[bb7,bb8]

nsdRAIDBufferPoolSizePct 80

[common]

maxMBpS 2048

[all16,bb7,bb8,bb9,gpu4,turin,ubuntu]

maxMBpS 50000

[bb7,bb8,bb9]

pagepoolMaxPhysMemPct 90

[all16,ubuntu]

proactiveReconnect no

[bb7,bb8,client]

logBufferCount 64

[all16,bb7,bb8,gpu4,turin,ubuntu]

fsyncIsGlobal no

[bb7]

maxDirtyBuffers 9664

[bb7,bb8,bb9]

nsdMinWorkerThreads 3842

nsdMaxWorkerThreads 3842

nsdSmallThreadRatio 1

pitWorkerThreadsPerNode 32

[common]

readReplicaPolicy local

[bb8]

allowFullblockRead 0

[bb9]

nsdRAIDThreadsPerQueue 16

ignorePrefetchLUNCount yes

nsdRAIDPagepoolBaseline 973394042880

[bb8]

verbsRdmasPerConnection 56

verbsRdmasPerConnectionOverride 56

[common]

cesCidrPool <IP=10.200.3.45><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.46><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.47><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.48><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.40><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.41><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.42><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.43><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.44><ATTRIBUTE=><GROUP=><PREFIX=>

[all16,bb8]

aioWorkerThreads 1024

[bb7,bb8]

verbsPorts mlx5_0/1/77 mlx5_1/1/78 mlx5_2/1/77 mlx5_3/1/78 mlx5_4/1/77 mlx5_5/1/78 mlx5_6/1/77 mlx5_7/1/78

[gpu4]

verbsPorts mlx5_0/1/77 mlx5_1/1/77 mlx5_2/1/77 mlx5_3/1/77 mlx5_0/1/78 mlx5_1/1/78 mlx5_2/1/78 mlx5_3/1/78

[all16]

verbsPorts mlx5_0/1/77 mlx5_0/1/78 mlx5_0/1/79 mlx5_1/1/77 mlx5_1/1/78 mlx5_0/1/79

[fscc-sr650-54,fscc-sr655v3-13]

verbsPorts mlx5_0/1/77 mlx5_0/1/78 mlx5_1/1/79

[ubuntu]

verbsPorts mlx5_0/1/77 mlx5_0/1/78

[bb9]

verbsPorts mlx5_0/1/77 mlx5_1/1/77 mlx5_2/1/77 mlx5_3/1/77 mlx5_0/1/78 mlx5_1/1/78 mlx5_2/1/78 mlx5_3/1/78

[turin]

verbsPorts mlx5_0/1/77 mlx5_0/1/78

[common]

verbsRdmaSend yes

[bb8]

nsdRAIDMaxPdiskQueueDepth 500000

[common]

maxTcpConnsPerNodeConn 2

[fscc-sr655v3-20]

maxTcpConnsPerNodeConn 3

tscPrioSmallMsgConnections 7

tscSmallMsgSizeThreshold 262144

tscSmallMsgTcpConnPct 34

[common]

preferDesignatedMnode yes

disableExpelHistory 1

[bb7]

pagepool 973399941120

[bb8]

pagepool 486345535488

[gpu4]

pagepool 50G

[fscc-sr675v3-1]

pagepool 100g

[bb9]

pagepool 973394042880

[clients,ubuntu]

pagepool 100G

[all16,bb8,ubuntu]

tscLostMessageRetry 3

[bb8]

verbsRecvBufferCount 64

[bb7,bb9]

nsdRAIDTracks 320K

[bb8]

nsdRAIDTracks 480K

[common]

mmapRangeLock yes

minReleaseLevel 6.0.1.0

workerThreads 1k

verbsNumRecvChannel 2

verbsNumSendChannel 2

[fscc-ess6000-2-a,bb8,clients]

nBucketGroups 1024

[gpu4]

nBucketGroups 1k

[bb7]

nBucketGroups 128

[bb7,bb8]

nsdRAIDDesiredThreadPct 100

[common]

nspdQueues 240

cipherList AUTHONLY

verbsRdmaRoCEToS 106

[all16,bb9,gpu4,turin,ubuntu]

dataShipClientBufferPct 50

dataShipClientBuffersPerServer 50

[bb7,bb8,bb9]

dataShipServerBufferPct 50

[common]

kernelLockType 0

[bb8,clients]

maxBufferDescs 10000000

[bb7]

maxBufferDescs 2M

[gpu4]

maxBufferDescs 1M

[bb9]

maxBufferDescs 10000000

[bb7,bb8,bb9]

nspdThreadsPerQueue 2

[bb8]

verbsRdmaEnableSendQueue no

[common]

verbsRdma enable

verbsRdmaUseIPv6LinkLocal yes

[test]

verbsRdmaUseIPv6LinkLocal no

[bb8]

ioHistorySize 0

[bb7]

nsdRAIDAllowTraditionalNSD no

[bb7,bb8,bb9,gpu4]

maxFilesToCache 128k

[ces]

maxFilesToCache 2M

[clients]

maxFilesToCache 131072

[turin]

maxFilesToCache 128k

[bb7,bb8,bb9,gpu4]

maxStatCache 128k

[ces]

maxStatCache 2M

[clients]

maxStatCache 131072

[turin]

maxStatCache 128k

[clients]

prefetchMaxThreadsPerOpenFile 12

[turin]

prefetchMaxThreadsPerOpenFile 40

[clients]

maxPrefetchInstDelay 160000

[gpu4]

maxPrefetchInstDelay 300

[common]

trace all 0

[bb8]

trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3 fsckx 3 iolite 0

tracedevTimeFormat absolute

[common]

traceRecycle off

[bb8]

tracedevOverwriteBufferSize 2147483648

tracedevWriteMode overwrite

[fscc-ess6000-3-b,clients]

bdevCaching yes

[common]

swapdInstances 0

adminMode central

File systems in cluster storage.ess6000:

----------------------------------------

/dev/bb7nvme16m

/dev/bb8nvme16m

/dev/bb9nvme16m

/dev/nvme

Using the ‘mmchnode’ all nodes (clients and servers) have been defined as ‘manager’ role nodes as shown in ‘mmlscluster’ data below:

mmlscluster

GPFS cluster information

========================

GPFS cluster name:         storage.ess6000

GPFS cluster id:           8861791687003745391

GPFS UID domain:           storage.ess6000

Remote shell command:      /usr/bin/ssh

Remote file copy command:  /usr/bin/scp

Repository type:           CCR

Node  Daemon node name  IP address   Admin node name   Designation

--------------------------------------------------------------------

1   fscc-ess6000-2-a  9.87.92.114  fscc-ess6000-2-a  manager-perfmon

2   fscc-ess6000-2-b  9.87.92.116  fscc-ess6000-2-b  manager-perfmon

3   fscc-ess6000-3-a  9.87.93.171  fscc-ess6000-3-a  manager-perfmon

4   fscc-ess6000-3-b  9.87.93.173  fscc-ess6000-3-b  manager-perfmon

5   fscc-un002vm      9.87.95.209  fscc-un002vm      perfmon

7   fscc-sr655v3-18   9.87.92.137  fscc-sr655v3-18   manager-perfmon

8   fscc-sr655v3-19   9.87.92.193  fscc-sr655v3-19   manager-perfmon

9   fscc-sr655v3-20   9.87.92.195  fscc-sr655v3-20   manager-perfmon

10   fscc-sr655v3-21   9.87.92.197  fscc-sr655v3-21   manager-perfmon

11   fscc-sr655v3-22   9.87.92.199  fscc-sr655v3-22   manager-perfmon
  12   fscc-sr655v3-23   9.87.92.201  fscc-sr655v3-23   manager-perfmon

13   fscc-sr655v3-24   9.87.92.203  fscc-sr655v3-24   manager-perfmon

14   fscc-sr655v3-25   9.87.92.236  fscc-sr655v3-25   manager-perfmon

15   fscc-sr655v3-26   9.87.92.250  fscc-sr655v3-26   manager-perfmon

16   fscc-sr655v3-27   9.87.92.75   fscc-sr655v3-27   manager-perfmon

17   fscc-sr655v3-28   9.87.92.77   fscc-sr655v3-28   manager-perfmon

18   fscc-sr655v3-29   9.87.93.164  fscc-sr655v3-29   manager-perfmon

19   fscc-sr655v3-30   9.87.93.204  fscc-sr655v3-30   manager-perfmon

20   fscc-sr655v3-31   9.87.93.31   fscc-sr655v3-31   manager-perfmon

21   fscc-sr655v3-32   9.87.93.68   fscc-sr655v3-32   manager-perfmon

22   fscc-sr655v3-1    9.87.92.26   fscc-sr655v3-1    manager-perfmon

23   fscc-sr655v3-2    9.87.92.28   fscc-sr655v3-2    manager-perfmon

24   fscc-sr655v3-3    9.87.92.30   fscc-sr655v3-3    manager-perfmon

25   fscc-sr655v3-4    9.87.92.32   fscc-sr655v3-4    manager-perfmon

26   fscc-sr655v3-5    9.87.92.34   fscc-sr655v3-5    manager-perfmon

27   fscc-sr655v3-6    9.87.92.46   fscc-sr655v3-6    manager-perfmon

28   fscc-sr655v3-7    9.87.92.48   fscc-sr655v3-7    manager-perfmon

29   fscc-sr655v3-8    9.87.92.50   fscc-sr655v3-8    manager-perfmon

30   fscc-sr655v3-9    9.87.92.52   fscc-sr655v3-9    manager-perfmon

31   fscc-sr655v3-10   9.87.92.55   fscc-sr655v3-10   manager-perfmon

32   fscc-sr655v3-11   9.87.92.57   fscc-sr655v3-11   manager-perfmon

33   fscc-sr655v3-12   9.87.92.59   fscc-sr655v3-12   manager-perfmon

34   fscc-sr655v3-13   9.87.92.61   fscc-sr655v3-13   manager-perfmon

35   fscc-sr655v3-14   9.87.94.201  fscc-sr655v3-14   manager-perfmon

36   fscc-sr655v3-15   9.87.94.203  fscc-sr655v3-15   manager-perfmon

37   fscc-sr655v3-16   9.87.94.205  fscc-sr655v3-16   manager-perfmon

40   fscc-sr650-54     9.87.95.80   fscc-sr650-54     quorum-perfmon

73   fscc-sr655v3-33   9.87.92.128  fscc-sr655v3-33   manager-perfmon

74   fscc-sr655v3-34   9.87.92.208  fscc-sr655v3-34   manager-perfmon

75   fscc-sr655v3-35   9.87.92.232  fscc-sr655v3-35   manager-perfmon

76   fscc-sr655v3-36   9.87.93.106  fscc-sr655v3-36   manager-perfmon

77   fscc-sr655v3-37   9.87.93.112  fscc-sr655v3-37   manager-perfmon

78   fscc-sr655v3-38   9.87.93.114  fscc-sr655v3-38   manager-perfmon

79   fscc-sr655v3-39   9.87.93.116  fscc-sr655v3-39   manager-perfmon

80   fscc-sr655v3-40   9.87.93.208  fscc-sr655v3-40   manager-perfmon

81   fscc-sr655v3-41   9.87.94.131  fscc-sr655v3-41   manager-perfmon

82   fscc-sr655v3-42   9.87.94.138  fscc-sr655v3-42   manager-perfmon

83   fscc-sr655v3-43   9.87.94.218  fscc-sr655v3-43   manager-perfmon

84   fscc-sr655v3-44   9.87.92.24   fscc-sr655v3-44   manager-perfmon

85   fscc-sr655v3-45   9.87.92.78   fscc-sr655v3-45   manager-perfmon

86   fscc-sr655v3-46   9.87.92.80   fscc-sr655v3-46   manager-perfmon

87   fscc-sr655v3-47   9.87.92.82   fscc-sr655v3-47   manager-perfmon

88   fscc-sr655v3-48   9.87.92.87   fscc-sr655v3-48   manager-perfmon

89   fscc-arm-07       9.87.95.123  fscc-arm-07

91   fscc-ess6000-4-a  9.87.92.188  fscc-ess6000-4-a  manager-perfmon

92   fscc-ess6000-4-b  9.87.92.190  fscc-ess6000-4-b  manager-perfmon

95   fscc-sr655v3-17   9.87.92.101  fscc-sr655v3-17   manager

2.    In this version of the Storage Scale Server The PCI tuning option PCI_WR_ORDERING has been enabled in the ESS code.

The client nodes have manually enabled this setting using this process:
 
Find the MST device name of each adapter, e.g.:

mst start
mst status
/dev/mst/mt4123_pciconf0

Enable PC_WR_ORDERING for each such device, e.g.:

mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1 
<reboot for the change

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

IBM Storage Scale servers are designed with power redundancy in mind.


The network switch is a single point of failure (not-dual redundant switches) but Storage Scale supports such redundant switch configurations.


This run uses a storage subsystem employing IBM Storage Scale RAID with a 16+2P erasure encoding for data and metadata.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

./io500.sh config_withFGWS_PFIND_FGRS.bb7nvme16m.ini 36 > io500.out 2>&1

The entire system was dedicated to the io500 run.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

On the client side, the Storage Scale pagepool (in RAM) is used to cache data in the same manner that the Linux file cache is used to cache data with other file systems such as ext4.  Data is not cached on NVMe drives on the clients as those client drives are not used in the I/O path by the Storage Scale file system.

The IBM Storage Scale DAT (Data Acceleration Tier) technology is used here but this is a replication feature, not caching.

On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching.  The effect of this caching provides similar caching to that obtained using solid state drive caching systems, and the NVMe drives on the servers also provide solid state caching.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

It depends on the definition of 'burst buffer' but, while the system includes two distinct storage pools (the erasure coded  pool and the high performance NVMe-oF accessed DAT pool), both are fully integrated within the same coherent namespace. All writes are immediately visible across all nodes; there is no eventual-consistency or staging mechanism. 

This design differs from “burst buffer” systems described in On the Role of Burst Buffers in Leadership-Class Storage Systems (MCS ANL, 2012), which define burst buffers as “low-capacity, high-throughput staging layers that absorb short-term I/O bursts”.

The DAT performance pool exists on the IBM Storage Scale System 6000 drives, and can hold large amounts of data indefinitely, but for protection against local disk failures, data replication or migration to the GNR pool is recommended.  That's one reason the DAT pool has been referred to as a burst buffer when used with a single data replica only in the DAT pool.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

Very similar scores have been achieved in several other runs of the benchmark on the same system.

We have validated that the scores achieved are within the physical capabilities of the system.