DACH Storage Scale System 6000 - 2 BBs

The system is primarily used for performance engineering, test, and measurements/benchmarking.  An example of performance engineering work would be debugging customer reported performance issues. 

These resources are always in use and generally in the same cluster format although the Storage Scale servers may be connected in different manners (e.g. multiple building blocks of storage and the client nodes may mount other file systems for different storage servers).   Various configurations of clients and servers typically last in the range of days to weeks before reconfiguration.

The system is available for team members that work on IBM Storage Scale, with the primary users being performance engineers.

IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available. 

Anyone can purchase the Storage Scale software or download a limited use version (it is available commercially).  

On top of G/A code, the Storage Scale System 6000 servers are running a development build based on the IBM Storage Scale 6.0.1 code stream, incoporating updates currently planned for the IBM Storage Scale 6.0.2 release (tentatively targeted for 4Q 2026).   For the development build that incoporates updates currently planned for the IBM Storage Scale 6.0.2in question, IBM intends to keep a copy of the build available for the purposes of reproducing this io500 submission results.

The product webpage is https://www.ibm.com/products/storage-scale

As previously noted the Storage Scale System 6000 servers are running a development build based on the IBM Storage Scale 6.0.1 code stream, incoporating updates currently planned for the IBM Storage Scale 6.0.2 release (tentatively targeted for 4Q 2026).  In the case we need to reproduce our results, IBM intends to keep this code available for several months for reproduction purposes.

Details on the runtime environment are:

1.    The configuration is captured in mmlsconfig below:

mmlsconfig

Configuration data for cluster storage.ess6000:

-----------------------------------------------

clusterName storage.ess6000

clusterId 8861791687003745391

dmapiFileHandleSize 32

minReleaseLevel 6.0.0.0

tscCmdAllowRemoteConnections no

ccrEnabled yes

sdrNotifyAuthEnabled yes

maxblocksize 16m

[bb7]

nspdQueues 2045

[bb7,bb8,bb9]

nspdBufferMemPerQueue 24m

nspdThreadsPerQueue 2

nsdMinWorkerThreads 3842

nsdMaxWorkerThreads 3842

nsdRAIDEventLogToConsole all

nsdRAIDBlockDeviceMaxSectorsKB 0

nsdRAIDBlockDeviceNrRequests 0

nsdRAIDBlockDeviceQueueDepth 0

nsdRAIDBlockDeviceScheduler off

nsdRAIDSmallThreadRatio 2

nsdRAIDDefaultGeneratedFD no

nsdRAIDMasterBufferPoolSize 2G

nsdRAIDSSDPerformanceShortTimeConstant 2500000

nsdRAIDDiskDiagTimeout 130

numaMemoryInterleave yes

panicOnIOHang yes

pitWorkerThreadsPerNode 32

nsdRAIDClientOnlyChecksum yes

nsdRAIDEnableSpaceMonitor 1

nsdRAIDFastWriteFSMetadataLimit 256k

nsdRAIDMaxPdiskQueueDepth default:0;hdd:128;ssd:0;nvme:0;nvdimm:0;nvram:0

nsdRAIDEnableMasterBufferRootLG yes

nsdRAIDEnableRGCMRebalanceWeight yes

pagepoolMaxPhysMemPct 90

[bb7]

nsdRAIDPagepoolBaseline 973399941120

[bb8,bb9]

nspdQueues 240

[bb8]

nsdRAIDPagepoolBaseline 486345535488

[all16,ubuntu]

proactiveReconnect no

[common]

verbsRdmaCm enable

allToAllConnection all

[all16,bb7,bb8,bb9,turin,ubuntu]

prefetchPct 50

[all16,turin,ubuntu]

dataShipClientBufferPct 50

dataShipClientBuffersPerServer 50

[common]

mmhealth_gpfs_essconfigsyncmonitor_disable yes

mmhealth-gpfs-essconfigsyncmonitor_disable yes

[bb7]

maxReceiverThreadsOverride yes

maxReceiverThreads 96

[common]

cesSharedRoot /gpfs/bb7ces

cesCidrPool <IP=10.200.3.45><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.46><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.47><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=10.200.3.48><ATTRIBUTE=><GROUP=><PREFIX=>

autoload no

[bb9]

ignorePrefetchLUNCount yes

nsdRAIDPagepoolBaseline 973393426022

[bb7,bb8,bb9]

nsdRAIDTracks 320K

nsdRAIDThreadsPerQueue 16

[common]

disablePrefetchThreadRebalance yes

[bb7,bb8,bb9]

nsdSmallThreadRatio 1

[bb7,bb8,bb9]

nsdRAIDReconstructAggressiveness 0

nsdRAIDDesiredThreadPct 100

nsdRAIDBufferPoolSizePct 80

[bb8]

nsdRAIDDefaultIoTimeout 50000

[fscc-ess6000-2-a,bb7,bb8,bb9]

backgroundSpaceReclaim Good: 1 10% 60 1% Info: 0 6% 30 1% Warning: 0 3% 10 0% Critical: 0 2% 5 0% OOS: 0 1% 1 0%

[common]

readReplicaPolicy fastest

verbsRdmaSend yes

[all16,bb7,bb8,bb9,turin,ubuntu]

maxMBpS 50000

[common]

subnets 10.200.0.0/16 10.200.0.0/client.all16 10.200.0.0/storage.ess3500

cipherList AUTHONLY

verbsRdma enable

maxTcpConnsPerNodeConn 2

[bb7,bb8,bb9]

dataShipServerBufferPct 50

[bb7]

pagepool 973399941120

[bb8]

pagepool 486345535488

[bb9]

pagepool 973393426022

[clients]

pagepool 140G

[bb7,bb8,bb9,client]

logBufferCount 64

[common]

preferDesignatedMnode yes

[clients]

prefetchPartitions 2

[bb7,bb8,bb9,clients]

workerThreads 1024

maxFilesToCache 131072

maxStatCache 131072

[all16,bb7,bb8,bb9,turin,ubuntu]

fsyncIsGlobal no

[bb7,bb8,bb9,clients]

nBucketGroups 1024

[common]

fgdlTokenBatchAcquire yes

autoCompactDir no

dataShipAssertOnFailure yes

[bb7,bb8,bb9,clients]

maxBufferDescs 10000000

[clients]

prefetchMaxThreadsPerOpenFile 12

maxPrefetchInstDelay 160000

[common]

inodePrefetchThreshold 5

[bb8]

traceRecycle local

[bb9,clients]

traceRecycle off

[bb8,bb9,clients]

tracedevOverwriteBufferSize 4294967296

tracedevTimeFormat absolute

tracedevWriteMode overwrite

trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3 fsckx 3 iolite 0

[bb7,bb8]

verbsPorts mlx5_0/1/77 mlx5_1/1/78 mlx5_2/1/77 mlx5_3/1/78 mlx5_4/1/77 mlx5_5/1/78 mlx5_6/1/77 mlx5_7/1/78

[all16]

verbsPorts mlx5_0/1/77 mlx5_0/1/78 mlx5_1/1/77 mlx5_1/1/78

[fscc-sr650-54,fscc-sr655v3-13,turin,ubuntu]

verbsPorts mlx5_0/1/77 mlx5_0/1/78

[bb9]

verbsPorts mlx5_0/1/77 mlx5_1/1/77 mlx5_2/1/77 mlx5_3/1/77 mlx5_0/1/78 mlx5_1/1/78 mlx5_2/1/78 mlx5_3/1/78

[fscc-ess6000-4-a]

verbsPorts mlx5_0/1/77 mlx5_1/1/77 mlx5_2/1/77 mlx5_3/1/99

[fscc-ess6000-4-b]

verbsPorts mlx5_0/1/78 mlx5_1/1/78 mlx5_2/1/78  mlx5_3/1/99

[common]

adminMode central

File systems in cluster storage.ess6000:

----------------------------------------

/dev/bb78nvme16m

Using the ‘mmchnode’ all nodes (clients and servers) have been defined as ‘manager’ role nodes as shown in ‘mmlscluster’ data below:

mmlscluster

GPFS cluster information

========================

GPFS cluster name:         storage.ess6000

GPFS cluster id:           8861791687003745391

GPFS UID domain:           storage.ess6000

Remote shell command:      /usr/bin/ssh

Remote file copy command:  /usr/bin/scp

Repository type:           CCR

Node  Daemon node name  IP address   Admin node name   Designation

--------------------------------------------------------------------

1   fscc-ess6000-2-a  9.87.92.114  fscc-ess6000-2-a  manager-perfmon

2   fscc-ess6000-2-b  9.87.92.116  fscc-ess6000-2-b  manager-perfmon

3   fscc-ess6000-3-a  9.87.93.171  fscc-ess6000-3-a  manager-perfmon

4   fscc-ess6000-3-b  9.87.93.173  fscc-ess6000-3-b  manager-perfmon

5   fscc-un002vm      9.87.95.209  fscc-un002vm      perfmon

7   fscc-sr655v3-18   9.87.92.137  fscc-sr655v3-18   manager-perfmon

8   fscc-sr655v3-19   9.87.92.193  fscc-sr655v3-19   manager-perfmon

9   fscc-sr655v3-20   9.87.92.195  fscc-sr655v3-20   manager-perfmon

10   fscc-sr655v3-21   9.87.92.197  fscc-sr655v3-21   manager-perfmon

11   fscc-sr655v3-22   9.87.92.199  fscc-sr655v3-22   manager-perfmon
  12   fscc-sr655v3-23   9.87.92.201  fscc-sr655v3-23   manager-perfmon

13   fscc-sr655v3-24   9.87.92.203  fscc-sr655v3-24   manager-perfmon

14   fscc-sr655v3-25   9.87.92.236  fscc-sr655v3-25   manager-perfmon

15   fscc-sr655v3-26   9.87.92.250  fscc-sr655v3-26   manager-perfmon

16   fscc-sr655v3-27   9.87.92.75   fscc-sr655v3-27   manager-perfmon

17   fscc-sr655v3-28   9.87.92.77   fscc-sr655v3-28   manager-perfmon

18   fscc-sr655v3-29   9.87.93.164  fscc-sr655v3-29   manager-perfmon

19   fscc-sr655v3-30   9.87.93.204  fscc-sr655v3-30   manager-perfmon

20   fscc-sr655v3-31   9.87.93.31   fscc-sr655v3-31   manager-perfmon

21   fscc-sr655v3-32   9.87.93.68   fscc-sr655v3-32   manager-perfmon

22   fscc-sr655v3-1    9.87.92.26   fscc-sr655v3-1    manager-perfmon

23   fscc-sr655v3-2    9.87.92.28   fscc-sr655v3-2    manager-perfmon

24   fscc-sr655v3-3    9.87.92.30   fscc-sr655v3-3    manager-perfmon

25   fscc-sr655v3-4    9.87.92.32   fscc-sr655v3-4    manager-perfmon

26   fscc-sr655v3-5    9.87.92.34   fscc-sr655v3-5    manager-perfmon

27   fscc-sr655v3-6    9.87.92.46   fscc-sr655v3-6    manager-perfmon

28   fscc-sr655v3-7    9.87.92.48   fscc-sr655v3-7    manager-perfmon

29   fscc-sr655v3-8    9.87.92.50   fscc-sr655v3-8    manager-perfmon

30   fscc-sr655v3-9    9.87.92.52   fscc-sr655v3-9    manager-perfmon

31   fscc-sr655v3-10   9.87.92.55   fscc-sr655v3-10   manager-perfmon

32   fscc-sr655v3-11   9.87.92.57   fscc-sr655v3-11   manager-perfmon

33   fscc-sr655v3-12   9.87.92.59   fscc-sr655v3-12   manager-perfmon

34   fscc-sr655v3-13   9.87.92.61   fscc-sr655v3-13   manager-perfmon

35   fscc-sr655v3-14   9.87.94.201  fscc-sr655v3-14   manager-perfmon

36   fscc-sr655v3-15   9.87.94.203  fscc-sr655v3-15   manager-perfmon

37   fscc-sr655v3-16   9.87.94.205  fscc-sr655v3-16   manager-perfmon

40   fscc-sr650-54     9.87.95.80   fscc-sr650-54     quorum-perfmon

73   fscc-sr655v3-33   9.87.92.128  fscc-sr655v3-33   manager-perfmon

74   fscc-sr655v3-34   9.87.92.208  fscc-sr655v3-34   manager-perfmon

75   fscc-sr655v3-35   9.87.92.232  fscc-sr655v3-35   manager-perfmon

76   fscc-sr655v3-36   9.87.93.106  fscc-sr655v3-36   manager-perfmon

77   fscc-sr655v3-37   9.87.93.112  fscc-sr655v3-37   manager-perfmon

78   fscc-sr655v3-38   9.87.93.114  fscc-sr655v3-38   manager-perfmon

79   fscc-sr655v3-39   9.87.93.116  fscc-sr655v3-39   manager-perfmon

80   fscc-sr655v3-40   9.87.93.208  fscc-sr655v3-40   manager-perfmon

81   fscc-sr655v3-41   9.87.94.131  fscc-sr655v3-41   manager-perfmon

82   fscc-sr655v3-42   9.87.94.138  fscc-sr655v3-42   manager-perfmon

83   fscc-sr655v3-43   9.87.94.218  fscc-sr655v3-43   manager-perfmon

84   fscc-sr655v3-44   9.87.92.24   fscc-sr655v3-44   manager-perfmon

85   fscc-sr655v3-45   9.87.92.78   fscc-sr655v3-45   manager-perfmon

86   fscc-sr655v3-46   9.87.92.80   fscc-sr655v3-46   manager-perfmon

87   fscc-sr655v3-47   9.87.92.82   fscc-sr655v3-47   manager-perfmon

88   fscc-sr655v3-48   9.87.92.87   fscc-sr655v3-48   manager-perfmon

89   fscc-arm-07       9.87.95.123  fscc-arm-07

91   fscc-ess6000-4-a  9.87.92.188  fscc-ess6000-4-a  manager-perfmon

92   fscc-ess6000-4-b  9.87.92.190  fscc-ess6000-4-b  manager-perfmon

95   fscc-sr655v3-17   9.87.92.101  fscc-sr655v3-17   manager

2.    In this version of the Storage Scale Server The PCI tuning option PCI_WR_ORDERING has been enabled in the ESS code.

The client nodes have manually enabled this setting using this process:
 
Find the MST device name of each adapter, e.g.:

mst start
mst status
/dev/mst/mt4123_pciconf0

Enable PC_WR_ORDERING for each such device, e.g.:

mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1 
<reboot for the change to take effect>


The server nodes also have manually set INT_LOG_MAX_PAYLOAD_SIZE to the default/automatic setting as follows:

Find the MST device name of each adapter, e.g.:

mst start
mst status
/dev/mst/mt4123_pciconf0

Set INT_LOG_MAX_PAYLOAD_SIZE for each such device, e.g.:

echo y | mlxconfig -d $i -e s INT_LOG_MAX_PAYLOAD_SIZE=0
<reboot for the change to take effect>

3.    The file system block is 16MB (file system block size is a choice based on the use case)

Instead of using an external pfind script to control inode prefetch, a helper script is run (as shown in the 'runit' script attached to the io500 submission).

cat find_script/track_io500.sh

RESULT_FILE="${RESULT_DIR}/${WRK_DIR}/result.txt"

#echo "work_dir: $WRK_DIR $RESULT_FILE"

#echo "result_file:: $RESULT_FILE"

# look out for

mdtest_hard_write='\[mdtest-hard-write\]'

find='\[find\]'

ior_easy_read='\[ior-easy-read\]'

disable_inodePrefetchThreshold () {

date

echo 999 | mmchconfig inodePrefetchThreshold=2000000 -i -N $HOSTLIST

sleep 5s

mmfsadm dump config | grep inodePrefetchThreshold

printf "Disable complete\n"

date

}

enable_inodePrefetchThreshold () {

date

echo 999 | mmchconfig inodePrefetchThreshold=5 -i -N $HOSTLIST

sleep 5s

mmfsadm dump config | grep inodePrefetchThreshold

printf "Enable complete\n"

date

}

tail -fn 0 $RESULT_FILE | while read line; do

if echo "$line" | grep -qE '\[.*?\]'; then

printf "%s workload started ... \n" "$line"

fi

if echo "$line" | grep -qE "$find"; then

printf "Disabling inodePrefetchThreshold\n"

disable_inodePrefetchThreshold

fi

if echo "$line" | grep -qE "$ior_easy_read"; then

printf "enabling inodePrefetchThreshold\n"

enable_inodePrefetchThreshold

break

fi

done

4.    The choice of erasure encoding used is 16+2P, and while this is available on ECE systems already, it is being used here slightly before 16+2P G/A's on ESS.

5.    The file system created for the io500 run preallocates 1500M inodes and uses a file system version of 6.0.0.0 (--mmcrfs --inode-limit 2000M:1500M --version=6.0.0.0)

IBM Storage Scale servers are designed with power redundancy in mind.

The network switch is a single point of failure (not-dual redundant switches) but Storage Scale supports such redundant switch configurations.

This run uses a storage subsystem employing IBM Storage Scale RAID with a 16+2P erasure encoding for data and metadata.

The benchmark was executed by building io500 with openMPI and calling the io500.sh script directly as shown in the 'runit' script attached to the submission:
./io500.sh config_withFGWS_PFIND_FGRS.bb8nvme16m.ini  24

The entire system was dedicated to the io500 run.

Note the very small changes to io500.sh that allow passing in the tasks per node as a parameter:

# This script takes its parameters from the same .ini file as io500 binary.
io500_ini="$1"          # You can set the ini file here
TPN="${2:-1}"
NNODES=$(grep -v '^\s*#' host.list | grep -v '^\s*$' | wc -l)
let TT=$TPN*$NNODES
io500_mpirun="mpiexec"
io500_mpiargs="-n $TT --oversubscribe -npernode $TPN --hostfile host.list"

On the client side, the Storage Scale pagepool (in RAM) is used to cache data in the same manner that the Linux file cache is used to cache data with other file systems such as ext4.  Data is not cached on NVMe drives on the clients as those client drives are not used in the I/O path by the Storage Scale file system.

On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching.  The effect of this caching provides similar caching to that obtained using solid state drive caching systems, and the NVMe drives on the servers also provide solid state caching.

There is no burst buffer used – it is the primary storage itself.  Caching on the server-side page pool is analogous to how disk drive caching works.

Very similar scores have been achieved in several other runs of the benchmark on the same system.

We have validated that the scores achieved are within the physical capabilities of the system.