Saguaro Storage Scale System 6000

The system is primarily used for performance engineering, test, and measurements/benchmarking.  An example of performance engineering work would be debugging customer reported performance issues.

These resources are always in use and generally in the same cluster format although the Storage Scale servers may be split into separate building blocks and the client nodes may mount other file systems for different storage servers. 
If there’s a need to do a larger config change, like split the servers into separate building blocks, the timeframe to redeploy this cluster in a different configuration is roughly 2-3 days.

The system is available for team members that work on IBM Storage Scale, with the primary users being performance engineers.

IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.

Anyone can purchase the software or download a limited use version (it is available commercially).

The IBM Storage Scale product webpage is https://www.ibm.com/products/storage-scale

Regarding the specific storage deployment, this submission uses two building blocks of the IBM Storage Scale System 6000 solution, and also incorporates a Data Acceleration Performance Pool using the local NVMe drives on the clients using an early (pre-GA) version of this code. 

Details on the the IBM Storage Scale System 6000 are here:

https://www.ibm.com/downloads/documents/us-en/10a99803f5afd8d6

Here are the tuning and configuration steps used:

The version of Storage Scale System 7.0.0.0 code used on the IBM Storage Scale Servers is a development early build of software. 
In the case in which a customer wants to reproduce our results, IBM intends to provide this version of code to licensed customers for purposes of reproducing io500 results in a non-production/test environment.  (This submission is not from a production system.)

1.    The mmlsconfig info is below

Configuration data for cluster test01.s6k2a-hs:

-----------------------------------------------

clusterName test01.s6k2a-hs

clusterId 2115447780686195884

dmapiFileHandleSize 32

tscCmdAllowRemoteConnections no

ccrEnabled yes

cipherList AUTHONLY

sdrNotifyAuthEnabled yes

maxblocksize 16m

[client]

maxblocksize 16M

[s6k2aib,s6k2bib]

nsdRAIDDefaultIoTimeout 100000

[common]

nsdRAIDClientOnlyChecksum yes

subnets 192.168.0.0/16

nsdRAIDLookUpNoFlush 1

tasksPerNode 20

allowSuperstrideReadFastpath 1

[nc1]

nsdRAIDClientOnlyChecksum yes

nsdRAIDEnableSpaceMonitor 1

pagepoolMaxPhysMemPct 90

[common]

dataShipDebugInterval 1

[client,clienty,nc1]

verbsRdma enable

verbsRdmaSend yes

[common]

fsyncIsGlobal no

preferDesignatedMnode yes

[nc1]

disableV5KernBIO 1

[saguaro10ib,saguaro1ib,saguaro2ib,saguaro3ib,saguaro4ib,saguaro5ib,saguaro6ib,saguaro7ib,saguaro8ib,saguaro9ib,client,clienty]

dataShipClientBuffersPerServer 50

[common]

inodePrefetchThreshold 5

[nc1]

nspdQueues 240

[ncd]

nspdQueues 65

[nc1,ncd]

nspdBufferMemPerQueue 24m

nspdThreadsPerQueue 2

nsdRAIDEventLogToConsole all

nsdRAIDBlockDeviceMaxSectorsKB 0

nsdRAIDBlockDeviceNrRequests 0

nsdRAIDBlockDeviceQueueDepth 0

nsdRAIDBlockDeviceScheduler off

nsdRAIDSmallThreadRatio 2

nsdRAIDDefaultGeneratedFD no

nsdRAIDMasterBufferPoolSize 2G

nsdRAIDReconstructAggressiveness 0

nsdRAIDThreadsPerQueue 16

nsdRAIDSSDPerformanceShortTimeConstant 2500000

nsdRAIDDiskDiagTimeout 130

[ncd]

nsdRAIDEnableRGCMRebalanceWeight yes

[common]

numaMemoryInterleave no

[client,clienty,nc1,ncd]

numaMemoryInterleave yes

[nc1,ncd]

panicOnIOHang yes

[common]

ignorePrefetchLUNCount no

[client,clienty,nc1,ncd]

prefetchPartitions 2

[common]

autoCompactDir no

dataShipServerBufferPct 50

fgdlTokenBatchAcquire yes

maxBufferDescs 10000000

maxTcpConnsPerNodeConn 2

nBucketGroups 1024

proactiveReconnect no

[client]

prefetchMaxThreadsPerOpenFile 12

[clienty]

disablePrefetchThreadRebalance 1

[client]

disablePrefetchThreadRebalance yes

[nc1,ncd]

nsdSmallThreadRatio 1

[client]

nsdSmallThreadRatio 2

nsdThreadsPerQueue 16

nsdbufspace 70

[nc1,ncd]

nsdMinWorkerThreads 3842

[client]

nsdMinWorkerThreads 1024

[nc1,ncd]

nsdMaxWorkerThreads 3842

[client]

nsdMaxWorkerThreads 1024

[common]

pagepool 1G

[clienty]

pagepool 48G

[ncd]

pagepool 226G

[nc1]

pagepool 973405011148

[client]

pagepool 300G

[common]

minReleaseLevel 6.0.0.0

[nc1]

verbsPorts mlx5_0/1 mlx5_1/1 mlx5_2/1 mlx5_3/1 mlx5_10/1 mlx5_11/1 mlx5_12/1

[client,clienty]

verbsPorts mlx5_0/1 mlx5_1/1

[nc1]

nsdRAIDTracks 320K
[client]
numactlOptioni 0
numactlOptionN 0
[common]

adminMode central

2.Using the ‘mmchnode’ all nodes (clients and servers) have been defined as ‘manager’ role nodes as shown in ‘mmlscluster’ data.  The yellowstone clients do not run ior or act as storage servers.

mmlscluster below:

GPFS cluster information

========================

GPFS cluster name:         test01.s6k2a-hs

GPFS cluster id:           2115447780686195884

GPFS UID domain:           test01.s6k2a-hs

Remote shell command:      /usr/bin/ssh

Remote file copy command:  /usr/bin/scp

Repository type:           CCR

Node  Daemon node name  IP address      Admin node name  Designation

----------------------------------------------------------------------

1   s6k2aib           100.168.83.48   s6k2aib          quorum-manager-perfmon

2   s6k2bib           100.168.83.49   s6k2bib          quorum-manager-perfmon

3   saguaro1ib        100.168.83.51   saguaro1ib       manager-perfmon

4   saguaro2ib        100.168.83.52   saguaro2ib       quorum-manager-perfmon

5   saguaro3ib        100.168.83.53   saguaro3ib       quorum-manager-perfmon

6   saguaro4ib        100.168.83.54   saguaro4ib       quorum-manager-perfmon

7   saguaro5ib        100.168.83.55   saguaro5ib       manager-perfmon

8   saguaro6ib        100.168.83.56   saguaro6ib       manager-perfmon

9   saguaro7ib        100.168.83.57   saguaro7ib       manager-perfmon

10   saguaro8ib        100.168.83.58   saguaro8ib       manager-perfmon

11   saguaro9ib        100.168.83.59   saguaro9ib       manager-perfmon

12   saguaro10ib       100.168.83.60   saguaro10ib      manager-perfmon

14   yellowstone1ib    100.168.83.131  yellowstone1ib   manager-perfmon

15   yellowstone2ib    100.168.83.132  yellowstone2ib   manager-perfmon

16   yellowstone3ib    100.168.83.133  yellowstone3ib   manager-perfmon

17   yellowstone4ib    100.168.83.134  yellowstone4ib   manager-perfmon

18   yellowstone5ib    100.168.83.135  yellowstone5ib   manager-perfmon

19   yellowstone6ib    100.168.83.136  yellowstone6ib   manager-perfmon

20   yellowstone7ib    100.168.83.137  yellowstone7ib   manager-perfmon

21   yellowstone8ib    100.168.83.138  yellowstone8ib   manager-perfmon

22    yellowstone9ib    100.168.83.139  yellowstone9ib   manager-perfmon

23    yellowstone10ib   100.168.83.140  yellowstone10ib  manager-perfmon

3.    The PCI tuning option PCI_WR_ORDERING has been enabled on all nodes as follows:

Find the MST device name of each adapter, e.g.:

mst start
mst status
/dev/mst/mt4123_pciconf0

Enable PCI_WR_ORDERING for each such device, e.g.:
mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1 

<reboot for the change to take effect>

4.    The file system block is 16 MB (file system block size is a choice based on the use case)

5.  An early version of the IBM Storage Scale 7.0.0.0 release is installed, and the decentralized DAT feature is used.  Each saguaro client assigned a failure group ID equal to the client node number (e.g., saguaro1ib has failure group ID 1,0,1, and saguaro2ib has failure group ID 1,0,2, etc) and the following policy is applied to allocate just ior_file_easy files in the DAT performance pool to client drives (rather than the GNR erasure protected pool):

RULE 'ior_file_easy.0-24' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000000' AND name <= 'ior_file_easy.00000024') AND (setWADFGP(1,0,10))

RULE 'ior_file_easy.25-49' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000025' AND name <= 'ior_file_easy.00000049') AND (setWADFGP(1,0,1))

RULE 'ior_file_easy.50-74' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000050' AND name <= 'ior_file_easy.00000074') AND (setWADFGP(1,0,2))

RULE 'ior_file_easy.75-99' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000075' AND name <= 'ior_file_easy.00000099') AND (setWADFGP(1,0,3))

RULE 'ior_file_easy.100-124' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000100' AND name <= 'ior_file_easy.00000124') AND (setWADFGP(1,0,4))

RULE 'ior_file_easy.125-149' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000125' AND name <= 'ior_file_easy.00000149') AND (setWADFGP(1,0,5))

RULE 'ior_file_easy.150-174' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000150' AND name <= 'ior_file_easy.00000174') AND (setWADFGP(1,0,6))

RULE 'ior_file_easy.175-199' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000175' AND name <= 'ior_file_easy.00000199') AND (setWADFGP(1,0,7))

RULE 'ior_file_easy.200-224' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000200' AND name <= 'ior_file_easy.00000224') AND (setWADFGP(1,0,8))

RULE 'ior_file_easy.225-249' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000225' AND name <= 'ior_file_easy.00000249') AND (setWADFGP(1,0,9))

RULE 'default' SET POOL 'system'

6.    The GNR erasure encoded pool uses 8+2P encoding, which is a choice based on the use case (8+2P is the most common erasure encoding option used for this storage configuration). 

7.    The file system created for the io500 run preallocates 750M inodes (mmcrfs --inode-limit 750M:750M)

IBM Storage Scale System 6000 servers are designed with power redundancy in mind.

The network switch is a single point of failure (not-dual redundant switches)

The IBM Storage Scale System 6000 has 48 Samsung 1743 NVMe drives that provide the storage for the IBM Storage Scale RAID pool.  These 48 drives use an 8+2P erasure encoding scheme to protect the data and, for some metadata, replication to protect file system data from hardware component failures.  All the data for the io500 benchmark is written to this IBM Storage Scale RAID pool, except the ior-easy files.

The local drives on the saguaro clients that comprise the Data Acceleration Performance Pool do not employ erasure encoding.  These drives were not erasure-coded; as a result, data on these client-resident volumes would be at risk if a drive failed. Data can optionally be migrated or replicated to the protected RAID pool via mechanisms such as IBM Storage Scale policies but no such copy is done in this case.  The data is still completely consistent across the file system, but it is exposed to disk failures.

The io500 benchmark suite was first built with AIO support for the ior benchmark code as described in this issue that Julian Kunkel assisted with:
https://github.com/IO500/io500/issues/102

The benchmark was run on a dedicated cluster using mpich with the following details:
mpiexec -version

HYDRA build details:

Version:                                 3.2

Release Date:                            Wed Nov 11 22:06:48 CST 2015

CC:                              gcc

CXX:                             g++

F77:                             gfortran

F90:                             gfortran

Configure options:                       '--disable-option-checking' '--prefix=/nshare/gpfs-core/x86/benchmarks/rhel8.x/mpich' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= -I/tmp/mpich-3.2/src/mpl/include -I/tmp/mpich-3.2/src/mpl/include -I/tmp/mpich-3.2/src/openpa/src -I/tmp/mpich-3.2/src/openpa/src -D_REENTRANT -I/tmp/mpich-3.2/src/mpi/romio/include'
    Process Manager:                         pmi

Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist

Topology libraries available:            hwloc

Resource management kernels available:   user slurm ll lsf sge pbs cobalt

Checkpointing libraries available:

Demux engines available:                 poll select


The run script is included in the results archive.  The only changes to the default io500 run script are to specify the mpiexec arguments and also pass in the tasks per node and host.list file name, as per below:
# This script takes its parameters from the same .ini file as io500 binary.

io500_ini="$1"          # You can set the ini file here

host_name="$2"

procs="$3"

#io500_mpirun="/home/mpich/bin/mpiexec"

io500_mpirun="mpiexec"

#io500_mpiargs="-f ${host_name} -n ${procs} -ppn 30"

#orig

#io500_mpiargs="-f ${host_name} -ppn 20"

#client first socket only

io500_mpiargs="--bind-to numa:1 -f ${host_name} -genv I_MPI_PIN_DOMAIN=core -ppn $procs" 

On the client side, the Storage Scale pagepool (in RAM) is used to cache data in the same manner that the Linux file cache is used to cache data with other file systems such as ext4.  Data is not cached by IBM Storage Scale on the NVMe drives on the client drives that provide the storage for the IBM Data Acceleration Tier performance pool.

On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching.  The effect of this caching provides similar caching to that obtained using solid state drive caching systems. 

While the system includes two distinct storage pools (the RAID pool and the client NVMe pool), both are fully integrated within the same coherent namespace. All writes are immediately visible across all nodes; there is no eventual-consistency or staging mechanism.

While the mode of writing to the local drives on the clients may be referred to as a burst buffer use case, this design differs from “burst buffer” systems described in early uses of the term, e.g., On the Role of Burst Buffers in Leadership-Class Storage Systems (MCS ANL, 2012), which define burst buffers as “low-capacity, high-throughput staging layers that absorb short-term I/O bursts”.

The client-side pool can hold large amounts of data indefinitely, but for protection against local disk failures, data replication or migration to the IBM Storage Scale RAID (GNR) pool is recommended in the longer term.  In the io500 run done here, the data is not copied to the GNR pool

Very similar scores have been achieved in several other runs of the benchmark on the same system.

We have validated that the scores achieved are within the physical capabilities of the system.