IO500 - Yellowstone Storage Scale System 6000

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The system is primarily used for performance engineering, test, and measurements/benchmarking. An example of performance engineering work would be debugging customer reported performance issues.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

These resources are always in use and generally in the same cluster format although the Storage Scale servers may be split into separate building blocks and the client nodes may mount other file systems for different storage servers.
If there’s a need to do a larger config change, like split the servers into separate building blocks, the timeframe to redeploy this cluster in a different configuration is roughly 2-3 days.

The system is available for team members that work on IBM Storage Scale, with the primary users being performance engineers.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

IBM Storage Scale (the file system software) and Red Hat (the operating system) are generally commercially available.

Anyone can purchase the software or download a limited use version (it is available commercially).

The IBM Strorage Scale product webpage is:
https://www.ibm.com/products/storage-scale

Regarding the specific storage deployment, this submission uses one building block of the IBM Storage Scale System 6000 solution

Details on the the IBM Storage Scale System 6000 are here:

https://www.ibm.com/downloads/documents/us-en/10a99803f5afd8d6

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

Commands used to set striping information (either for the entire system or for particular directories)
File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

Here are the tuning and configuration steps used:

The version of Storage Scale System 7.0.1.0 code used on the IBM Storage Scale Servers is a development early build of software.
In the case in which a customer wants to reproduce our results, IBM intends to provide this version of code to licensed customers for purposes of reproducing io500 results in a non-production/test environment. (This submission is not from a production system.)

The mmlsconfig info is below

mmlsconfig

Configuration data for cluster test01.s6k2a-hs:

-----------------------------------------------

[common]

dataShipServerBufferPct 50

[clienty]

dataShipServerBufferPct 40

[common]

maxMBpS 50000

[client]

maxStatCache 131072

[clienty,nc1]

maxStatCache 128k

[common]

cipherList AUTHONLY

enableDangerousOptions yes

[ems]

autoBuildGPL yes

[common]

autoload yes

[nc1]

pagepool 973395709132

workerThreads 1024

prefetchPct 50

maxMBpS 50000

nsdRAIDFreshLayoutCaching yes

nsdRAIDEnableFastRebalancePlacementPolicy yes

nsdRAIDScratchpadRebalance no

nsdRAIDPagepoolBaseline 973395709132

[common]

autoCompactDir no

dataShipClientBufferPct 50

dataShipClientBuffersPerServer 50

disablePrefetchThreadRebalance yes

fgdlTokenBatchAcquire yes

fsyncIsGlobal no

maxBufferDescs 10000000

maxTcpConnsPerNodeConn 2

nBucketGroups 1024

preferDesignatedMnode yes

kernelLockType 0

mmapRangeLock yes

minReleaseLevel 6.0.1.0

adminMode central

File systems in cluster test01.s6k2a-hs:

----------------------------------------

/dev/fs1

2.Using the ‘mmchnode’ all nodes (clients and servers) have been defined as ‘manager’ role nodes as shown in ‘mmlscluster’ data. The yellowstone clients do not run ior or act as storage servers.

mmlscluster below:

GPFS cluster information

========================

GPFS cluster name: test01.s6k2a-hs

GPFS cluster id: 2115447780686195884

GPFS UID domain: test01.s6k2a-hs

Remote shell command: /usr/bin/ssh

Remote file copy command: /usr/bin/scp

Repository type: CCR

Node Daemon node name IP address Admin node name Designation

----------------------------------------------------------------------

1 s6k2aib 100.168.83.48 s6k2aib quorum-manager-perfmon

2 s6k2bib 100.168.83.49 s6k2bib quorum-manager-perfmon

3 saguaro1ib 100.168.83.51 saguaro1ib manager-perfmon

4 saguaro2ib 100.168.83.52 saguaro2ib quorum-manager-perfmon

5 saguaro3ib 100.168.83.53 saguaro3ib quorum-manager-perfmon

6 saguaro4ib 100.168.83.54 saguaro4ib quorum-manager-perfmon

7 saguaro5ib 100.168.83.55 saguaro5ib manager-perfmon

8 saguaro6ib 100.168.83.56 saguaro6ib manager-perfmon

9 saguaro7ib 100.168.83.57 saguaro7ib manager-perfmon

10 saguaro8ib 100.168.83.58 saguaro8ib manager-perfmon

11 saguaro9ib 100.168.83.59 saguaro9ib manager-perfmon

12 saguaro10ib 100.168.83.60 saguaro10ib manager-perfmon

14 yellowstone1ib 100.168.83.131 yellowstone1ib manager-perfmon

15 yellowstone2ib 100.168.83.132 yellowstone2ib manager-perfmon

16 yellowstone3ib 100.168.83.133 yellowstone3ib manager-perfmon

17 yellowstone4ib 100.168.83.134 yellowstone4ib manager-perfmon

18 yellowstone5ib 100.168.83.135 yellowstone5ib manager-perfmon

19 yellowstone6ib 100.168.83.136 yellowstone6ib manager-perfmon

20 yellowstone7ib 100.168.83.137 yellowstone7ib manager-perfmon

21 yellowstone8ib 100.168.83.138 yellowstone8ib manager-perfmon

22 yellowstone9ib 100.168.83.139 yellowstone9ib manager-perfmon

23 yellowstone10ib 100.168.83.140 yellowstone10ib manager-perfmon

3. The PCI tuning option PCI_WR_ORDERING has been enabled on all nodes as follows:

Find the MST device name of each adapter, e.g.:

mst start
mst status
/dev/mst/mt4123_pciconf0

Enable PC_WR_ORDERING for each such device, e.g.:

mlxconfig -d /dev/mst/mt4123_pciconf0 s PCI_WR_ORDERING=1

4. The file system block is 16 MB (file system block size is a choice based on the use case)

5. An early version of the IBM Storage Scale 7.0.1-0 release is installed

6. The GNR erasure encoded pool uses 8+2P encoding, which is a choice based on the use case (8+2P is the most common erasure encoding option used for the storage configuration).

7. The file system created for the io500 run preallocates 750M inodes (mmcrfs --inode-limit 750M:750M)

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

Power (For example, can power failure of a node or rack bring down the storage system?)
Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

IBM Storage Scale System 6000 servers are designed with power redundancy in mind.

The network switch is a single point of failure (not-dual redundant switches)

The IBM Storage Scale System 6000 has 48 Samsung 1743 NVMe drives that provide the storage for the IBM Storage Scale RAID pool. These 48 drives use an 8+2P erasure encoding scheme to protect the data and, for some metadata, replication to protect file system data from hardware component failures. All the data for the io500 benchmark is written to this IBM Storage Scale RAID pool.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The benchmark was run on a dedicated cluster using mpich.

The cluster was dedicated to the io500 job. The idle client nodes were not running another workload.

The io500 executable is built with mpich and run as follows:
./io500.sh ess6000_config.ini yellowstone_nodesib.10 20

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

Client data/metadata caching (in Linux page cache or in other memory cache)
Client side NVMe read-only data cache
Storage server metadata/data caching in RAM
Storage controller caching
RAID card caching

On the client side, the Storage Scale pagepool (in RAM) is used to cache data in the same manner that the Linux file cache is used to cache data with other file systems such as ext4.

The NVMe drives on the client drives are not used for this io500 run.

On the server side, a portion of the servers’ Storage Scale pagepool (RAM) is also available for caching, providing caching similar to the solid state caches that many drives use.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

There is no burst buffer in this configuration. The storage provided by the IBM SSS 6000 is the single source of truth.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

Did you run the benchmark multiple times and get similar scores?
Did you validate the score is below the physical capabilities of the deployed hardware?
How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

Very similar scores have been achieved in several other runs of the benchmark on the same system.

We have validated that the scores achieved are within the physical capabilities of the system.