Randi

Center for Research Informatics at University of Chicago · IBM GPFS · Fully Reproducible

Files
Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The University of Chicago Biological Sciences Division (BSD) operates a research computing facility that serves researchers across the BSD and both researchers and clinicians in the University of Chicago Medical Center. This facility is funded through periodic investments from the BSD Dean's office. All equipment in managed and maintained by The Center for Research Informatics (CRI) within the BSD. The CRI consists of a scientific computing team which manages the infrastructure for both HPC and storage, a bioinformatics core which provides research support, a data warehousing team, and an application development team.

The benchmarked system is the general purpose storage to support researchers and clinicians in the University of Chicago Biological Sciences Division. The applications supported by the system are used by the the basic sciences (Ecology and Evolution, Organismal Biology), biomedical research (Human Genetics), and clinical operations at the University of Chicago Medical Center (Clinical Pathology).

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The benchmarked system is expected to be deployed for 5 years unitl it is replaced by another storage system. The availability of the system is 24/7. Uptime is important because both Genetic and Clinical Pathology data is stored here that is used to influence patient treatment. We plan to have 4 short maintenance outages a year.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The storage system is an IBM ESS 3500 running Spectrum Scale 5.1.6. It is used to support biological applications such as those used in genetic assemby, alignment, and variant annotation, FEA applications such as Abaqus, and CFD applications such as XFlow. The purpose of the system is to support the publications produced by the researchers in the Biological Sciences Division and to influence patient treatment at the University of Chicago Medical Center.

The storage software is Spectrum Scale so it is commercially available.

The MPI used in the benchmark result was openmpi 4.1.2.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

The runtime environment consisted of the ESS cluster and 10 compute nodes from our HPC cluster. All nodes were connected via a Mellanox Infiniband HDR100 fabric. All clients were designated as managers for benchmarking.

On the storage cluster, the following parameters were modified from the default:

# mmdiag --config | grep -E "\!|\*"

! ccrEnabled 1

! cifsBypassShareLocksOnRename 1

! cifsBypassTraversalChecking 1

! cipherList AUTHONLY

! clusterId 4534293217701222423

! clusterName cri-rss.cri.uchicago.edu

! dataShipAssertOnFailure yes

! dataShipServerBufferPct 10

! dmapiFileHandleSize 32

! maxblocksize 16777216

! maxFilesToCache 131072

! maxMBpS 20000

! minReleaseLevel 3000

! myNodeConfigNumber 3

! pagepool 17179869184

! sdrNotifyAuthEnabled yes

! syncSambaMetadataOps 1

! tscCmdAllowRemoteConnections no

! verbsPorts mlx5_0 mlx5_2

! verbsRdma 1

! verbsRdmaCm disable

! verbsRdmaSend 1

On the clients, the following parameters were modified from the defaults:

# mmdiag --config | grep -E "\!|\*"

! ccrEnabled 1

! cipherList AUTHONLY

! clusterId 5510338773499324762

! clusterName randi.gpfs

! dmapiFileHandleSize 32

! ignorePrefetchLUNCount 1

! maxblocksize 16777216

! maxBufferDescs 2097152

! maxFilesToCache 131072

! maxGeneralThreads 1280

! maxMBpS 20000

! maxStatCache 0

! minReleaseLevel 2301

! myNodeConfigNumber 1239

! nsdInlineWriteMax 8192

! numaMemoryInterleave yes

! pagepool 98784247808

! prefetchPct 50

! syncWorkerThreads 16

! verbsPorts mlx5_0/1

! verbsRdma 1

! verbsRdmaSend 1

! verbsRdmasPerNode 0

! workerThreads 1024

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

A power failure of a node or rack would not bring down the storage system as it is split between racks and PDUs.

A power failure across the whole data center would cause the system to be switched to UPS for 20 minutes and then to generator.

The controllers are duel NSD servers. If one controller fails, the other takes over the workloads to keep the system operational. Erasure coding is implemented with 8+2P RAID stripes. The storage system is split between Mellanox Infiniband switches so a loss of a switch or a single NIC will not affect the ability of the clients to access the data.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

I executed the IO500 at the command line. The execution of the cluster as root was to avoid hitting our LDAP system (FreeIPA). The benchmark was ran on 10 compute nodes that were empty and drained from SLURM during the execution. The storage system was not yet in production, so I had exclusive access to it, but I had to share the Infiniband fabric with other HPC users.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

We did not do any local caching outside of setting up a 92GB pagepool on the GPFS clients and dedicating 50% of that to prefetching.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The storage system is a stand-alone storage system and the primary storage for the entire division.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

Did you run the benchmark multiple times and get similar scores? Yes, within 1 or 2 pts.

Did you validate the score is below the physical capabilities of the deployed hardware? Yes. IBM provided an IO500 benchmark in their proposal that was significantly higher, but they had dedicated access to the fabric and a single switch.

How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?   After the first couple write tests I Ctrl-C'd to stop the benchmark. Files were available and took a long time to clean up.