IRIS

Memorial Sloan Kettering Cancer Center · WekaIO · Fully Reproducible

Files
Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

IRIS cluster supports MSKCC researchers and collaborators with Cancer research in areas such as Molecular Dynamics, Single Cell RNA Analysis , Protein Interactions and reconstructions, Structural Biology cryoem applications , Deep learning applications on genome datasets and microscope images to train models - most of which need GPUs to compute.

Some of the examples include Openmm, Alphafold, Relion, Cryosparc, Pytorch, Tensorflow, Horovod, Jupyter Notebook and Rstudio.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

Deployment Timeframe is from November 2023 to 2028. It would not be destroyed, but instead constantly refreshed. Cloud bursting is not enabled for this cluster.

Cluster is available for MSKCC researchers and Collaborators to support Cancer Research.

Most regular users would be MSKCC researchers running :

1. Analysis on drug discovery using applications such as OpenMM, and Alphafold, 

2. Protein Structure reconstuctions with Cryoem data using applications such as Relion and Cryosparc.

3. Single Cell RNA analysis pipelines

4. Pytorch and Horovod framework applications with Genome and Microscope 

4. Deep learning applications on Genome and microcope images to generate AI models.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).


How is this storage software available? 
Storage system software used is WekaIO, which is commercially available.
It is vendor agnostic software, and can be installed on any hardware that supports NVME and Mellanox ConnectX cards.

Can anyone download/purchase this software?
It is available to download and use, once licenses are purchased from WekaIO for the respective storage size.

List either product webpage or open-source repo of the exact software used in IO500.
https://www.weka.io/

Additonal Details:

WekaIO in current cluster is installed on :
22 Dell R650 Nodes with 8 15TB RW intensive NVME drives each for WekaFS, and 2 SSDs for OS.
Two Mellanox Connectx-6 Infiniband HDR cards with 1 Infiniband HDR cable each connected to spine leaf Mellanox HDR infiniband Fabric
Single Dual port 25Gb ethernet connection on Mellanox Connectx-5 for ssh and io traffic to VMs.
Hardware raid only exists for OS disk on RAID 1. 
Hardware raid does not exist for NVME drives, since WekaIO takes care of setting up RAID and Data protection on the respective NVME disks. 
These filesystems are snapshotted daily and uploaded to object storage for backups.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
  1. Compiled and built Openmpi locally, version - 4.1.5
  2. Compiled and built IO500 locally using the above OpenMPI in the path.
  3. Each compute node has 128 logical CPUs ( 64 physical cores ) . Made sure that 8 CPUs ( 4 physical cores ) are reserved for WekaIO, 2 are reserved for OS. Thus used 118 CPUs per compute node.
  4. Created hostfile that contains 36 compute nodes and respective core counts on each server in this format

host1 slots=118
host2 slots=118
host3 slots=118 ....... and so on upto 36 nodes.

  1. Modified io500.sh to include the openmpi binary and mpi arguments as below: 

io500_mpirun="/home/<username>/software/openmpi/bin/mpirun"
io500_mpiargs="-x TMPDIR -np 4248 -map-by node --hostfile /home/<username/software/io500_sc23/io500/hostfile -bind-to none -mca pml ob1 -mca btl ^openib"

  1. Used particular -mca options according to the IB fabric present, used -map-by node to distribute process per node instead of per slot.
  2. WekaIO filesystem is mounted with these options: rw,noatime,writecache,inode_bits=auto,readahead_kb=32768,dentry_max_age_positive=600000,dentry_max_age_negative=600000,container_name=client 
  3. Created the parent directories in WekaIO filesystem and gave the paths to config-all.ini.
  4. Changed nproc to 1200 and pfind-queue-length to 50000 in all "find" categories.
  5. Made sure WekaIO filesystem was not tiering down to any object storage.

Ran the below command in a script:

sh io500.sh config-all.ini &> io500_output.log 

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Power:

Each server has dual power modules to different circuits. Storage system can sustain a power module failure per server or circuit/PDU failure per rack. It can also sustain power failures to 2 servers. 

Networking:

Each server has two Connectx6 infiniband ports on HA connected to two different HDR IB switches. So each server can sustain a single NIC failure or a single HDR IB leaf switch failure.  Each switch is dual connected to power circuits, so can sustain a power module failure or circuit failure. Weka IO software takes care of HA for IB cards.
Each server has a two ethernet ports/cables in a active-active bond. So the server can sustain a failure to a single ethernet port.

Inter/Intra storage data and metadata server:
Distributed Data Protection on Weka Filesystem/NVME drives is done by WekaIO software in 16+2p. It can sustain 2 simultaneous node failures or drive failures.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

Cluster was dedicated to run IO500 benchmark. ( although snapshots and background processes in the storage system continued to run as per the schedule )

Did not use jobs to prepare or configure/setup the client/filesystem. IO500 benchmark was run manually on these 36 compute nodes without the slurm scheduler.

Followed the below steps:

  1. 36 compute nodes were drained from Slurm. 
  2. Logged into the first compute node:
    1. Downloaded openmpi and IO500 source files.
    2. Compiled and built Openmpi locally, version - 4.1.5
    3. Compiled and built IO500 locally using the above OpenMPI in the path.
    4. Each compute node has 128 logical CPUs ( 64 physical cores ) . Made sure that 8 CPUs ( 4 physical cores ) are reserved for WekaIO, 2 are reserved for OS. Thus used 118 CPUs per compute node.
    5. Created hostfile that contains 36 compute nodes and respective core counts on each server in this format

    host1 slots=118
    host2 slots=118
    host3 slots=118 ....... and so on upto 36 nodes.

    1. Modified io500.sh to include the openmpi binary and mpi arguments as below: 

    io500_mpirun="/home/<username>/software/openmpi/bin/mpirun"
    io500_mpiargs="-x TMPDIR -np 4248 -map-by node --hostfile /home/<username/software/io500_sc23/io500/hostfile -bind-to none -mca pml ob1 -mca btl ^openib"

    1. Used particular -mca options according to the IB fabric present, used -map-by node to distribute process per node instead of per slot.
    2. WekaIO filesystem is mounted with these options: rw,noatime,writecache,inode_bits=auto,readahead_kb=32768,dentry_max_age_positive=600000,dentry_max_age_negative=600000,container_name=client 
    3. Created the parent directories in WekaIO filesystem and gave the paths to config-all.ini.
    4. Changed nproc to 1200 and pfind-queue-length to 50000 in all "find" categories.
    5. Made sure WekaIO filesystem was not tiering down to any object storage.

    Ran the below command in a script:

    sh io500.sh config-all.ini &> io500_output.log 

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

No explicit caching was enabled in WekaIO software, except the below mount option, which enables dentry cache:

rw,noatime,writecache,inode_bits=auto,readahead_kb=32768,dentry_max_age_positive=600000,dentry_max_age_negative=600000,container_name=client 

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The submitted system is a stand-alone primary storage file system, with persistent NVME storage space.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

Did you run the benchmark multiple times and get similar scores?

Benchmarks were run multiple times, and similar scores were obtained with slight deviations. The highest score was taken into consideration.

Did you validate the score is below the physical capabilities of the deployed hardware?

Yes

How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The job was continously run without interruption, and output files were observed on other nodes as and how the benchmarks were finished within the measured time. Since the output files could be read from other nodes that were not involved in the benchmark - it was understood that the files were being written to persistent share filesystem storage before acknowledge the close, and were not being cached on the compute nodes that were involved in the benchmark.