Tier1-CNAF

Istituto Nazionale di Fisica Nucleare · Huawei OceanStor · Fully Reproducible

Files
Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

This storage cluster was used for data processing of LHC experiments at CERN as TIER 1 site of Worldwide LHC Computing Grid (WLCG)

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The system was deployed during the period of December 2018 to February 2019, and was in production from March 2019 to December 2024. 
Access to the storage was granted to users of LHC experiments.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The software used is Spectrum Scale from IBM, which is commercially available. We made a single filesystem of about 11PiB, divided into multiple filesets pointing to two different storage pools, one with 60 SSDs and another with 2412 HDDs.
This specific submission refers to the pool with SSDs only. 


The benchmark was executed using openmpi version 5.0.7

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

[root@xs-403 ~]# mmlsconfig
Configuration data for cluster io500.cr.cnaf.infn.it:
-----------------------------------------------------
clusterName io500.cr.cnaf.infn.it
clusterId 16723997371215827715
dmapiFileHandleSize 32
minReleaseLevel 5.1.9.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
verbsPorts mlx5_0/1 mlx5_1/1
nsdThreadsPerQueue 12
maxFilesToCache 200000
nsdMaxWorkerThreads 512
ioHistorySize 512
maxMBpS 32000
maxInodeDeallocPrefetch 128
prefetchLargeBlockThreshold 4194304
linuxStatfsUnits subblocks
workerThreads 512
prefetchThreads 256
verbsRdmaSend yes
autoload yes
prefetchPct 60
pagepool 96G
maxStatCache 800000
nsdSmallThreadRatio 2
verbsRdma enable
[common]
adminMode central

File systems in cluster io500.cr.cnaf.infn.it:
----------------------------------------------
/dev/gpfs_test


[root@xs-403 ~]# mmlsfileset gpfs_test
Filesets in file system 'gpfs_test':
Name                     Status    Path
root                     Linked    /gpfs/gpfs_test
ssd                      Linked    /gpfs/gpfs_test/ssd

[root@xs-403 ~]# mmlspolicy gpfs_test -L
RULE 'SSD' SET POOL data FOR FILESET ('ssd')
RULE 'DEFAULT' SET POOL 'data2'


[root@xs-403 ~]# mmlspool gpfs_test
Storage pools in file system at '/gpfs/gpfs_test':
Name                    Id   BlkSize Data Meta Total Data in (KB)   Free Data in (KB)   Total Meta in (KB)    Free Meta in (KB)
system                   0      4 MB   no  yes              0              0 (  0%)     7914651648     5452574720 ( 69%)
data                 65537      4 MB  yes   no    67203301376    63312744448 ( 94%)              0              0 (  0%)
data2                65538      4 MB  yes   no 11521494253568 11510959566848 (100%)              0              0 (  0%)





[root@xs-403 schema-tools]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         io500.cr.cnaf.infn.it
  GPFS cluster id:           16723997371215827715
  GPFS UID domain:           io500.cr.cnaf.infn.it
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name        IP address       Admin node name         Designation
------------------------------------------------------------------------------------
   1   xs-201.cr.cnaf.infn.it  131.154.130.152  xs-201.cr.cnaf.infn.it  quorum-manager
   2   xs-301.cr.cnaf.infn.it  131.154.130.156  xs-301.cr.cnaf.infn.it  quorum-manager
   3   xs-401.cr.cnaf.infn.it  131.154.130.160  xs-401.cr.cnaf.infn.it  quorum-manager
   4   xs-203.cr.cnaf.infn.it  131.154.130.154  xs-203.cr.cnaf.infn.it  manager
   5   xs-302.cr.cnaf.infn.it  131.154.130.157  xs-302.cr.cnaf.infn.it  manager
   6   xs-303.cr.cnaf.infn.it  131.154.130.158  xs-303.cr.cnaf.infn.it  manager
   7   xs-402.cr.cnaf.infn.it  131.154.130.161  xs-402.cr.cnaf.infn.it  manager
   8   xs-403.cr.cnaf.infn.it  131.154.130.162  xs-403.cr.cnaf.infn.it  manager
   9   xs-404.cr.cnaf.infn.it  131.154.130.163  xs-404.cr.cnaf.infn.it  manager
  10   xs-004.cr.cnaf.infn.it  131.154.130.144  xs-004.cr.cnaf.infn.it  manager


[root@xs-403 ~]#  mmlsfs gpfs_test
flag                value                    description
------------------- ------------------------ -----------------------------------
 -f                 8192                     Minimum fragment (subblock) size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata replicas
 -M                 2                        Maximum number of metadata replicas
 -r                 1                        Default number of data replicas
 -R                 2                        Maximum number of data replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in effect
 -k                 all                      ACL semantics in effect
 -n                 32                       Estimated number of nodes that will mount file system
 -B                 4194304                  Block size
 -Q                 none                     Quotas accounting enabled
                    none                     Quotas enforced
                    none                     Default quotas enabled
 --perfileset-quota No                       Per-fileset quota enforcement
 --filesetdf        No                       Fileset df enabled?
 -V                 33.00 (5.1.9.0)          File system version
 --create-time      Tue Apr  1 16:30:33 2025 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 No                       Exact mtime mount option
 -S                 Yes                      Suppress atime mount option
 -K                 whenpossible             Strict replica allocation option
 --fastea           Yes                      Fast external attributes enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      576716800                Maximum number of inodes
 --uid              98829A83:67EBF889        File system UID
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 512              Number of subblocks per full block
 -P                 system;data;data2        Disk storage pools in file system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 --flush-on-close   No                       flush cache on file close enabled?
 --auto-inode-limit No                       Increase maximum number of inodes per inode space automatically?
 --nfs4-owner-write-acl Yes                  NFSv4 implicit owner WRITE_ACL permission enabled?
 -d                 os18k4_ssd01;os18k5_ssd04;os18k5_ssd03;os18k5_ssd02;os18k5_ssd01;os18k3_ssd04;os18k3_ssd03;os18k3_ssd02;os18k3_ssd01;os18k4_ssd04;os18k4_ssd03;os18k4_ssd02;os18k3_sas01;
 -d                 os18k3_sas02;os18k3_sas03;os18k3_sas04;os18k3_sas05;os18k3_sas06;os18k3_sas07;os18k3_sas08;os18k3_sas09;os18k3_sas10;os18k3_sas11;os18k3_sas12;os18k3_sas13;os18k3_sas14;
 -d                 os18k3_sas15;os18k3_sas16;os18k3_sas17;os18k3_sas18;os18k3_sas19;os18k3_sas20;os18k3_sas21;os18k3_sas22;os18k3_sas23;os18k3_sas24;os18k3_sas25;os18k3_sas26;os18k3_sas27;
 -d                 os18k3_sas28;os18k3_sas29;os18k3_sas30;os18k3_sas31;os18k3_sas32;os18k3_sas33;os18k3_sas34;os18k3_sas35;os18k3_sas36;os18k3_sas37;os18k3_sas38;os18k3_sas39;os18k3_sas40;
 -d                 os18k3_sas41;os18k3_sas42;os18k3_sas43;os18k3_sas44;os18k3_sas45;os18k3_sas46;os18k3_sas47;os18k3_sas48;os18k4_sas01;os18k4_sas02;os18k4_sas03;os18k4_sas04;os18k4_sas05;
 -d                 os18k4_sas06;os18k4_sas07;os18k4_sas08;os18k4_sas09;os18k4_sas10;os18k4_sas11;os18k4_sas12;os18k4_sas13;os18k4_sas14;os18k4_sas15;os18k4_sas16;os18k4_sas17;os18k4_sas18;
 -d                 os18k4_sas19;os18k4_sas20;os18k4_sas21;os18k4_sas22;os18k4_sas23;os18k4_sas24;os18k4_sas25;os18k4_sas26;os18k4_sas27;os18k4_sas28;os18k4_sas29;os18k4_sas30;os18k4_sas31;
 -d                 os18k4_sas32;os18k4_sas33;os18k4_sas34;os18k4_sas35;os18k4_sas36;os18k4_sas37;os18k4_sas38;os18k4_sas39;os18k4_sas40;os18k4_sas41;os18k4_sas42;os18k4_sas43;os18k4_sas44;
 -d                 os18k4_sas45;os18k4_sas46;os18k4_sas47;os18k4_sas48;os18k5_sas01;os18k5_sas02;os18k5_sas03;os18k5_sas04;os18k5_sas05;os18k5_sas06;os18k5_sas07;os18k5_sas08;os18k5_sas09;
 -d                 os18k5_sas10;os18k5_sas11;os18k5_sas12;os18k5_sas13;os18k5_sas14;os18k5_sas15;os18k5_sas16;os18k5_sas17;os18k5_sas18;os18k5_sas19;os18k5_sas20;os18k5_sas21;os18k5_sas22;
 -d                 os18k5_sas23;os18k5_sas24;os18k5_sas25;os18k5_sas26;os18k5_sas27;os18k5_sas28;os18k5_sas29;os18k5_sas30;os18k5_sas31;os18k5_sas32;os18k5_sas33;os18k5_sas34;os18k5_sas35;
 -d                 os18k5_sas36;os18k5_sas37;os18k5_sas38;os18k5_sas39;os18k5_sas40;os18k5_sas41;os18k5_sas42;os18k5_sas43;os18k5_sas44;os18k5_sas45;os18k5_sas46;os18k5_sas47;os18k5_sas48  Disks in file system
 -A                 yes                      Automatic mount option
 -o                 none                     Additional mount options
 -T                 /gpfs/gpfs_test          Default mount point
 --mount-priority   0                        Mount priority

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Everything is redundant.

The storage racks are equipped with 2 PDU, powerd by independant sources and backed UPS + generator. The storage subsystems have redundant PSU, each PSU is connected to a different PDU, as a consequence the failure of a PSU, PDU or the UPS would not bring down the storage system.

The Huawei system storage are also dual controllers, each controller has 2 dual port NIC and each NIC is connected to 2 different switches, allowing the storage to remain available in case of failure of a controller, NIC, cable, or switch.

Disk are configured in 16+2 RAID6 over groups of 67 HDDs, and 8+2 RAID6 over groups of 20 SSDs.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The benchmark was executed after decomissioning as a production system, so no other jobs were running at the same time, and the system and storage were entirely dedicated to running the benchmark.

The benchmark was executed using the provided io500.sh script using openmpi and 10 computing nodes. 

Changes to the default script were:
io500_mpirun="mpiexec --map-by socket"
io500_mpiargs="--allow-run-as-root -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1 -host xs-201.cr.cnaf.infn.it:10,xs-403.cr.cnaf.infn.it:10,xs-301.cr.cnaf.infn.it:10,xs-401.cr.cnaf.infn.it:10,xs-203.cr.cnaf.infn.it:10,xs-302.cr.cnaf.infn.it:10,xs-303.cr.cnaf.infn.it:10,xs-402.cr.cnaf.infn.it:10,xs-404.cr.cnaf.infn.it:10,xs-004.cr.cnaf.infn.it:10 -np 100"

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

We set up a 96GB pagepool on the GPFS clients and dedicating 60% of that to prefetching.

Storage controller caching (mirrored between controllers): write-back and read-through

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

This is a standalone storage system and for this submission we used only the SSD partition. 

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

We run the benchmark multiple time and the best run has been uploaded. All runs gave similar results whithin +/- 10% . 

We validated that the results of the benchmark to be within the physical capabilities of the system.

Yes, the data used by the benchmark relies on persistent storage (SSDs drives in this case), and no burst-buffer layer in use.