IO500 - Tier1-CNAF - Reproducibility

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

This storage cluster will be used to store data for HPC tasks for CERN users, and for Italian PNRR projects.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The system will be deployed for production in June 2025, after a setup and test period which took 6 months.

Access to the storage will be granted to users of HPC project at CERN and users of PNRR projects via client machines.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The system will be used for the purpose of storing and retrieving data, not to execute jobs directly.
The storage cluster will be used for research purposes of hpc and italian PNRR projects, for example in the field of medicine or physics.

The storage software is Ceph which is open-source and publicly available. Everyone can download the software. https://ceph.io/en/
The exact version of the software is ceph 19.2.1

24 different servers across 2 racks, with 2 x NVMe Micron 7450 MTFDKCC12T8TFS-1BC1ZABDB 14 TB. CephFS is deployed in replica 3 and all replicas of an object are confined inside a single rack.

The benchmark was executed using openmpi version 5.0.7

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

Commands used to set striping information (either for the entire system or for particular directories)
File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

The storage system contains also HDDs but they are not used.
[root@ceph-s10-04-03 ~]# ceph config dump
mon advanced mon_allow_pool_delete false
mon advanced mon_warn_on_insecure_global_id_reclaim false
mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false
mgr advanced mgr/dashboard/ceph-mon-s10-04-35/server_addr 131.154.129.65 *
mgr advanced mgr/dashboard/ceph-mon-s10-04-36/server_addr 131.154.129.80 *
mgr advanced mgr/dashboard/ceph-mon-s10-04-37/server_addr 131.154.129.81 *
mgr advanced mgr/dashboard/server_addr 131.154.129.65 *
mgr advanced mgr/dashboard/server_port 8080 *
mgr advanced mgr/dashboard/ssl false *
mgr advanced mgr/dashboard/standby_behaviour redirect *
mgr advanced mgr/prometheus/cache true *
mgr advanced mgr/prometheus/exclude_perf_counters false
mgr advanced mgr/prometheus/scrape_interval 15.000000 *
mgr advanced mgr/prometheus/server_addr *
mgr advanced mgr/prometheus/stale_cache_strategy return *
mgr advanced mon_pg_warn_max_object_skew 10.000000
osd.0 basic osd_mclock_max_capacity_iops_hdd 418.867441
osd.1 basic osd_mclock_max_capacity_iops_hdd 466.546435
osd.102 basic osd_mclock_max_capacity_iops_ssd 33084.358829
osd.103 basic osd_mclock_max_capacity_iops_ssd 43656.232143
osd.107 basic osd_mclock_max_capacity_iops_hdd 313.392561
osd.108 basic osd_mclock_max_capacity_iops_hdd 379.925314
osd.109 basic osd_mclock_max_capacity_iops_hdd 419.211335
osd.128 basic osd_mclock_max_capacity_iops_ssd 40271.178059
osd.129 basic osd_mclock_max_capacity_iops_ssd 33444.557669
osd.133 basic osd_mclock_max_capacity_iops_hdd 326.264445
osd.15 basic osd_mclock_max_capacity_iops_hdd 428.490551
osd.167 basic osd_mclock_max_capacity_iops_ssd 33289.165638
osd.168 basic osd_mclock_max_capacity_iops_ssd 34679.120972
osd.18 basic osd_mclock_max_capacity_iops_hdd 344.278455
osd.180 basic osd_mclock_max_capacity_iops_ssd 42444.047074
osd.181 basic osd_mclock_max_capacity_iops_ssd 46225.808359
osd.196 basic osd_mclock_max_capacity_iops_hdd 222.665733
osd.206 basic osd_mclock_max_capacity_iops_ssd 38052.191523
osd.207 basic osd_mclock_max_capacity_iops_ssd 41171.934410
osd.232 basic osd_mclock_max_capacity_iops_ssd 48165.503606
osd.233 basic osd_mclock_max_capacity_iops_ssd 41891.734666
osd.24 basic osd_mclock_max_capacity_iops_ssd 31995.665313
osd.25 basic osd_mclock_max_capacity_iops_ssd 41047.201682
osd.280 basic osd_mclock_max_capacity_iops_ssd 42414.628013
osd.282 basic osd_mclock_max_capacity_iops_ssd 40186.173971
osd.284 basic osd_mclock_max_capacity_iops_ssd 35040.297276
osd.285 basic osd_mclock_max_capacity_iops_ssd 41709.255292
osd.306 basic osd_mclock_max_capacity_iops_hdd 487.573611
osd.332 basic osd_mclock_max_capacity_iops_ssd 40401.867681
osd.334 basic osd_mclock_max_capacity_iops_ssd 36414.580583
osd.345 basic osd_mclock_max_capacity_iops_ssd 40964.838841
osd.346 basic osd_mclock_max_capacity_iops_ssd 31671.963926
osd.371 basic osd_mclock_max_capacity_iops_hdd 483.820436
osd.384 basic osd_mclock_max_capacity_iops_ssd 44185.346157
osd.386 basic osd_mclock_max_capacity_iops_ssd 33840.587191
osd.410 basic osd_mclock_max_capacity_iops_ssd 41909.885448
osd.414 basic osd_mclock_max_capacity_iops_ssd 44894.234123
osd.441 basic osd_mclock_max_capacity_iops_hdd 396.843432
osd.445 basic osd_mclock_max_capacity_iops_ssd 44952.630267
osd.448 basic osd_mclock_max_capacity_iops_ssd 45287.687772
osd.455 basic osd_mclock_max_capacity_iops_ssd 37166.264137
osd.457 basic osd_mclock_max_capacity_iops_ssd 43004.480508
osd.469 basic osd_mclock_max_capacity_iops_ssd 47458.500033
osd.470 basic osd_mclock_max_capacity_iops_ssd 33085.144387
osd.496 basic osd_mclock_max_capacity_iops_hdd 481.481759
osd.50 basic osd_mclock_max_capacity_iops_ssd 44839.145842
osd.508 basic osd_mclock_max_capacity_iops_hdd 440.633315
osd.51 basic osd_mclock_max_capacity_iops_ssd 43311.902989
osd.537 basic osd_mclock_max_capacity_iops_ssd 42027.146931
osd.540 basic osd_mclock_max_capacity_iops_ssd 41283.975112
osd.554 basic osd_mclock_max_capacity_iops_ssd 33633.722839
osd.555 basic osd_mclock_max_capacity_iops_ssd 39216.035374
osd.571 basic osd_mclock_max_capacity_iops_ssd 30023.245498
osd.574 basic osd_mclock_max_capacity_iops_ssd 44022.491795
osd.586 basic osd_mclock_max_capacity_iops_ssd 28778.770845
osd.587 basic osd_mclock_max_capacity_iops_ssd 47411.742409
osd.599 basic osd_mclock_max_capacity_iops_ssd 45468.710336
osd.601 basic osd_mclock_max_capacity_iops_ssd 27481.021675
osd.61 basic osd_mclock_max_capacity_iops_hdd 298.483919
osd.622 basic osd_mclock_max_capacity_iops_ssd 32421.408803
osd.623 basic osd_mclock_max_capacity_iops_ssd 47222.932665
osd.76 basic osd_mclock_max_capacity_iops_ssd 34541.353063
osd.77 basic osd_mclock_max_capacity_iops_ssd 45664.131364
osd.78 basic osd_mclock_max_capacity_iops_hdd 416.102236
osd.86 basic osd_mclock_max_capacity_iops_hdd 298.986253
osd.9 basic osd_mclock_max_capacity_iops_hdd 395.461084
osd.92 basic osd_mclock_max_capacity_iops_hdd 388.723485
mds basic mds_cache_memory_limit 68719476736

We added 3 additional physical server that behave as monitor, manager, metadata server. These are older ones with 2 x Intel Xeon E5-2630L, 128 GB DDR4 RAM, 10GbE

Block size in ceph is 4M and no striping set ceph.dir.layout.stripe_count="1".

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

Power (For example, can power failure of a node or rack bring down the storage system?)
Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

The storage racks have 2 PDUs powered by independant sources, which can then be expanded to 4.
Each rack has 2 ToR switches with DELL VLT routing each connected to a storage server and connected to two differente core routers.
Each rack has 12 server and objects are in replica 3 in a single rack, so we can loose 2 of the 12 servers of each rack without loosing data.

Each server has a dual port NIC in bonding, and loosing a link we loose only half the bandwidth.
There is no hardware raid, but is not necessary with Ceph. The WAL/DB device for each osd, that is for each hdd, is to be found in a partition of the 2 NVMes set in raid 1.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The benchmark was executed before setting the system in production, so no other jobs
were running at the same time, and the system and storage were entirely dedicated to running
the benchmark.

The benchmark was executed using the provided io500.sh script using openmpi and 10
computing nodes

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

Client data/metadata caching (in Linux page cache or in other memory cache)
Client side NVMe read-only data cache
Storage server metadata/data caching in RAM
Storage controller caching
RAID card caching

64GB cache on the MDS server. Each HDD of 16TB has it's write ahead cache in a NVMe partition of 60GB.

All other parameters are left ad Ceph defaults.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

This is a standalone storage system and for this submission we used only the SSD partition.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

Did you run the benchmark multiple times and get similar scores?
Did you validate the score is below the physical capabilities of the deployed hardware?
How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

We run the benchmark multiple time and the best run has been uploaded. All runs gave similar
results whithin +/- 10% .
We validated that the results of the benchmark to be within the physical capabilities of the
system.
Yes, the data used by the benchmark relies on persistent storage (SSDs drives in this case), and
no burst-buffer layer in use.