UCloud HPC Cluster

UCloud · UCloud HPC Cluster · Fully Reproducible

Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The submitted system is currently deployed as an internal production and research platform within UCloud. Its primary purpose is to provide high-performance storage services for data-intensive AI workloads, including large language model training, AIGC applications, data preprocessing, model evaluation, and related analytics workflows.

The platform is designed to deliver storage performance that matches the rapidly growing compute capabilities of modern AI infrastructure, enabling efficient large-scale data ingestion, storage, and retrieval. In addition to supporting internal business workloads, the system also serves as a validation and development environment for next-generation storage technologies that are planned for future commercial releases.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

Deployment Timeframe

The system was fully deployed and entered regular operation on April 15, 2025.

System Availability and User Community

The platform is primarily used by UCloud's internal AI and large-model teams for workload validation, performance tuning, and business testing. It is also utilized by the storage engineering team for architecture research, feature development, and performance evaluation of future storage products.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The storage software used in this submission is proprietary and is not currently available as an open-source or commercial product.

At present, the platform is used internally by the UCloud Storage Team and selected internal business groups. Members of the IO500 committee who are interested in evaluating the system may contact us to discuss opportunities for test access.

The specific deployment used for this IO500 submission consists of:

  1. Storage Servers
  2. Networking
    • Each storage server is equipped with two Mellanox MCX75310AAS-NEAT 400Gb/s InfiniBand adapters.
  3. Storage Media
    • Each storage server contains ten ScaleFlux CSD500 Series NVMe SSDs.
    • PCIe 5.0 x4 interface.
    • 7.68 TB capacity per drive.
  4. Compute Configuration
    • Two Intel Xeon Gold 6530 processors per storage node.
  5. Storage Architecture
    • The storage software directly manages the underlying SSD devices through a self-developed storage stack.
    • No third-party distributed file system or local file system layer is involved in the data path.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

All publicly shareable materials required to reproduce the benchmark environment have been included in the submission package. These materials include benchmark execution scripts, MPI launch configurations, tuning parameters, and the source code for the custom upfs_find utility used during the IO500 run.

Certain implementation details, such as internal networking framework components, protocol definitions, and message structures, remain proprietary and are therefore not included in the public package.

Given equivalent hardware resources and access to the same storage software, users should be able to reproduce a similar benchmark environment and achieve comparable performance characteristics.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

The submitted system provides hardware-level fault tolerance in several areas. However, the current deployment contains one known single point of failure: user data is stored using a single-copy architecture without replica-based protection.

A multi-replica implementation derived from our previous-generation product is currently under development but was not enabled in this benchmark configuration.

Power

All storage and client nodes utilize dual redundant power supplies. The failure of a single power module does not interrupt system operation.

Networking

Each client and storage server is equipped with multiple InfiniBand adapters. In the event of an individual NIC failure, network routes are updated automatically and traffic is redirected through remaining operational paths, allowing services to continue without interruption.

Inter-Node Data and Metadata Protection

Metadata and user data are distributed across multiple storage service nodes. However, because the system currently operates with a single-copy data model, a permanent failure of a node containing unique data or metadata may result in data loss. Therefore, the current configuration does contain a storage-level single point of failure.

Intra-Node Storage Protection

Operating system drives on both clients and storage servers are protected using RAID 1. Consequently, the loss of a single system drive does not affect node availability.

Data SSDs used for benchmark storage are not protected by RAID or replication. Failure of a data SSD would result in loss of the data stored on that device.

No additional information is required to determine the existence of single points of failure within this deployment.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

Benchmark Procedure

The IO500 benchmark was executed directly from the first client node in the cluster by running the provided run_io500.sh script.

After benchmark completion:

  1. Relevant benchmark artifacts, including host files, rank files, source code, and configuration files, were collected into a dedicated results directory.
  2. The directory was archived into a compressed package.
  3. A screenshot showing successful execution of the benchmark was captured from the command-line environment.
  4. The compressed archive and accompanying screenshot were uploaded through the official IO500 submission process.

The storage system remained available to internal users during the benchmark period and continued to serve amount of normal business traffic. The IO500 benchmark was executed alongside these background workloads without requiring exclusive access to the storage infrastructure.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

The submitted platform consists of ten HPC client servers connected to ten storage servers.

During the benchmark, application data was ultimately stored on the SSD devices located in the storage servers.

The storage software employs memory resources for normal metadata management and request processing. However, benchmark data volumes significantly exceeded available memory capacity, preventing the benchmark workload from being satisfied entirely from RAM. As a result, benchmark traffic exercised the underlying SSD storage subsystem rather than operating as a memory-resident workload.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

UPFS2 does not function as a cache, burst buffer, gateway, or front-end acceleration tier for another storage system. The storage servers participating in this submission are responsible for both data persistence and metadata management.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

We consider the submitted results to be reliable and reproducible.

The benchmark was executed repeatedly throughout the evaluation process, with more than ten complete runs performed. Across these runs, the observed scores remained highly consistent, providing confidence in the stability of the results.

We also verified that the measured performance remains within the physical capabilities of the deployed hardware platform.

Regarding persistence requirements, the aggregate benchmark data volume substantially exceeded the available memory capacity of the backend storage services. Consequently, benchmark data necessarily reached persistent SSD storage during execution and could not remain entirely buffered in memory. This ensures compliance with the IO500 persistence rules requiring data to be durably stored before benchmark completion.