IO500 - UCloud Compute cluster

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

We are a cloud computing vendor from China called UCloud. The system was independently developed by the company's storage department. Our system is available both internally and externally.

Internally, our system provides storage capacity for other products, such as our UDataArk product which provide second-level backup capability (https://www.ucloud.cn/site/product/udataark.html) and our UPFS product which provide high-performance file storage( https://www.ucloud.cn/site/product/upfs.html).

Externally, Our system supports delivery in the form of an all-in-one machine or separate deployment of the storage system. Up to now, our system has served customers in the fields of large models, big data, EDA, and games.

The system for running the benchmark this time is mainly for internal product use.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The system was deployed in 2024. As of May 10, 2025, it has been available for the rest of the time except for a few server relocations

The common users of the system are other product operation and maintenance personnel within the company, such as udataart and upfs, in order to provide corresponding product services to online customers. The online customers come from all walks of life, including healthcare, big data, large models, games, and so on.

The system has been actively used while running benchmark.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The system is generally used in scenarios such as big data, ai training, and EDA. The generated data is generally used for large model training, chip simulation services, real-time animation rendering, etc.

Storage software is commercial, public cloud version can be found on our website related description.(https://www.ucloud.cn/site/product/upfs.html), we also have a private cloud version, provide delivery of computing and storage or storage delivery separately

Anyone can purchase this software

For this benchmark test, we used two storage servers, and its model is XFUSION - 2288H V7. Related parameters can be found in the website: https://www.xfusion.com/en/product/rack-server/fusionserver-2288h-v7

Each server is equipped with a Mellanox McX755106a-heat network card with an Ethernet rate of 200Gb/s. We have done bonding and used the rdma function.

Each server has 8 SSDS, model is dapustor-J5100, individual capacity is 7.68 terabytes, Related parameters can be found in the website: https://en.dapustor.com/product/12.html

Each server has 2 cpu, model is INTEL_Xeon(R)_4516Y+_48_24_2.2GHz_V10_185W_X86

Our storage program directly manages SSDS and does not act as a file system

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

Commands used to set striping information (either for the entire system or for particular directories)
File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

Because our system is a self-developed commercial system, it is not very convenient for me to provide the system deployment and optimization script. However, it can be reproduced using our public cloud environment: https://www.ucloud.cn/site/product/upfs.html

It is necessary to create a 10-node virtual machine, then create UPFS, mount UPFS on the 10-node, and conduct the io500 test.

Sorry, currently only our data center in Ulanqab, China, is online. You may need to register a Chinese account. The English website only has relevant documents but no resources: https://www.ucloud-global.com/en/docs/upfs/performance

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

Power (For example, can power failure of a node or rack bring down the storage system?)
Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Because it is used for commercial purposes, our system has no single point of failure problem. The specific implementation is as follows

Power: Firstly, each node is powered by multiple power sources to address the issue of a single power failure. Secondly, different nodes are mounted on different racks. If the entire rack loses power and a node is disconnected, the corresponding node will be automatically removed. There is also a backup power supply in the computer room

Networking: Each network card has two network ports. If the network card malfunctions and a node loses connection, the system will also automatically remove that node

Inter storage data and metadata server: We adopt a multi-replica mechanism for both the data and metadata. For example, in this test, we used two servers to form a double replica. If one of the nodes goes down, the system will automatically eliminate it, and the remaining nodes can provide services normally. For some businesses that are relatively strict with data, we will also use 3-copy or EC.

Intra storage data and metadata server storage media: The system disk of the node has been raid 1. SSDS that store data and metadata use multiple replicas and EC mechanisms. Even if one disk fails, another hard disk with the same data will be able to provide services

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

First, we mount UPFS at 10 clients, and then manually execute the following mpi command for testing

mpiexec -np 320 -H 10.0.1.114:32,10.0.1.90:32,10.0.1.92:32,10.0.1.94:32,10.0.1.96:32,10.0.1.98:32,10.0.1.100:32,10.0.1.108:32,10.0.1.110:32,10.0.1.112:32 --mca btl_tcp_if_include eth0 --allow-run-as-root /root/io500/io500 /root/io500/config-full.ini

During the period of IO500 benchmark execution, our system was also providing services for ordinary users, ordinary users and the IO500 benchmark execution shared the system.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

Client data/metadata caching (in Linux page cache or in other memory cache)
Client side NVMe read-only data cache
Storage server metadata/data caching in RAM
Storage controller caching
RAID card caching

1.Client metadata caching: our client uses fuse access, the fuse kernel will have some caches

2.Storage server metadata caching in RAM: We have a small amount of caching for the attributes of the recently accessed files on the metadata server

During the test, we used the odirect parameter. Meanwhile, the storage service directly writes each I/O transaction to the disk, so there will be no data cache at all.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The submitted system is a stand-alone storage system. As a stand-alone system, it offers storage services through standard file system access methods

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

Did you run the benchmark multiple times and get similar scores?
Did you validate the score is below the physical capabilities of the deployed hardware?
How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

We run the benchmark multiple times and get similar scores. There were several times when the scores were higher, but we submitted the relatively average scores, which was more realistic

we validate the score is below the physical capabilities of the deployed hardware.

Our system is a commercial one. To ensure data consistency, no caching is done. Every I/O transaction is directly written to the ssd