Maximus-01

Core42 · Enakta Labs DAOS-TCP · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The submitted system is owned and operated by Core42, a leading provider of AI-driven, high-performance computing (HPC) solutions. This system is designed as a general-purpose, multi-tenant HPC-as-a-Service (HPCaaS) platform, enabling organizations to access scalable, on-demand computational resources without the overhead of managing physical infrastructure.

Typical Applications:

  • AI/ML Training and Inference: Supports large-scale machine learning workloads, including deep learning model training and optimization.
  • Scientific Research: Facilitates computational research in fields such as genomics, climate modeling, and computational chemistry.
  • Industrial Simulations: Enables high-fidelity simulations for engineering, manufacturing, and energy sectors.

General Use and Purpose of Data: The system generates and processes data for a wide range of applications, including:

  • Training datasets for AI/ML models.
  • Simulation outputs for scientific and industrial research.
  • Intermediate and final results for collaborative research projects.

By offering a multi-tenant, cloud-native HPC environment, Core42 empowers researchers, enterprises, and developers to accelerate innovation, reduce time-to-insight, and optimize resource utilization.

Storage backends have multiple options available, with Enakta Labs Platform being one of them.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The submitted system is currently in the final stages of deployment and provisioning, with general availability for customers targeted by the end of 2025. As a on-demand HPC-as-a-Service (HPCaaS) platform, the system is designed for dynamic scalability, allowing users to provision and deprovision resources based on their computational needs. This ensures optimal resource utilization and cost efficiency, while maintaining high availability and performance.

Planned life span is 3+ years.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

Availability: The storage software powering this system is commercially available as part of the Enakta Labs Platform v1.3. This platform is built on an open-source foundation, leveraging the Distributed Asynchronous Object Storage (DAOS) system—a high-performance, scalable storage solution designed for next-generation HPC and AI workloads. DAOS provides the underlying architecture, while Enakta Labs Platform v1.3 delivers enterprise-grade features, protocol access gateways, optimizations, and support.


Accessibility:

  • Commercial Licensing: Organizations can purchase or license the Enakta Labs Platform v1.3 directly from Enakta Labs.
  • Open-Source Foundation: The core DAOS technology is open-source and available for exploration and contribution via its official repository.

Deployment Details

Storage Architecture:

  • Storage Server Product: The system utilizes high-performance, dual-connected storage servers optimized for low-latency, high-throughput workloads.
  • File System: The Enakta Labs Platform v1.3 leverages DAOS as its primary storage engine, which is designed to replace traditional file systems with a more scalable and efficient object-based architecture.

Key Features:

  • Scalability: DAOS is engineered to scale seamlessly across thousands of nodes, making it ideal for exascale computing environments.
  • Performance: Optimized for NVMe, DAOS delivers ultra-low latency and high bandwidth for data-intensive applications.
  • Resilience: Built-in data protection and self-healing capabilities ensure data integrity and system reliability.

This deployment combines the innovation of open-source DAOS with the enterprise robustness of Enakta Labs Platform v1.3, creating a storage solution that is both cutting-edge and production-ready.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

Enakta Labs reproducibility information for this submission is made available here:

https://github.com/enaktalabs/io500_reproducibility

For the 10 node challenge, a total of 10 Dell GPU nodes were used. 

For the network, this was a pure TCP/IP run. RDMA was not being used.

The process was executed via simple openmpi4 / mpirun on bare metal installation.

Enakta Labs Platform v1.3 (based on DAOS 2.6.4) was being run on 164 storage nodes.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Power was fully redundant to every component of the system.

Network was fully redundant non blocking fabric by Arista.

  • Each server used for storage had 2 x 400Gbit Ethernet based on Thor2 NICs (1 per NUMA node)
  • Each server used for clients had 8 x 400Gbit Ethernet based on Thor2 NICs available (4 per NUMA node)

Storage

  • Storage data can be distribued across multiple servers based on user selected configurations.
  • Servers datasets are selected based on fault-domain which is based on a server, so configurations that stipulate multiple data stripes, each component is put on a different server.
  • For this test case, data protection was configured as EC_16P1GX or RP_2GX for bulk data and RP_2G1 for metadata which allows for 1 server failure.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

Executed by simple mpirun job from the primary client across 10 clients via hostfile.

Storage pools were created before the execution and had no existing data

Client nodes and storage system were idle while benchmarking was being done.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

The DAOS File System (DFS) API that was used for the IO500 runs does not perform any caching.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

Enakta Labs Platform is a standalone storage system, and the DAOS dataset used in the IO500 benchmarks is the primary storage in this case.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was run by Enakta Labs staff who have extensive experience with DAOS specifically.
We are also one of the founding members of the DAOS Foundation.

The benchmark was run quite a few times with various tuning changes applied to find the optimal performance.

The results for each stage fall far below the theoretical peaks for any of the hardware components available in the system.

We have confirmed DAOS is able to store data durably between storage restarts across many tests and different installations.