Aurora

Argonne National Laboratory · Intel Aurora DAOS · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

Aurora is Argonne National Labratory's high performance super-computer for open science computing deployed within the Argonne Leadership Computing Facility (ALCF). The Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science User Facility located at Argonne National Laboratory, enables breakthroughs in science and engineering by providing supercomputing resources and expertise to the research community.

ALCF computing resources are available to researchers from academia, industry, and government agencies, supporting large-scale, computationally intensive projects and are aimed at solving some of the world's most complex and challenging scientific problems. Through competitive awards (INCITE, ALCC) of computing time and support services, the ALCF enables its users to accelerate the pace of scientific discovery and innovation across disciplines.

Aurora features several technological innovations, including a revolutionary I/O system—the Distributed Asynchronous Object Store (DAOS)—to support new types of workloads. Aurora is built on a future generation of Intel Xeon Scalable processor accelerated by Intel’s Xe compute architecture. Slingshot 11 fabric and HPE Cray EX supercomputer platform form the backbone of the system. The system is highly optimized across multiple dimensions that are key to success in simulation, data, and learning applications.

More information can be found at  https://www.alcf.anl.gov/aurora.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

Aurora and the associated DAOS storage system is available to scientific researchers worldwide via three different allocations programs:
  * The Innovative and Novel Computational Impact on Theory and Experiment (INCITE)
  * ASCR Leadership Computing Challenge (ALCC)
  * Director's Discretionary


Aurora is expected to be generally available from 2025 through 2030.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The Aurora DAOS storage system is based on the open source DAOS storage software publicly available on GitHub. The software is free to use and available under the BSD-2-Clause Plus Patent License.

Aurora is built on Intel Coyote Pass platform using 800 servers for the daos_user volume.

  • 2x Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz
  • 16x 32GB DDR4
  • 16x 512GB Intel Optane 200 DCPM
  • 16x 15.3TB Samsung PM1733 or Solidigm D7-P5520
  • 2x HPE Slingshot NIC

Aurora compute system are Aurora Exascale Compute Blades.

  • 2x Intel(R) Xeon(R) CPU Max 9470C
  • 6x Intel Data Center GPU Max
  • 16x 64GB DDR5
  • 8x HPE Slingshot NIC

The Aurora storage system is connected directly to the same high speed interconnect as the compute system. The Aurora DAOS storage system is available to any user on the Aurora system. Users are provided storage allocations based on their computing allocation requests.

More information about the Aurora DAOS deployment can be found at: https://docs.alcf.anl.gov/aurora/data-management/daos/daos-overview/

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

The Aurora daos_user volume is configured with 800 servers as described above and tested with 4096 Aurora compute nodes. DAOS clients will use all 8 NICs provided by each client for a total of 32,768 client endpoints. The DAOS servers are configured to run two IO engines on each server for a total of 1600 daos server engines. The underlying storage is configured by the `daos scm prepare` and `daos storage scan` as documented in the DAOS Adminstration Guide.

The code is built with the Intel icc compiler, icc (ICC) 2021.9.0 20230201, and uses the Aurora MPICH MPI.

The Aurora MPICH is built from the open source MPICH - https://github.com/pmodels/mpich.

The daos code used is 2.6.5-p2, https://github.com/daos-stack/daos/tree/v2.6.5.p2

The io500 script is executed with these environment variables.

FI_CXI_OFLOW_BUF_COUNT=3
FI_CXI_DEFAULT_CQ_SIZE=131072
MPIR_CVAR_ERROR_CHECKING=1
NA_OFI_CXI_PROTO_RNR=0
CCL_CONFIGURATION=cpu_gpu_dpcpp
FI_CXI_DEFAULT_TX_SIZE=1024
DAOS_AGENT_DRPC_DIR=/run/daos_agent_oneScratch
FI_CXI_RX_MATCH_MODE=hybrid
FI_PROVIDER=cxi,tcp;ofi_rxm
MPIR_CVAR_CH4_OFI_MAX_NICS=1
MPIR_CVAR_CH4_OFI_EAGER_THRESHOLD=1000000
FI_MR_CACHE_MONITOR=kdreg2
FI_CXI_REQ_BUF_MIN_POSTED=6
MPIR_CVAR_PROGRESS_TIMEOUT=600
FI_MR_CACHE_MAX_COUNT=2000000
FI_MR_CACHE_MAX_SIZE=-1
FI_CXI_REQ_BUF_SIZE=12582912
FI_CXI_RDZV_EAGER_SIZE=2048
FI_CXI_OFLOW_BUF_SIZE=12582912

The environment variables above are set by the following modules on Aurora.

> module list

Currently Loaded Modules:
  1) gcc/14.3.0                            4) libfabric/2.3.1      7) daos/base
  2) oneapi/release/2026.1.0               5) cray-pals/1.8.0
  3) mpich/prd/5.0.0.aurora_test.87e2045   6) cray-libpals/1.8.0
  

The pool used for the test run using all available servers.

Pool 15cbbe92-1d66-42d1-a1fa-7abc4e51723f, ntarget=25536, disabled=64, leader=1569, version=201, state=Degraded
Pool health info:
- Rebuild done, 0 objs, 0 recs
Pool space info:
- Target(VOS) count:25472
- Storage tier 0 (SCM):
  Total size: 1.5 PB
  Free: 1.1 PB, min:44 GB, max:46 GB, mean:44 GB
- Storage tier 1 (NVMe):
  Total size: 48 PB
  Free: 30 PB, min:1.2 TB, max:1.3 TB, mean:1.2 TB
  

The container configuration used:

daos cont create $DAOS_POOL $DAOS_CONT --type=POSIX --properties=cksum:crc32,srv_cksum:on,rd_fac:1,ec_cell_sz:1mib

daos cont query $DAOS_POOL $DAOS_CONT
  Container UUID              : 572c1b79-af7e-4bb7-baa1-f24fce3a2fe4
  Container Label             : io500
  Container Type              : POSIX
  Pool UUID                   : 15cbbe92-1d66-42d1-a1fa-7abc4e51723f
  Container redundancy factor : 1
  Number of open handles      : 2
  Latest open time            : 0x28ca0dd059740000 (2026-10-28 03:22:35.957362688 +0000 UTC)
  Latest close/modify time    : 0x28ca36850a980003 (2026-10-28 04:09:13.240559616 +0000 UTC)
  Number of snapshots         : 0
  Object Class                : UNKNOWN
  Dir Object Class            : RP_2G1
  File Object Class           : EC_16P1G1383
  Chunk Size                  : 32 MiB

server.yaml

access_points:
- aurora-daos-0022
- aurora-daos-0154
- aurora-daos-0410
- aurora-daos-0538
- aurora-daos-0642
- aurora-daos-0771
- aurora-daos-0897
- aurora-daos-0127
- aurora-daos-0252
control_log_file: /mnt/log/daos/daos_control.log
control_log_mask: INFO
crt_timeout: 120
enable_hotplug: true
engines:
- env_vars:
  - DAOS_PIPELINE=1
  - DAOS_MD_CAP=1024
  - SWIM_PROTOCOL_PERIOD_LEN=3000
  - SWIM_PING_TIMEOUT=900
  - SWIM_SUSPECT_TIMEOUT=60000
  - SWIM_TRAFFIC_CLASS=low_latency
  - FI_CXI_RX_MATCH_MODE=hybrid
  - FI_CXI_REQ_BUF_SIZE=8388608
  - FI_CXI_OFLOW_BUF_SIZE=8388608
  - FI_CXI_REQ_BUF_MIN_POSTED=8
  - FI_MR_CACHE_MONITOR=disabled
  - FI_CXI_RDZV_THRESHOLD=20480
  - FI_CXI_DEFAULT_TX_SIZE=4096
  - D_POST_INIT=128
  - D_MRECV_BUF=128
  - D_MRECV_BUF_COPY=4
  - CRT_EVENT_DELAY=20
  - D_LOG_FILE_APPEND_PID=1
  - DAOS_POOL_RF=2
  - D_LOG_FILE_APPEND_RANK=1
  - NA_OFI_CXI_PROTO_RNR=0
  - FI_CXI_OPTIMIZED_MRS=0
  - NA_OFI_SKIP_DOMAIN_OPS=1
  - DAOS_SCHED_MONITOR_KILL=0
  - HG_LOG_LEVEL=warn
  - HG_LOG_SUBSYS=hg,na,libfabric
  - DAOS_SCHED_MONITOR_KILL=0
  fabric_iface: hsn0
  fabric_iface_port: 317
  log_file: /mnt/log/daos/daos_engine0.log
  log_mask: INFO
  nr_xs_helpers: 4
  pinned_numa_node: 0
  storage:
  - class: dcpm
    scm_list:
    - /dev/pmem0
    scm_mount: /var/lib/daos_server/scm/pmem0
  - bdev_list:
    - 0000:4a:00.5
    - 0000:64:00.5
    class: nvme
  targets: 16
- env_vars:
  - DAOS_PIPELINE=1
  - DAOS_MD_CAP=1024
  - SWIM_PROTOCOL_PERIOD_LEN=3000
  - SWIM_PING_TIMEOUT=900
  - SWIM_SUSPECT_TIMEOUT=60000
  - SWIM_TRAFFIC_CLASS=low_latency
  - FI_CXI_RX_MATCH_MODE=hybrid
  - FI_CXI_REQ_BUF_SIZE=8388608
  - FI_CXI_OFLOW_BUF_SIZE=8388608
  - FI_CXI_REQ_BUF_MIN_POSTED=8
  - FI_MR_CACHE_MONITOR=disabled
  - FI_CXI_RDZV_THRESHOLD=20480
  - FI_CXI_DEFAULT_TX_SIZE=4096
  - D_POST_INIT=128
  - D_MRECV_BUF=128
  - D_MRECV_BUF_COPY=4
  - CRT_EVENT_DELAY=20
  - D_LOG_FILE_APPEND_PID=1
  - DAOS_POOL_RF=2
  - D_LOG_FILE_APPEND_RANK=1
  - NA_OFI_CXI_PROTO_RNR=0
  - FI_CXI_OPTIMIZED_MRS=0
  - NA_OFI_SKIP_DOMAIN_OPS=1
  - DAOS_SCHED_MONITOR_KILL=0
  - HG_LOG_LEVEL=warn
  - HG_LOG_SUBSYS=hg,na,libfabric
  - DAOS_SCHED_MONITOR_KILL=0
  fabric_iface: hsn1
  fabric_iface_port: 417
  log_file: /mnt/log/daos/daos_engine1.log
  log_mask: INFO
  nr_xs_helpers: 4
  pinned_numa_node: 1
  storage:
  - class: dcpm
    scm_list:
    - /dev/pmem1
    scm_mount: /var/lib/daos_server/scm/pmem1
  - bdev_list:
    - 0000:97:00.5
    - 0000:e2:00.5
    class: nvme
  targets: 16
helper_log_file: /mnt/log/daos/daos_admin.log
name: daos_server
port: 10001
provider: ofi+cxi
socket_dir: /var/run/daos_server
telemetry_port: 9191
transport_config:
  allow_insecure: false
  ca_cert: /etc/daos/certs/daosCA.crt
  cert: /etc/daos/certs/server.crt
  client_cert_dir: /etc/daos/certs/clients
  key: /etc/daos/certs/server.key


io500-config.ini

[global]
datadir = /datafiles
timestamp-datadir = TRUE
resultdir = /lus/flare/projects/Aurora_deployment/mschaara/results
timestamp-resultdir = TRUE
api = DFS --dfs.pool=daos_io500 --dfs.cont=io500
drop-caches = FALSE
drop-caches-cmd = sudo -n bash -c "echo 3 > /proc/sys/vm/drop_caches"
io-buffers-on-gpu = FALSE
verbosity = 1
scc = FALSE
dataPacketType = timestamp

[debug]
stonewall-time = 300
pause-dir = 

[ior-easy]
API = DFS --dfs.pool=daos_io500 --dfs.cont=io500 --dfs.dir_oclass=RP_2G1 --dfs.oclass=EC_16P1G32 --dfs.chunk_size=16777216
transferSize = 16m
blockSize = 99200000m
filePerProc = TRUE
uniqueDir = FALSE
run = TRUE
verbosity = 

[ior-easy-write]
API = 
run = TRUE

[mdtest-easy]
API = DFS --dfs.pool=daos_io500 --dfs.cont=io500 --dfs.dir_oclass=RP_2G32 --dfs.oclass=RP_2G1
n = 10000000
run = TRUE

[mdtest-easy-write]
API = 
run = TRUE

[mdworkbench]
API = 
waitingTime = 0.0
precreatePerSet = 
filesPerProc = 
run = TRUE
verbosity = 

[mdworkbench-create]
run = TRUE

[timestamp]

[find-easy]
external-script = 
external-mpi-args = 
external-extra-args = 
nproc = 
run = TRUE
pfind-queue-length = 10000
pfind-steal-next = FALSE
pfind-parallelize-single-dir-access-using-hashing = FALSE

[ior-hard]
API = DFS --dfs.pool=daos_io500 --dfs.cont=io500 --dfs.dir_oclass=RP_2G1 --dfs.oclass=EC_16P1GX --dfs.chunk_size=4700800
segmentCount = 10000000
collective = 
run = TRUE
verbosity = 

[ior-hard-write]
API = 
collective = 
run = TRUE

[mdtest-hard]
API = DFS --dfs.pool=daos_io500 --dfs.cont=io500 --dfs.dir_oclass=RP_2GX --dfs.oclass=RP_2G1
n = 10000000
files-per-dir = 
run = TRUE

[mdtest-hard-write]
API = 
run = TRUE

[find]
external-script = 
external-mpi-args = 
external-extra-args = 
nproc = 
run = TRUE
pfind-queue-length = 10000
pfind-steal-next = FALSE
pfind-parallelize-single-dir-access-using-hashing = FALSE

[find-hard]
external-script = 
external-mpi-args = 
external-extra-args = 
nproc = 
run = TRUE
pfind-queue-length = 10000
pfind-steal-next = FALSE
pfind-parallelize-single-dir-access-using-hashing = FALSE

[mdworkbench-bench]
run = TRUE

[ior-easy-read]
API = 
run = TRUE

[mdtest-easy-stat]
API = 
run = TRUE

[ior-hard-read]
API = 
collective = 
run = TRUE

[mdtest-hard-stat]
API = 
run = TRUE

[mdworkbench-delete]
run = TRUE

[mdtest-easy-delete]
API = 
run = TRUE

[mdtest-hard-read]
API = 
run = TRUE

[mdtest-hard-delete]
API = 
run = TRUE

[ior-rnd4K-easy-read]
API = DFS --dfs.pool=daos_io500 --dfs.cont=io500 --dfs.dir_oclass=RP_2G1 --dfs.oclass=EC_16P1G32
run = TRUE

A customized version of pfind is used that is optimized for DAOS: https://github.com/mchaarawi/pfind/tree/dfs_filter

This data should be available in daos reproducibility repo once the results are published.
https://github.com/daos-stack/daos-reproducibility/

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

The Aurora storage system hardware and software is able to provide full fault tolerance at various levels of redundancy depedending on the component. 

  • Power
    • ALCF has a dual-feed from Commerial and UPS power allowing either feed to fail but the system to continue operating.
    • Storage servers have 2+1 power supplies allowing a single power supply failure on any server, or rack PDU failure.
  • Networking
    • Each storage server is connected to two different switches, however, each DAOS engine is connected to a single NIC. If a NIC fails, the IO engine is unreachable and the system will reconstruct from another server and then data will be fully replicated again.
  • Storage
    • Storage data can be distribued across multiple servers based on user selected configurations.
    • Servers datasets are selected based on fault-domain which is based on a server, so configurations that stipulate multiple data stripes, each component is put on a different server.
    • For this test case, data protection was configured as EC_16P1G32 or EC_16P1GX, RP_2G1 for bulk data and RP_2G1 ot RP_2GX for metadata which allows for 1 server failure.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The benchmark was run using the PBS scheduler via an interactive job using stlightly more than 4096 nodes. The io500 runs then used 4096 compute nodes from the set available to the job. The DAOS storage system and pool was configured prior to the execution and contained no existing data.

The Aurora system was not idle during the io500 run and continuted running production jobs on the other ~6000 free compute nodes. Other compute jobs were not able to utilize the daos_user volume during this period but users could still access data on daos_user via the login nodes.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

The DAOS storage system utilizes Optane Persistent Memory and standard NVMe SSDs. These devices may have internal caching that is not exposed to the storage software. The DAOS server does not explicitly cache any data in the server side RAM. The metadata on the server side is contained wholly within the Optane PM and so can be thought of as "always in memory" but is always persistent. The DAOS File System (DFS) API that was used for the IO500 runs does not perform any caching on the client side.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

DAOS is a complete storage system and wholly contains its data. The storage system has two types of storage media, Optane Persistent Memory and NVMe SSD drives. DAOS metadata is contained in Optane and bulk data is kept in SSD. Small writes can land in Optane but will always be fully visible upon read and write.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was run by HPE staff who have extensive experience in running and assessing the benchmark. The benchmark was run at smaller scale to prepare and previously subcomponents have been run to help understand the system performance prior to the full run. When running at scale (4096 client nodes, 800 servers, 12800 SSDs), any hardware failure occuring during the run requires the full test to be restarted, thus their isn't a budget of time to run the benchmark in full multiple times due to the runtime duration requirements. The performance capabilities of the components are well understood and performance lies within the performance of those components. DAOS provides persistent durable storage, the benchmark is run on our production DAOS volume, daos_user. This volume holds 10s of PB of production user data and has been restarted many times demonstrating its persistence and durability.