Samsung AE-25 Cluster

Samsung Electronics · Hammerspace Hyperscale NAS · Fully Reproducible

Files
Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The AE-25 system is used by Samsung’s Memory Division to evaluate various high-performance memory products in an environment similar to actual production conditions and to process and store logs and dumps generated during analysis.

It provides high-speed, shareable storage resources within a large-scale computing environment that handles massive data generated during benchmarking and simulation tasks essential for next-generation memory product development, including SSDs, DRAM, and CXL.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

The AE-25 system was established in 2025 and operates 24/7 year-round, except during scheduled preventive maintenance periods.

It is accessible to authorized users within Samsung’s Memory Division and provides services to researchers involved in memory product analysis and development, as well as engineers from external partner companies.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

Hammerspace software is commercially available for purchase by anyone through authorized channel partners. It is also available for purchase in the Amazon AWS, Microsoft Azure, Google Cloud, and Oracle OCI cloud marketplaces.

Extensive information regarding Hammerspace is available on the company website: https://hammerspace.com/

Hammerspace is the high-performance, standards-based data platform for AI anywhere. It unifies unstructured data across sites and clouds and feeds GPUs at full speed with Tier 0 performance. Built on open standards (Linux, NFSv4.2/pNFS, SMB, S3), Hammerspace turns existing infrastructure into a high-performance, unified AI data plane. Objective-based policies orchestrate data to compute, and context-enriched metadata triggers file-granular services for data placement, protection, and activation.

The architecture for this benchmark run consisted of:

·         Anvil Nodes

o   Role: Metadata Servers

o   Qty 2, High-Availability Configuration

o   Kernel 6.12.24-22.hs.121.el8.x86_64

o   Hammerspace 5.2.3-90

·         DSX Data Services Nodes

o   Roles: Storage Server, NFS/SMB/S3 Portal, Data Mover

o   Qty 10

o   Kernel 6.12.53-2.hs.149.el8.x86_64

o   Hammerspace 5.2.3-90

o   Storage: 12x 15.36 TB NVMe Samsung Model PM1753 SSDs

·         Client Nodes

o   Role: Benchmark Clients

o   Qty 10

o   Ubuntu Linux 24.04.2 LTS

o   Kernel 6.14.0-33-generic

o   HPC-X Version: hpcx-v2.22-gcc-doca_ofed-ubuntu24.04-cuda12-x86_64

§  Provides Open MPI Version: 4.1.7rc1

§  Provides UCX Version: 1.18.0

o   NVIDIA Network Driver: OFED-internal-25.07-0.9.7

o   NVIDIA DOCA Version: doca-ofed 3.1.0-091000

o   No Hammerspace code required or installed on clients

·         Network (all nodes)

o   2x 400Gb NDR Infiniband

§  IPoIB configured

o   2x NVIDIA CX-7 single-port adapters per node

o   RDMA enabled

·         Network Switch

o   NVIDIA MQM9700-NS2F

o   MLNX-OS release 3.11.1004

Hammerspace aggregates the storage served by all DSX nodes and presents a single unified NFS share to Clients. Client machines mount the Hammerspace NFS share with NFSv4.2 using the standard Linux NFS client software. Clients use standard parallel NFS (pNFS) to access the storage servers directly under the direction of the Anvils. Total usable capacity as configured was 1.84PB.

Please review the submitted system configuration for more details.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

Anvil Node Optimizations:

  • allow_open_layout_get = False
  • preallocated_inst_runway = 300000L
  • maxInitialNASInstances = 10
  • DRBD replication reconfigured to use RDMA

DSX Node Optimizations:

  • Set SunRPC Pool Mode to pernode:
    • options sunrpc pool_mode=pernode
  • Enable NFSD_IO_DIRECT:
    • echo 2 > /sys/kernel/debug/nfsd/io_cache_write
    • echo 2 > /sys/kernel/debug/nfsd/io_cache_read
  • Set read_ahead to 1MB to match NFS native transport size:
    • blockdev --setra 2048 /dev/nvme*n1
  • Set default fallback handling of unaligned IO head and tail to buffered
  • Disable CPU Hyperthreading
  • Increase RDMA slot table entries and svc rpc per-connection limits:
    • echo 256 > /sys/module/sunrpc/parameters/svc_rpc_per_connection_limit
    • echo 256 > /proc/sys/sunrpc/rdma_slot_table_entries
  • Increase RDMA max inline write:
    • echo 1024 > /proc/sys/sunrpc/rdma_max_inline_write
  • Format NVMe drives to 4KB native

Client Node Optimizations:

  •        Mount the Hammerspace NFS share with the option “nconnect=12”
  • Disable CPU Hyperthreading
  • Increase RDMA slot table entries:
    • echo 192 > /proc/sys/sunrpc/rdma_slot_table_entries
  • Increase RDMA max inline read and write:
    • echo 1024 > /proc/sys/sunrpc/rdma_max_inline_read
    • echo 1024 > /proc/sys/sunrpc/rdma_max_inline_write
  • Set tuned profile to throughput-performance
    • tuned-adm profile throughput-performance

Network Optimizations: None

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Anvil Nodes: The two Anvil nodes are clustered, and the failure of one will not bring down the storage system. All services will continue to operate with a single Anvil. Within each Anvil, NVMe boot drives are mirrored in hardware (Dell BOSS). Metadata is striped (RAID0) across twelve NVMe drives. All metadata is replicated between Anvils.

DSX Nodes: The ten DSX nodes operate as independent storage servers, unified by Hammerspace into a global namespace. The failure of a DSX node will not bring down the storage system, but simply make the affected storage volumes unavailable until the node is recovered.

In order to protect against DSX failure, pNFS client-side mirroring (CSM) may be used. For this IO500 run only the IOR-HARD tests used CSM.

Data resiliency can also be achieved using Hammerspace data orchestration to maintain a resilient copy within an object storage system or additional resilient NAS tier.

Networking: Single switch, two single-port adapters per node.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The IO500 benchmark was run without scheduler involvement using a bash script facilitated by NVIDIA HPC-X Open MPI with DOCA UCX.

During the IO500 benchmark run no other jobs were running on the system.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

On the Client O_DIRECT was used for all IO.  

In pNFS with NFSv4.2, the metadata server (Anvil) maintains authoritative control over file metadata, such as layout information and file attributes, while clients can cache metadata locally to reduce round trips. Clients use delegations and stateids to determine cache validity, but any conflicting access (e.g., from another client) causes the Anvil to recall those delegations, forcing clients to revalidate or purge stale entries. NFSv4.2 adds enhancements like READ_PLUS and SEEK to optimize data and metadata access, but strict coherency is still enforced through callback mechanisms and lease expiration. pNFS does not acknowledge writes until data is made persistent on disk. 

On the DSX Nodes, page cache was used on the head and tail of all unaligned IO (IOR-HARD, MDTEST-HARD). For aligned IO and aligned portions of unaligned IO, the new NFSD_IO_DIRECT feature enabled O_DIRECT.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The submitted system is a stand-alone primary storage system. This style of deployment is one example of what Hammerspace refers to as “Tier 1” storage.

Hammerspace also enables the creation of a global namespace that spans multiple storage systems. For example, the submitted deployment could be combined with one or more existing third-party NAS systems (which could reside at different sites) and cloud storage to form a unified namespace. Hammerspace data orchestration services are used to place and protect data in such environments.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The submitted run is an accurate sample of the many iterations that were run. Run-to-run variation for identical configurations was less than 2.5 points of the IO500 Score. Here we have provided a series of additional run scores to help illustrate the stability of the performance.


2025.11.03-20.52.05/result_summary.txt:[SCORE ] Bandwidth 76.721301 GiB/s : IOPS 92.959314 kiops : TOTAL 84.450930
2025.11.04-15.42.35/result_summary.txt:[SCORE ] Bandwidth 71.178112 GiB/s : IOPS 97.271072 kiops : TOTAL 83.208000
2025.11.04-19.31.36/result_summary.txt:[SCORE ] Bandwidth 74.913402 GiB/s : IOPS 93.940143 kiops : TOTAL 83.889067
2025.11.04-20.48.23/result_summary.txt:[SCORE ] Bandwidth 73.209706 GiB/s : IOPS 95.077875 kiops : TOTAL 83.430350
2025.11.05-05.09.18/result_summary.txt:[SCORE ] Bandwidth 74.089823 GiB/s : IOPS 95.870864 kiops : TOTAL 84.279626
2025.11.05-06.29.04/result_summary.txt:[SCORE ] Bandwidth 71.920185 GiB/s : IOPS 95.963780 kiops : TOTAL 83.076668
2025.11.05-16.08.46/result_summary.txt:[SCORE ] Bandwidth 71.869874 GiB/s : IOPS 96.432628 kiops : TOTAL 83.250230
2025.11.05-17.38.15/result_summary.txt:[SCORE ] Bandwidth 76.191388 GiB/s : IOPS 93.451757 kiops : TOTAL 84.381391

The 12x NVMe drives in each DSX node are capable of, in aggregate, 168 GB/s of Read and 120 GB/s of write throughput while the configured network allows for up to 1TB/s. Our io500s results are less then 1TB/s for IOR-EASY, indicating we still have room to improve with this hardware.


pNFS requires all writes to be persisted before being acknowledged.