Hammerspace Performance Lab - Tier0

Hammerspace · Hammerspace pNFS · Proprietary

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

The submitted system is the Hammerspace Performance Lab Tier 0 cluster, used for research and test. It runs Hammerspace on 10 data/client nodes and 2 metadata nodes, with each tier0 node acting as both a compute client and data server, to measure Hammerspace storage performance. This submission runs a research (pre-release) software stack, not the generally available Hammerspace release. It incorporates client-side striping, client kernel performance enhancements, and future metadata node software. These innovations are currently under active development, with plans for future integration into the Linux kernel and upcoming Hammerspace software releases.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

This system is used for weeks to months at a time to meet project demands like the IO500. Most users of this system work at Hammerspace.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

Software: Hammerspace (vendor: Hammerspace).

Product page: https://hammerspace.com/

Software availability: Hammerspace is commercially available. However, this run used pre-release components that cannot currently be purchased: (1) client-side striping and NFS client kernel changes, which are being upstreamed to the Linux kernel and are planned to ship with the upcoming Hammerspace release; and (2) research metadata node software.

Deployment details:

  • Data servers / clients: 

    • 10x Supermicro SYS-122H-TN (plsm122h-03 to -12), 

    • Rocky Linux 10.2. 

    • Each node has 2x Intel Xeon 6737P (Granite Rapids, 32 cores per socket, 2.9 GHz, SMT off), 512 GiB DDR5 ECC, 

    • 2x 400 Gb Ethernet links on Mellanox ConnectX-7 (MCX75310AAS-NEAT) with RDMA.

  • Data media per node: 

    • 6x Samsung PM1753 (MZWL97T6HFLA-00AW7) NVMe SSD and 6x ScaleFlux CSD5000 (CSDU7UVG76) NVMe SSD, each formatted XFS. 

    • 768 TiB usable across the cluster.

  • Metadata nodes: an active-active pair of experimental Hammerspace Anvils on

    • 2x Supermicro SYS-122H-TN (plsm122h-25/26)

    • Rocky Linux
    • same CPUs and NICs, 1 TB DDR5 ECC,

    • 3x Samsung PM1753 used as raw NVMe (no file system) for the SPDK blobstore metadata store, plus 1x Samsung PM1753 with XFS.

  • Client access: NFS v4.2 (pNFS flex files) over RoCE. Clients and data servers are the same hosts, so client and server cost cannot be separated.

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
nfsd threads 640
nfsd nice −10, persistent via nfs-server.service.d/20-nfsd-nice.conf → /usr/local/sbin/hs-nfsd-nice.sh
max_block_size 1048576 (1 MiB — deliberate, RPC perf; do not revert)
pools 64
hs-nfsd-tuning env IO_CACHE_WRITE=4 IO_CACHE_READ=2 PIN_OFFSET=0 WANT_POOL_MODE=percpu
preempt=full on cmdline + /etc/kernel/cmdline; runtime confirms (full) active
dataserver_nconnect

32 live, and 32 in /etc/modprobe.d/io500-dsnc.conf (baked into nfs6 initramfs)

mkfs.xfs -f -l size=256m -d agcount=1024 -m rmapbt=1,reflink=1
mount noatime,logbsize=256k,nodiscard,nofail,x-systemd.device-timeout=30s
extsize 5644288 with EXTSZINHERIT (xflags 0x1000) on all 240 roots
LBA format 4096 (flbas preserved through nvme format)

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Metadata: two Metadata Nodes run as an active-active pair, replicated with Raft, so losing one metadata server does not take the file system offline.

Networking: every node has two 400 Gb Ethernet links, so a single NIC failure does not take a node offline. 

Data: this is a Tier 0 scratch configuration (IE single instance) with replication options available with Client Side Mirrioring. The IOR-Hard file is striped across the local NVMe of all 10 nodes with no erasure coding, RAID, or HA enclosures. 

Power: nodes have no shared power protection beyond their own power supplies.

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The benchmark ran across 10 nodes with 120 processes per node (1,200 processes total). The client processes ran on the same 10 hosts that serve as data servers. The system was dedicated entirely to the IO500 benchmark during the run, with no other jobs on the cluster or storage. Custom bash scripts were used for orchestration of runs. 

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

On the Client O_DIRECT was used for all IO. O_DIRECT was used on every measured phase, so clients bypassed the Linux page cache for data I/O.

The NFS servers were configured with NFSD_IO_DIRECT enabled.  The moves all server side IO into an O_DIRECT path as well.

In pNFS with NFSv4.2, the metadata server (MDS) maintains authoritative control over file metadata, such as layout information and file attributes, while clients can cache metadata locally to reduce round trips. Clients use delegations and state ids to determine cache validity, but any conflicting access (e.g., from another client) causes the MDS to recall those delegations, forcing clients to revalidate or purge stale entries. NFSv4.2 adds enhancements like READ_PLUS and SEEK to optimize data and metadata access, but strict coherency is still enforced through callback mechanisms and lease expiration. The metadata servers utilized atomic intent logs combined with in memory inode and dirent caches.

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The submitted system is standalone primary storage. Data was written directly to NVMe inside the 10 nodes, with no burst buffer layer and no backing file system or archive tier behind it during the benchmark.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The IO500 run provided is representative of the many runs performed on this system. Across
three runs in an identical configuration the scores were 682.432, 684.941 and 688.006 — a
mean of 685.126 with a sample standard deviation of 2.79 points, a coefficient of variation
of 0.41%. Run-to-run variation in an identical configuration is within ±3 IO500 points.
Bandwidth is tighter still: 449.09, 449.63 and 449.86 GiB/s, a coefficient of variation of
0.09%, so what variation there is comes almost entirely from the metadata phases.


rqSbMM [SCORE ] Bandwidth 449.628136 GiB/s : IOPS 1043.404006 kiops : TOTAL 684.940726
fYVQ57 [SCORE ] Bandwidth 449.864792 GiB/s : IOPS 1052.209439 kiops : TOTAL 688.005799
pdJXju [SCORE ] Bandwidth 449.091511 GiB/s : IOPS 1037.011430 kiops : TOTAL 682.431703

Against the physical capability of the hardware. Each node has 12 NVMe drives and two
400 Gbit Ethernet links, giving 100 GB/s per node and 1 TB/s across the ten nodes. This is a
Tier0 configuration in which the ten client nodes are also the ten data servers, so a share
of every workload is served from local NVMe and never crosses the network.

ior-easy-read measured 873.98 GiB/s, which is 938.4 GB/s, or 94% of aggregate line rate.
Read bandwidth is therefore bounded by the network rather than by the media.

ior-easy-write measured 1113.42 GiB/s, which is 1195.5 GB/s, or 120% of aggregate line rate.
A figure above line rate is only reachable because writes are affinitized to local NVMe, so a
substantial fraction is persisted on the writing node and never traverses the fabric.

Both results sit below what the deployed media can sustain; the limit in each case is the
network path, not the drives.

Persistence. Every measured phase ran with O_DIRECT (`api = POSIX --posix.odirect` in
config.ini, and on twelve of the thirteen command lines in result.txt — the exception is the
find phase, which is an external script and takes no IOR API), so no write was buffered in
client page cache. The IOR write phases carry `-e`, which fsyncs before close with the fsync
inside the measured time, and the mdtest write phases carry `-Y`, which fsyncs after each
write phase. IOR reports bandwidth from bytes actually moved to the files rather than from
its own accounting. Under pNFS the server acknowledges the close only once data is on stable
storage. Cache behaviour materially affects write latency and was monitored throughout the
build-up to this run using Prometheus and Grafana.