HiPerGator AI

University of Florida · DDN Red · Fully Reproducible

Files
Fully Reproducible

How the reproducibility levels are defined

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

HiPerGator AI is a production DGX A100 SuperPOD with 7x Scalable Units, each with 20x DGX A100 nodes, deployed per the NVIDIA DGX SuperPOD reference architecture.  The system supports the needs of researchers at the Univerity of Florida (UF) and their collaborators,  as well as provide a platform to meet teaching and training needs of UF and institutions across the state of Florida and beyond.  The applications run on the system is very diverse, ranging from custom, community, and  commercial applications and frameworks supporting AI topics including training of large language models and generative AI, to molecular dynamics, computational fluid dynamics, image processing, quantum chemistry, astronomy, and more.  The data on the system includes (but is not limited to) text and image training inputs including data containing protected health informantion and associated outputs, simulation and visualization tool outputs, databases, and more.  HiPerGator AI and the Red storage system are fully integrated with the rest of HiPerGator's networking, storage, computation, and software environment, providing researchers, educators, and students with a single platform to meet cutting edge research needs, as well as those needed to train a workforce that is AI-aware.

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

HiPerGator AI and the Red storage system were delivered in late 2020.  The systems began producing scientific outputs in Jan 2021, completed early commissioning and an early adopter phase in March 2021 which produced the GatorTron clinical Large Language Model to unlock pationt information from electronic health records.  The system was benchmarked and entered production general availability in June 2021.

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

The Red storage system software is DDN EXAScaler, which is a commercially-supported product built on the open-source Lustre filesystem.  The product webpage can be found here: https://www.ddn.com/products/lustre-file-system-exascaler/ .  I believe anyone can purchase this software and support.  Lustre itself is available as open-source software here: https://www.lustre.org/

The Red storage system is composed ten AI400X appliances.  The AI400X is part of the DDN A3I product family (https://www.ddn.com/products/a3i-accelerated-any-scale-ai/).  Each AI400X appliance provides 4 virtual storage servers, eight objest storage targets (OSTs), and one metadata target (MDT).  Each storage server naturally mounts two OSTs, and one of them also mounts the MDT.  Each of the targets is highly available.  Each server has two HDR100 InfiniBand interfaces connected to a dedicated storage fabric that provides connectivity to the DGX A100 SuperPOD.

The Red storage system takes advantage of the following Lustre features - project quota, distributed namespace (DNE), progressive file layout (PFL), multi-rail LNet, and dynamic LNet configuration (on clients).

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

The script to run io500 is included in this submission along with its stdout/stderr, and the full io500 output tarball.  In this submission, openmpi v4.1.6 was used, compiled with gcc 12.2.0.  The io500-10x64-red-16mount.sh script's setup() function contains the directory creation and layout paramters used.  On the DGX A100 clients, SMT is enabled.

In this submission, we use 16 mounts of the same filesystem on each client host, each at a different mountpoint, and invoke io500 via a container. This allows for additional scalability vs a single mountpoint per client.

The individual mountpoints are bind-mounted into the containers to the same directory; thus each container can access the filesystem via different mountpoints on the underlying client host.  The container image used in this submission is for Rocky Linux v9.  The image was pulled with apptainer and saved as a sif file with this command:

apptainer pull docker://rockylinux:9

In io500-10x64-red-16mount.sh, there is a helper script to set up the container bind mounts based upon MPI rank referenced in the mpiargs variable, which is not included in the tarball (apptainer16.sh).  The contents of apptainer16.sh are:

#!/bin/sh

ARGV=($*)
FS=${ARGV[0]}
unset ARGV[0]

MNT_ID=$((OMPI_COMM_WORLD_RANK % 16))

apptainer exec --bind /${FS}_${MNT_ID}:/lustre/${FS} ${ARGV[*]}

On clients, the following non-default settings were applied with this submission:

  • for libcfs kernel module - options libcfs cpu_npartitions=32 cpu_pattern=""
  • network checksums were disabled: lctl set_param mdc.red-*.checksums=0; lctl set_param osc.red-*.checksums=0
  • max OST rpc size set to 1M (lctl set_param osc.red-*.max_pages_per_rpc=1M) - I believe this is actually the default value, though our server config sets it to 16M, so I set it back to default here


On servers, the following settings are in place for production use, and were not changed for the benchmark (some of these may replicate  default values), from exascaler.toml:


[host_defaults.modprobe_cfg.ko2iblnd]
ntx = 5120
peer_credits = 42
concurrent_sends = 84
credits = 2560
ib_mtu = 4096
fmr_pool_size = 1280
fmr_flush_trigger = 1024

[host_defaults.modprobe_cfg.mdt]
max_mod_rpcs_per_client = 128

[host_defaults.modprobe_cfg.ptlrpc]
at_early_margin = 10

[set_param_tunings]
"llite.red-*.max_read_ahead_mb" = 2048
"llite.red-*.max_read_ahead_per_file_mb" = 256
"llite.red-*.max_read_ahead_whole_mb" = 64
"osc.red-*.max_pages_per_rpc" = "16M"
"osc.red-*.max_dirty_mb" = 2048
"osc.red-*.max_rpcs_in_flight" = 128
"osc.red-*.checksum_type" = "crc32c"
"osc.red-*.checksums" = 1
"osd-ldiskfs.red-*.writethrough_cache_enable" = 0
"osd-ldiskfs.red-*.read_cache_enable" = 0
"ost.OSS.ost_io.threads_min" = 128
"obdfilter.red-*.brw_size" = 16
"obdfilter.red-*.precreate_batch" = 1024
"red.sys.jobid_var" = "SLURM_JOB_ID"
"mdd.red*.atime_diff" = 86400
"mdc.red-*.max_rpcs_in_flight" = 128
"mdc.red-*.max_mod_rpcs_in_flight" = 127

[conf_param_tunings]
"red.quota.mdt" = "ugp"
"red.quota.ost" = "ugp"
"red.mdt.root_squash" = '"99:99"'
"red.mdt.enable_remote_dir_gid" = -1

I believe the above notes will allow anyone to reproduce the submitted result with the same equipment and network configuration.

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

The storage system does not have a single point of failure as defined by definition 7 of a Production System:

  • All storage appliances have dual PSUs connected to different rack PDUs
  • All rack PDUs are connected to a fault tolerant power distribution system in the data center, backed by a UPS and a generator
  • All virtual storage servers and DGX A100 clients are dual-connected to different IB switches using different NICs - loss of a single switch or NIC anywhere will not disrupt access to the storage system.
  • Virtual storage server failures are handled by high-availability software (corosync and pacemaker).  In the event of a virtual storage server failure, the failure will be detected and the storage and services running on it will be migrated to other virtual storage servers.
  • All AI400X appliances have two controllers - in the event one fails, services from down virtual storage servers will move to HA partners and the other controller will take over IOs
  • The 24 drives in each AI400X appliance are configured as a single declustered RAID pool, with 9 virtual disks created upon that pool (1x MDT and 8x OST).  The VDs are each configured with double-parity protection (RAID6).  The pool configuration reserves capacity on the pool to allow for rebuilds to begin immediately in the event of a drive failure (does not need to wait for drive replacement).

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

The io500 benchmark was executed interactively on 10 client nodes offline for FW and OS upgrade while the rest of the system was in regular production, though running via a SLURM batch job should produce the same result.  The system was not dedicated to running the benchmark, numerous other jobs (various AI training and molecular dynamics codes, mostly) were observed to be running on the system while the benchmark was run.

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

For ior-easy tests, Direct IO was used, so no client page cache was in use.  The rest of the tests used regular POSIX I/O, so Linux page cache could be used for those.  See the io500 config-10x64-red.ini included in the submission for verification.

Client side NVMe read-only data cache was not configured or used.

Virtual storage server settings writethrough_cache_enable and read_cache_enable settings were disabled, as is normal for this system in production.  Virtual storage server buffer caches were otherwise permitted to cache what Linux would allow.

Storage controller caching was enabled on the virtual disks as follows:
  Cache Settings:
    ReACT:                   TRUE
    Read Ahead:              FALSE
    Write Back:              TRUE
    Mirroring:               TRUE

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

The system is a stand-alone storage system.  It is not a burst buffer layered on primary storage.  All clients natively mount the storage as Lustre.

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?

The benchmark was run several times with little variation in the scores.  Using 8 mounts per client produced a very similar result.

The vendor provided guidance to the maximum throughput capabilities of the system, and I validated the score was below these figures, as expected.  In monitoring, I could see the throughput capability of the system could be reached, but not exceeded.

The storage system is a POSIX-compliant, cache-coherent parallel filesystem with global lock management and flock enabled.  I did not verify persistence personally, but the Lustre filesystem is known to comply with io500 rules regarding persistence.

Note that the IO500 rules link is giving me a 404.  I think it may need to be updated to https://io500.org/rules/submission?