PanEast ActiveStor Flash

Panasas · Panasas pgh-realm-65-244 · Fully Reproducible

System Purpose

Question asked

Please describe the purpose and general usage of the submitted system.  This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.

This in an internal Production class system used here at Panasas for Performance class work.   Benchmarking, software develompent and testing are it's comon tasks.  

Availability

Question asked

Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.

Please describe the availability of the system to users and who are its set of most regular users.

This system is over 1.5 years old and usd daily for internal tasks.   This is used by several Internal uses. 

Storage System Software

Question asked

How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.

Can anyone download/purchase this software?

List either product webpage or open-source repo of the exact software used in IO500.

Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).

This system is used to run a wide varity of HPC applictions and benchmarks.  CFD, DOD and LifeScience are some of our top uses. 

This is Commercial software shared with our Customers only.   The software used will be apart of our upcomming Panfs10 software release. 

https://www.panasas.com/products/activestor-flash/ is a link to it's product page.  This is a SuperMicro system with 6x PCIv3 NVMe Drives, and M2 NVMe for BTRFS metadata and presistant memroy used as a write cache. 

Runtime Environment

Question asked

State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.

NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.

Several examples include:

  • Commands used to set striping information (either for the entire system or for particular directories)
  • File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
  • Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.

Hardware needed: 

10 compute nodes with:

100GB Ethernet or better with at least 2 x 100GB worth of Bandiwth to the Panasas Storage. 

Our Compute nodes are modes with a single 16C AMD CPU and 96GB DDR. 

Redhat 8.4 base OS 

IO500 Benchmark + MPI 

Panasas Realm info:

3x ASD 200s 

2x ASU-Flash enclosures (16 OSDs total) 

Standard Panasas Mellanox Fabric Setup (1 Pair SN2510s in a MLAG Pair 2x 100GB upsteam to L2 switch) 

Benchmark Setup: 

Create 4 Volumes as follows: 

ior_easy : Raid 6+ with 256k Stripe size 

ior_hard : Raid10 all defaults 

mdtest_easy: Raid 10 all defaults 

mdtest_hard : Raid 10 all defaults 

The config.ini is setup to use these 4 volumes with each type of test running in above volumes.    This is easy to do in the GUI. 

I run the io500-scale.sh script from /panfs/pgh-realm-75-4/r6-256/io500-run/.  It is pretty easy to hack the script or config.ini to another location.  

Fault Tolerance Mechanisms

Question asked

Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.

  • Power (For example, can power failure of a node or rack bring down the storage system?)
  • Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
  • Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
  • Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)

Please list any additional information needed to determine whether this system has a single point of failure.

Power: 

There are 2 PDUs in the Cab and each server connects to both. 

Network: 

Each Server has a single 25GbE Ethnetnet adapter. Both ports are used, one to each SN2410 that are in an MLAG pair. 

If a cable or swith fails the server is not impacted. 

Raid: 

Pansasas raids data from the Client level.  With a Raid 6 Volume it takes 3 Servers going down to loose any data. 

Our ActiveStor Director node acts only as a read cache for Metedata and if you lose 2 the there is an FSCK process to restore volume services, no data is lost.  The Metadata is fully distrubued we have a default Raid6 + mode that puts 4 copies of medata out to storage.  In general metdata is N-way replicated depending on the mode.  

Users can set Raid levels on files, directories of volumes and we currently supprt:   Raid 10, Raid5, Raid5+ and Raid6+ (the + modes are an extra metadata copy compared to the standard).  Raid0 is possible but we do not permote it. 

Execution

Question asked

Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.

During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?

This system is setup for exclusive use.  

This job was ran as a basic MPI job as root.  

Caching

Question asked

Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).

A few examples would include:

  • Client data/metadata caching (in Linux page cache or in other memory cache)
  • Client side NVMe read-only data cache
  • Storage server metadata/data caching in RAM
  • Storage controller caching
  • RAID card caching

During Writes data is cache on each client but the drop cache between phases clears this all out. 

The ActiveStor directory keeps a metadata read cache, after mdtest_*_write an efficent btree is used for the read phases.    If the ASD did not have space for the btree then the mdtest_-_reads would go all the way the Active Store UltraLX nodes to retrive metadata at a reduced rate. 

Writes are landed on our ActiveStor Flash in a supercap powerd 16GB Dimm,  after the write is commited to disk the dimm space if freeded. 

The NVMe are in a software defined Raid0 with no raid controller with BTRFS ontop.    

Data Source

Question asked

Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.

Panasas is Primary Storage.  The Data is read back from the servers it was written do.  All data is writen to persistant Memory then sent to the NVMe   On read the data can be read from the servers page cache (IOR_Read wipes this out for everyone else) or from the back end media on the server.  

Trust

Question asked

Please describe any steps taken to ensure that the results are trustworthy.

  • Did you run the benchmark multiple times and get similar scores?
  • Did you validate the score is below the physical capabilities of the deployed hardware?
  • How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?
  1. We ran the io500 for a few days with a few different setups and the results we shared here are the best.  There some run to run variation mostly from 10.3 + class results every run in this config.
  2. 6 Months ago this systems was getting ~ 6.4 score. We have software work underway to furuer improve on performance of the HW but these numbers represent our current state of the system pretty well.  
  3. Panasas is production stroage. Data is allways persistently stored.