Altair
Poznan Supercomputing and Networking Center · xFiusion LustreKMD · Fully Reproducible
System Purpose
Question asked
Please describe the purpose and general usage of the submitted system. This would include the types of typical applications it supports (e.g., defense applications, molecular dynamics, benchmarking, system test, systems research), and the general use and purpose of the data generated by the applications running on it.
The system is used as a high-performance space for users of our HPC cluster. The HPC cluster supports mixed types of tasks from many scientific fields (biology, chemistry, physics, engineering).
Availability
Question asked
Please provide the deployment timeframe of the submitted system, or for on-demand cloud systems, the general period over which it is deployed and destroyed.
Please describe the availability of the system to users and who are its set of most regular users.
The system life cycle is four years. After this period, a new one is built.
Scientific users have access to the system after receiving a research grant.
The cluster was expanded with a new IB NDR network (200gbit).
Storage System Software
Question asked
How is this storage software available? (e.g., commercially, open-source, not publicly available) Note that if the storage software is not open-source or commercially available, then a general description would be requested, but this would limit the submission’s reproducibility score.
Can anyone download/purchase this software?
List either product webpage or open-source repo of the exact software used in IO500.
Give any and all additional details of this specific storage deployment: (e.g., type of storage server product such IBM ESS or DDN SFA400X2, use of Ext4 or some other file system on each storage node, dual connected storage media to storage servers).
The system is general-purpose for users representing many scientific fields. There is no single leading software used for counting on an HPC cluster.
Lustre, which is available as open-source, is used as a data storage system.
Software used to run the cluster: https://downloads.whamcloud.com/public/lustre/lustre-2.15.3/el8.8/
The system is composed of 24 OSS servers and 8 MDS servers.
Each 4 OSS servers have 3 Huawei Oceanstore 5510 arrays connected.
Each 4 MSD servers have a Huawei Dorado 6000 V6 array connected.
Runtime Environment
Question asked
State here that you provided all scripts/documentation that would allow someone else to reproduce your environment and attempt to achieve a similar IO500 score as the submitted system.
NOTE: provide all files/documentation/scripts that would enable a user to build your environment and deploy the custom scripts, software, or config files once they have obtained the appropriate storage system hardware and software. These may be included into the io500.tgz or into the extraCodes upload on the submission form.
Several examples include:
- Commands used to set striping information (either for the entire system or for particular directories)
- File system config and tuning information (or a reason why this is not available due to lack of root access, etc) on each node type (e.g., non-default config parameters on all three types of Lustre nodes: MDS, OSS, and client)
- Any additional scripts utilized that impact IO500 execution beyond the io500 config file. For example, with IBM Spectrum Scale, the output of mmlsconfig, mmlscluster, mmlsfs, and mmlsfileset commands (and possibly even a dump of the configuration if possible). Each file system probably has similar type of config/tuning information that would need to be shared for a user to fully reproduce the environment.
1. Lutre cluster installation
a)Serwer
- repositories
subscription-manager repos --enable=rhel-8-for-x86_64-baseos-debug-rpms --enable=rhel-8-for-x86_64-appstream-debug-rpms --enable codeready-builder-for-rhel-8-x86_64-rpms#aktualizaujemy pakiet e2fsprogs (w repo jest starsza wersja)wget --recursive --no-parent https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/
yum install downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/*.rpm
- installation of additional packages
yum install asciidoc audit-libs-devel automake bc binutils-devel bison elfutils-devel elfutils-libelf-devel expect flex gcc gcc-c++ git glib2 glib2-devel hmaccalc keyutils-libs-devel krb5-devel ksh libattr-devel libblkid-devel libselinux-devel libtool libuuid-devel lsscsi make ncurses-devel net-snmp-devel net-tools newt-devel numactl-devel parted patchutils pciutils-devel perl-ExtUtils-Embed pesign redhat-rpm-config rpm-build systemd-devel tcl tcl-devel tk tk-devel wget xmlto yum-utils zlib-devel libaio-devel libffi-devel device-mapper.x86_64 libyaml.x86_64 platform-python-devel.x86_64 python3-cffi.x86_64 platform-python-setuptools.noarch kernel-headers kernel-devel.x86_64 libmount.x86_64 libmount-devel.x86_64 kernel-debuginfo-x86_64 kernel-debuginfo-common-x86_64-$(uname -r) libnl3-devel.x86_64 libyaml-devel.x86_64 e2fsprogs-devel.x86_64 python3 kernel-rpm-macros.noarch kernel-abi-whitelists.noarch sysfsutils.x86_64 sg3_utils mc.x86_64 sysstat.x86_64 tmux.x86_64#proces kompilacji kodu Lustre
echo "LC_ALL=en_US.UTF-8" >> /etc/environment
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release
git checkout 2.15.3
sh autogen.sh
-code compilation
./configure --enable-server --with-linux=/usr/src/kernels/[...]./configure --enable-server --without-zfs --enable-ldiskfs --with-linux=/usr/src/kernels/4.18.0-477.27.1.el8_8.x86_64
make rpms
cd ..
mkdir rpms_lustre_server
mv lustre-release/*.rpm rpms_lustre_server
- Lustre packages installation
kmod-lustre-2.15.3-1.el8.x86_64.rpm
kmod-lustre-osd-ldiskfs-2.15.3-1.el8.x86_64.rpm
kmod-lustre-tests-2.15.3-1.el8.x86_64.rpm
lustre-2.15.3-1.el8.x86_64.rpm
lustre-debuginfo-2.15.3-1.el8.x86_64.rpm
lustre-devel-2.15.3-1.el8.x86_64.rpm
lustre-iokit-2.15.3-1.el8.x86_64.rpm
lustre-osd-ldiskfs-mount-2.15.3-1.el8.x86_64.rpm
lustre-tests-2.15.3-1.el8.x86_64.rpm
yum install -y rpms_lustre_server/*
b) Client
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release
git checkout 2.15.3
sh autogen.sh
./configure --disable-server --enable-client make rpms
./configure --disable-server --enable-client --with-linux=/usr/src/kernels/[...]./configure --disable-server --with-linux=/usr/src/kernels/4.18.0-477.27.1.el8_8.x86_64
make rpms
cd ..
mkdir -p rpms_lustre_client
mv lustre-release/*.rpm rpms_lustre_client
yum install -y rpms_lustre_client/kmod-lustre-client-[...].rpm
2. Example commands creating cluster components
a) MGT
mkfs.lustre --reformat --mgs --servicenode 10.48.50.125@o2ib6 --backfstype=ldiskfs /dev/mapper/mgt00
b) MDT
mkfs.lustre --reformat --mdt --fsname kmdfs --index 0 --mgsnode 10.48.50.125@o2ib6 --servicenode 10.48.50.125@o2ib6 --backfstype=ldiskfs /dev/mapper/mdt00
c) OST
mkfs.lustre --reformat --ost --fsname kmdfs --index 0 --mgsnode 10.48.50.125@o2ib6 --servicenode 10.48.50.101@o2ib6 --backfstype=ldiskfs /dev/mapper/A1_BST1_K01R09_OSS0000
3. Post-installation tuning
- DNE
lfs mkdir -c 8 -i 0,1,2,3,4,5,6,7 /mnt/lustre/io500PROD/
lfs mkdir -c 8 -D /mnt/lustre/io500PROD/
- DoM+PFL
lfs setstripe -E 1M -L mdt -E 64M -c1 -S 1M -E 8G -c4 -E -1 -c -1 -S 4M /mnt/lustre/io500PROD/
lfs getstripe /mnt/lustre/io500PROD/
Fault Tolerance Mechanisms
Question asked
Does your system have a single point of failure as defined by “Definition 7” of a Production System? Please describe all mechanisms that provide fault tolerance for the submitted storage system. Be specific to your submission, not general storage system capabilities.
- Power (For example, can power failure of a node or rack bring down the storage system?)
- Networking (For example, can failure of a single NIC bring down the storage system? Remember that any networking failure that brings down multiple nodes is acceptable)
- Inter storage data and metadata server (For example, what type of data protection mechanism does the storage system use to keep operating during node failures, e.g., active-active servers, client-directed RAID, declustered RAID, erasure-coding, 3-way replication)
- Intra storage data and metadata server storage media (For example, how does the storage eysstem protect against fiailure of a single storage device within a node, e.g., raid5)
Please list any additional information needed to determine whether this system has a single point of failure.
Environmental protection systems:
- two power paths (UPS, generator)
- network cards (active/backup)
- each server group (MDS/OSS) can take over the resources of another server in the event of its failure (Corosync / Pacemaker)
- storrage array (OST / MDT) has RAID6
Execution
Question asked
Please provide a description of how the IO500 benchmark was executed, e.g., via system scheduler (e.g., SLURM) to run a job on the compute cluster, which initially ran a setup process to configure the client and file system, and then started the full benchmark.
During the IO500 benchmark execution was the system entirely dedicated to running the benchmark or were there other jobs running in the same cluster and storage system?
We used openmpi libraries to run the io500 script.
During the io500 tests, the entire cluster was available - there were no other users.
Caching
Question asked
Please describe all caching mechanisms in client/server that were utilized during the IO500 run. This could include caching in any storage medium (e.g., SSD, RAM).
A few examples would include:
- Client data/metadata caching (in Linux page cache or in other memory cache)
- Client side NVMe read-only data cache
- Storage server metadata/data caching in RAM
- Storage controller caching
- RAID card caching
During the io500 tests, we did not use an additional caching layer.
Data Source
Question asked
Is the submitted system a stand-alone storage system or an acceleration layer in front of another storage system that is the source of truth for all application data? This question relates to whether the submitted system is a burst buffer layered on primary storage or primary storage itself.
The reported system is a stand-alone storage system. It is not an additional caching layer.
Trust
Question asked
Please describe any steps taken to ensure that the results are trustworthy.
- Did you run the benchmark multiple times and get similar scores?
- Did you validate the score is below the physical capabilities of the deployed hardware?
- How did you validate that the storage system follows the IO500 rules regarding persistence, which states, "All data must be written to persistent storage within the measured time for the individual benchmark, e.g. if a file system caches data, it must ensure that data is persistently stored before acknowledging the close"?
We ran the tests many times and obtained similar results.
We obtained results that met the hardware capabilities.