System: node0092.palmetto.clemson.edu filesystem_utilization=beegfs_nodev 805961809920 56929363968 749032445952 8% /burstbuffer date=2019.11.01-16.25.15 io500_clean_cache=False io500_clean_cache_cmd=drop_cache io500_cleanup_workdir=False io500_find_cmd=/home/jburto2/io-500-dev/bin/pfind io500_find_cmd_args='-s 100000 -r /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/pfind_results' io500_find_mpi=True io500_info_data_storage_type=SSD io500_info_filesysem=BeeGFS io500_info_filesystem_version=7.1.3 io500_info_institute_name='Clemson University' io500_info_metadata_storage_type=SSD io500_info_num_client_nodes=10 io500_info_num_data_server_nodes=4 io500_info_num_data_storage_devices=24 io500_info_num_metadata_server_nodes=4 io500_info_num_metadata_storage_devices=8 io500_info_procs_per_node=12 io500_info_storage_age_in_months=0 io500_info_storage_install_date=05/19 io500_info_storage_interface=NVMe io500_info_storage_network='infiniband EDR' io500_info_system_name='Palmetto burstbuffer AMD EPIC test' io500_info_whatever=infiniband io500_ior_cmd=/home/jburto2/io-500-dev/bin/ior io500_ior_easy_params='-t 2048k -b 174848m -F -a POSIX -b 174848m -F' io500_ior_easy_size=174848 io500_ior_hard_api=POSIX io500_ior_hard_api_specific_options= io500_ior_hard_other_options= io500_ior_hard_writes_per_proc=416666 io500_mdreal_cmd=/home/jburto2/io-500-dev/bin/md-real-io io500_mdreal_params='-P=5000 -I=1000' io500_mdtest_cmd=/home/jburto2/io-500-dev/bin/mdtest io500_mdtest_easy_files_per_proc=200000 io500_mdtest_easy_params=' -u -L' io500_mdtest_files_per_proc=416666 io500_mdtest_hard_api=POSIX io500_mdtest_hard_api_specific_options= io500_mdtest_hard_files_per_proc=416666 io500_mdtest_hard_other_options= io500_mpi_prefix=/software/openmpi/4.0.1_gcc io500_mpiargs='-np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc' io500_mpienv='-x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib' io500_mpirun=/software/openmpi/4.0.1_gcc/bin/mpirun io500_result_dir=/home/jburto2/io-500-dev/results/2019.11.01-16.25.15 io500_rules=regular io500_run_find=True io500_run_ior_easy=True io500_run_ior_easy_read=True io500_run_ior_hard=True io500_run_ior_hard_read=True io500_run_md_easy=True io500_run_md_easy_delete=True io500_run_md_easy_stat=True io500_run_md_hard=True io500_run_md_hard_delete=True io500_run_md_hard_read=True io500_run_md_hard_stat=True io500_run_mdreal=True io500_stonewall_timer=300 io500_workdir=/burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15 Running the IO500 Benchmark now [Creating] directories [Starting] ior_easy_write [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/ior -w -t 2048k -b 174848m -F -a POSIX -b 174848m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_easy/ior_file_easy -O stoneWallingStatusFile=/burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/ior_easy_write.txt. [RESULT] BW phase 1 ior_easy_write 20.867 GB/s : time 772.68 seconds [Starting] mdtest_easy_write [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -C -F -P -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_easy -n 200000 -u -L -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_easy-stonewall -N 1 -W 300 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_easy_write.txt. [RESULT] IOPS phase 1 mdtest_easy_write 34.392 kiops : time 550.38 seconds [Starting] ior_hard_write [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/ior -w -s 416666 -a POSIX -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_hard/IOR_file -O stoneWallingStatusFile=/burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/ior_hard_write.txt. [RESULT] BW phase 2 ior_hard_write 3.507 GB/s : time 487.17 seconds [Starting] mdtest_hard_write [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -C -t -F -P -w 3901 -e 3901 -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard -n 416666 -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard-stonewall -a POSIX -N 1 -W 300 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_hard_write.txt. [RESULT] IOPS phase 2 mdtest_hard_write 3.911 kiops : time 304.71 seconds [Starting] find [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/pfind /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15 -newer /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/timestampfile -size 3901c -name "*01*" -s 100000 -r /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/pfind_results [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/find.txt. [FIND] MATCHED 32430/20120286 in 21.0142 seconds [RESULT] IOPS phase 3 find 957.460 kiops : time 21.01 seconds [Starting] ior_easy_read [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/ior -r -R -t 2048k -b 174848m -F -a POSIX -b 174848m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_easy/ior_file_easy -O stoneWallingStatusFile=/burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_easy/stonewall [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/ior_easy_read.txt. [RESULT] BW phase 3 ior_easy_read 29.591 GB/s : time 545.02 seconds [Starting] mdtest_easy_stat [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -T -F -P -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_easy -n 200000 -u -L -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_easy-stonewall -N 1 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_easy_stat.txt. [RESULT] IOPS phase 4 mdtest_easy_stat 190.594 kiops : time 99.31 seconds [Starting] ior_hard_read [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/ior -r -R -s 416666 -a POSIX -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_hard/IOR_file -O stoneWallingStatusFile=/burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/ior_hard/stonewall [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/ior_hard_read.txt. [RESULT] BW phase 4 ior_hard_read 13.058 GB/s : time 130.79 seconds [Starting] mdtest_hard_stat [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -T -t -F -P -w 3901 -e 3901 -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard -n 416666 -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard-stonewall -a POSIX -N 1 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_hard_stat.txt. [RESULT] IOPS phase 5 mdtest_hard_stat 55.581 kiops : time 21.44 seconds [Starting] mdtest_easy_delete [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -r -F -P -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_easy -n 200000 -u -L -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_easy-stonewall -N 1 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_easy_delete.txt. [RESULT] IOPS phase 6 mdtest_easy_delete 48.815 kiops : time 387.76 seconds [Starting] mdtest_hard_read [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -X -E -t -F -P -w 3901 -e 3901 -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard -n 416666 -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard-stonewall -a POSIX -N 1 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_hard_read.txt. [RESULT] IOPS phase 7 mdtest_hard_read 15.515 kiops : time 76.80 seconds [Starting] mdtest_hard_delete [Exec] /software/openmpi/4.0.1_gcc/bin/mpirun -np 120 -x LD_LIBRARY_PATH=/opt/pbs/default/lib:/opt/orangefs/lib --mca btl_openib_allow_ib true --mca mpi_cuda_support 0 --map-by node --machinefile /home/jburto2/hdrnodesmpi --prefix /software/openmpi/4.0.1_gcc /home/jburto2/io-500-dev/bin/mdtest -r -t -F -P -w 3901 -e 3901 -d /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard -n 416666 -x /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15/mdt_hard-stonewall -a POSIX -N 1 [Results] in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15/mdtest_hard_delete.txt. [RESULT] IOPS phase 8 mdtest_hard_delete 2.614 kiops : time 459.20 seconds Running mdreal [Leaving] datafiles in /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15 [Summary] Results files in /home/jburto2/io-500-dev/results/2019.11.01-16.25.15 [Summary] Data files in /burstbuffer/fast/jburto2/io500/datafiles/io500.2019.11.01-16.25.15 [RESULT] BW phase 1 ior_easy_write 20.867 GB/s : time 772.68 seconds [RESULT] BW phase 2 ior_hard_write 3.507 GB/s : time 487.17 seconds [RESULT] BW phase 3 ior_easy_read 29.591 GB/s : time 545.02 seconds [RESULT] BW phase 4 ior_hard_read 13.058 GB/s : time 130.79 seconds [RESULT] IOPS phase 1 mdtest_easy_write 34.392 kiops : time 550.38 seconds [RESULT] IOPS phase 2 mdtest_hard_write 3.911 kiops : time 304.71 seconds [RESULT] IOPS phase 3 find 957.460 kiops : time 21.01 seconds [RESULT] IOPS phase 4 mdtest_easy_stat 190.594 kiops : time 99.31 seconds [RESULT] IOPS phase 5 mdtest_hard_stat 55.581 kiops : time 21.44 seconds [RESULT] IOPS phase 6 mdtest_easy_delete 48.815 kiops : time 387.76 seconds [RESULT] IOPS phase 7 mdtest_hard_read 15.515 kiops : time 76.80 seconds [RESULT] IOPS phase 8 mdtest_hard_delete 2.614 kiops : time 459.20 seconds [SCORE] Bandwidth 12.9676 GB/s : IOPS 35.8047 kiops : TOTAL 21.5476