System: ip-172-31-48-38 filesystem_utilization=172.31.48.29,172.31.57.160,172.31.63.177,172.31.51.28,172.31.58.64,172.31.50.58/default 4398046511104 1426551155416 2971495355688 33% /mnt/weka date=2019.11.11-04.23.30 io500_clean_cache=False io500_clean_cache_cmd=drop_cache io500_cleanup_workdir=False io500_find_cmd=/mnt/weka/io-500-dev/bin/custom_pfind.sh io500_find_cmd_args='-s 300 -M 1000 -q 50000 -r /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/pfind_results' io500_find_mpi=False io500_info_system_name=xxx io500_ior_cmd=/mnt/weka/io-500-dev/bin/ior io500_ior_easy_params='--posix.odirect -u -t 2048k -b 9920000m -F' io500_ior_easy_size=9920000 io500_ior_hard_api=POSIX io500_ior_hard_api_specific_options=--posix.odirect io500_ior_hard_writes_per_proc=1900000 io500_mdreal_cmd=/mnt/weka/io-500-dev/bin/md-real-io io500_mdreal_params='-P=5000 -I=1000' io500_mdtest_cmd=/mnt/weka/io-500-dev/bin/mdtest io500_mdtest_easy_files_per_proc=900000 io500_mdtest_easy_params='-u -L' io500_mdtest_hard_api=POSIX io500_mdtest_hard_api_specific_options= io500_mdtest_hard_files_per_proc=950000 io500_mpiargs='--oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp' io500_mpirun=mpirun io500_result_dir=/mnt/weka/io-500-dev/results/2019.11.11-04.23.30 io500_rules=regular io500_run_find=True io500_run_ior_easy=True io500_run_ior_easy_read=True io500_run_ior_hard=True io500_run_ior_hard_read=True io500_run_md_easy=True io500_run_md_easy_delete=True io500_run_md_easy_stat=True io500_run_md_hard=True io500_run_md_hard_delete=True io500_run_md_hard_read=True io500_run_md_hard_stat=True io500_run_mdreal=False io500_stonewall_timer=300 io500_workdir=/mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30 Running the IO500 Benchmark now [Creating] directories [Starting] ior_easy_write [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/ior -w --posix.odirect -u -t 2048k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_easy/ior_file_easy -O stoneWallingStatusFile=/mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/ior_easy_write.txt. [RESULT] BW phase 1 ior_easy_write 298.134 GB/s : time 820.62 seconds [Starting] mdtest_easy_write [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -C -F -P -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_easy -n 900000 -u -L -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_easy-stonewall -N 1 -W 300 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_easy_write.txt. [RESULT] IOPS phase 1 mdtest_easy_write 2235.660 kiops : time 436.35 seconds [Starting] ior_hard_write [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/ior -w -s 1900000 -a POSIX --posix.odirect -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_hard/IOR_file -O stoneWallingStatusFile=/mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/ior_hard_write.txt. [RESULT] BW phase 2 ior_hard_write 71.567 GB/s : time 904.18 seconds [Starting] mdtest_hard_write [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -C -t -F -P -w 3901 -e 3901 -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard -n 950000 -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard-stonewall -a POSIX -N 1 -W 300 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_hard_write.txt. [RESULT] IOPS phase 2 mdtest_hard_write 438.857 kiops : time 412.07 seconds [Starting] find [EXEC] /mnt/weka/io-500-dev/bin/custom_pfind.sh /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30 -newer /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/timestampfile -size 3901c -name "*01*" -s 300 -M 1000 -q 50000 -r /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/pfind_results [FIND] MATCHED 8341511/1154608344 in 35.2215 seconds [RESULT] IOPS phase 3 find 32781.350 kiops : time 35.22 seconds [Starting] ior_easy_read [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/ior -r -R --posix.odirect -u -t 2048k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_easy/ior_file_easy -O stoneWallingStatusFile=/mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_easy/stonewall [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/ior_easy_read.txt. [RESULT] BW phase 3 ior_easy_read 325.620 GB/s : time 751.54 seconds [Starting] mdtest_easy_stat [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -T -F -P -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_easy -n 900000 -u -L -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_easy-stonewall -N 1 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_easy_stat.txt. [RESULT] IOPS phase 4 mdtest_easy_stat 36673.300 kiops : time 26.60 seconds [Starting] ior_hard_read [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/ior -r -R -s 1900000 -a POSIX --posix.odirect -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_hard/IOR_file -O stoneWallingStatusFile=/mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/ior_hard/stonewall [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/ior_hard_read.txt. [RESULT] BW phase 4 ior_hard_read 134.196 GB/s : time 481.70 seconds [Starting] mdtest_hard_stat [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -T -t -F -P -w 3901 -e 3901 -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard -n 950000 -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard-stonewall -a POSIX -N 1 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_hard_stat.txt. [RESULT] IOPS phase 5 mdtest_hard_stat 28649.100 kiops : time 6.31 seconds [Starting] mdtest_easy_delete [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -r -F -P -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_easy -n 900000 -u -L -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_easy-stonewall -N 1 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_easy_delete.txt. [RESULT] IOPS phase 6 mdtest_easy_delete 3385.190 kiops : time 288.17 seconds [Starting] mdtest_hard_read [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -X -E -t -F -P -w 3901 -e 3901 -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard -n 950000 -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard-stonewall -a POSIX -N 1 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_hard_read.txt. [RESULT] IOPS phase 7 mdtest_hard_read 7709.030 kiops : time 23.46 seconds [Starting] mdtest_hard_delete [Exec] mpirun --oversubscribe -n 8625 -H 172.31.56.151,172.31.63.221,172.31.52.62,172.31.63.191,172.31.59.91,172.31.55.87,172.31.60.195,172.31.63.193,172.31.55.181,172.31.51.51,172.31.62.208,172.31.60.175,172.31.54.110,172.31.57.115,172.31.61.239,172.31.50.188,172.31.63.46,172.31.61.123,172.31.57.67,172.31.60.113,172.31.50.192,172.31.60.241,172.31.50.69,172.31.62.245,172.31.50.66,172.31.61.71,172.31.50.243,172.31.58.249,172.31.48.205,172.31.63.59,172.31.57.144,172.31.60.30,172.31.59.178,172.31.54.144,172.31.63.185,172.31.53.53,172.31.62.178,172.31.52.13,172.31.59.27,172.31.58.18,172.31.60.144,172.31.53.87,172.31.63.95,172.31.60.236,172.31.56.245,172.31.53.171,172.31.56.30,172.31.60.106,172.31.50.48,172.31.58.167,172.31.60.77,172.31.51.6,172.31.54.36,172.31.63.111,172.31.52.135,172.31.58.186,172.31.51.190,172.31.62.61,172.31.50.249,172.31.62.180,172.31.49.10,172.31.63.67,172.31.55.170,172.31.51.60,172.31.51.100,172.31.54.101,172.31.54.252,172.31.49.46,172.31.55.131,172.31.49.224,172.31.62.136,172.31.58.204,172.31.60.219,172.31.54.46,172.31.57.117,172.31.54.254,172.31.55.252,172.31.63.74,172.31.56.89,172.31.59.169,172.31.55.168,172.31.51.96,172.31.48.81,172.31.55.172,172.31.54.169,172.31.52.85,172.31.56.160,172.31.53.135,172.31.49.210,172.31.63.64,172.31.60.174,172.31.50.227,172.31.59.246,172.31.48.251,172.31.55.212,172.31.49.109,172.31.58.2,172.31.63.234,172.31.63.169,172.31.56.6,172.31.62.124,172.31.54.8,172.31.49.88,172.31.60.207,172.31.63.86,172.31.53.64,172.31.50.22,172.31.58.1,172.31.51.242,172.31.53.183,172.31.52.28,172.31.60.167,172.31.57.90,172.31.59.50,172.31.49.75,172.31.60.215,172.31.51.25,172.31.63.201,172.31.58.40,172.31.57.93,172.31.55.7,172.31.61.40,172.31.61.179,172.31.60.9,172.31.54.77,172.31.50.79,172.31.53.121,172.31.57.224,172.31.50.212,172.31.54.13,172.31.58.212,172.31.60.163,172.31.53.240,172.31.62.182,172.31.63.184,172.31.51.129,172.31.49.137,172.31.51.176,172.31.53.140,172.31.53.190,172.31.56.185,172.31.58.142,172.31.53.182,172.31.60.142,172.31.61.31,172.31.57.186,172.31.48.138,172.31.57.19,172.31.56.161,172.31.50.120,172.31.57.175,172.31.49.138,172.31.55.161,172.31.62.93,172.31.54.245,172.31.56.235,172.31.49.158,172.31.57.151,172.31.48.200,172.31.58.145,172.31.50.118,172.31.60.166,172.31.60.108,172.31.57.139,172.31.51.132,172.31.53.96,172.31.58.119,172.31.59.102,172.31.62.173,172.31.62.162,172.31.52.206,172.31.50.152,172.31.48.129,172.31.57.131,172.31.50.217,172.31.55.56,172.31.53.157,172.31.58.61,172.31.59.16,172.31.59.153,172.31.60.150,172.31.58.29,172.31.49.128,172.31.54.163,172.31.60.116,172.31.50.230,172.31.60.171,172.31.53.36,172.31.50.94,172.31.55.255,172.31.61.158,172.31.52.160,172.31.51.238,172.31.53.147,172.31.52.219,172.31.57.59,172.31.52.182,172.31.62.27,172.31.52.132,172.31.58.54,172.31.54.161,172.31.53.235,172.31.61.195,172.31.61.169,172.31.53.11,172.31.59.1,172.31.59.65,172.31.59.55,172.31.53.238,172.31.60.73,172.31.55.84,172.31.53.161,172.31.53.15,172.31.51.75,172.31.62.218,172.31.63.6,172.31.59.38,172.31.59.8,172.31.51.211,172.31.48.36,172.31.48.5,172.31.58.85,172.31.51.58,172.31.54.78,172.31.59.37,172.31.58.72,172.31.56.126,172.31.54.50,172.31.50.4,172.31.49.223,172.31.62.65,172.31.53.232,172.31.56.65,172.31.61.208,172.31.50.34,172.31.55.254,172.31.55.224,172.31.60.229,172.31.58.196,172.31.57.62,172.31.52.240,172.31.58.193,172.31.56.250,172.31.60.228,172.31.52.252,172.31.58.219,172.31.58.255,172.31.57.68,172.31.61.87,172.31.49.213,172.31.59.245,172.31.52.200,172.31.48.239,172.31.48.224,172.31.59.248,172.31.60.248,172.31.54.206,172.31.63.239,172.31.50.247,172.31.56.81,172.31.63.89,172.31.52.7,172.31.60.162,172.31.55.3,172.31.48.6,172.31.52.6,172.31.57.26,172.31.60.187,172.31.51.29,172.31.61.186,172.31.57.86,172.31.63.48,172.31.60.26,172.31.55.135,172.31.59.95,172.31.62.72,172.31.63.147,172.31.51.18,172.31.51.82,172.31.63.164,172.31.48.178,172.31.48.19,172.31.63.112,172.31.56.115,172.31.60.222,172.31.61.213,172.31.53.56,172.31.63.22,172.31.57.21,172.31.49.45,172.31.61.74,172.31.62.172,172.31.48.222,172.31.61.109,172.31.54.96,172.31.49.172,172.31.60.12,172.31.61.89,172.31.51.31,172.31.60.90,172.31.51.7,172.31.62.190,172.31.56.176,172.31.60.160,172.31.58.162,172.31.61.10,172.31.63.222,172.31.55.61,172.31.56.45,172.31.58.185,172.31.54.174,172.31.54.150,172.31.57.146,172.31.59.87,172.31.52.5,172.31.52.35,172.31.56.90,172.31.51.126,172.31.57.138,172.31.48.137,172.31.63.37,172.31.57.58,172.31.48.59,172.31.61.167,172.31.60.97,172.31.51.159,172.31.50.87,172.31.49.23,172.31.55.221,172.31.48.42,172.31.53.164,172.31.57.88,172.31.55.151,172.31.63.139,172.31.59.107,172.31.62.28,172.31.54.30,172.31.59.52,172.31.55.223,172.31.48.82,172.31.51.30,172.31.55.41,172.31.55.165,172.31.49.39,172.31.48.38 --mca pml ob1 --mca btl self,tcp /mnt/weka/io-500-dev/bin/mdtest -r -t -F -P -w 3901 -e 3901 -d /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard -n 950000 -x /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30/mdt_hard-stonewall -a POSIX -N 1 [Results] in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30/mdtest_hard_delete.txt. [RESULT] IOPS phase 8 mdtest_hard_delete 476.121 kiops : time 387.75 seconds [Skipping] mdreal [Leaving] datafiles in /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30 [Summary] Results files in /mnt/weka/io-500-dev/results/2019.11.11-04.23.30 [Summary] Data files in /mnt/weka/io-500-dev/datafiles/io500.2019.11.11-04.23.30 [RESULT] BW phase 1 ior_easy_write 298.134 GB/s : time 820.62 seconds [RESULT] BW phase 2 ior_hard_write 71.567 GB/s : time 904.18 seconds [RESULT] BW phase 3 ior_easy_read 325.620 GB/s : time 751.54 seconds [RESULT] BW phase 4 ior_hard_read 134.196 GB/s : time 481.70 seconds [RESULT] IOPS phase 1 mdtest_easy_write 2235.660 kiops : time 436.35 seconds [RESULT] IOPS phase 2 mdtest_hard_write 438.857 kiops : time 412.07 seconds [RESULT] IOPS phase 3 find 32781.350 kiops : time 35.22 seconds [RESULT] IOPS phase 4 mdtest_easy_stat 36673.300 kiops : time 26.60 seconds [RESULT] IOPS phase 5 mdtest_hard_stat 28649.100 kiops : time 6.31 seconds [RESULT] IOPS phase 6 mdtest_easy_delete 3385.190 kiops : time 288.17 seconds [RESULT] IOPS phase 7 mdtest_hard_read 7709.030 kiops : time 23.46 seconds [RESULT] IOPS phase 8 mdtest_hard_delete 476.121 kiops : time 387.75 seconds [SCORE] Bandwidth 174.741 GB/s : IOPS 5045.33 kiops : TOTAL 938.949