szaydel
9/19/2013 - 9:02 PM
fio test with iscsi+Multipath and LIO with 2 targets ZFS pool 8 vdevs, 16 disks - LUNs are 512b bs
fio test with iscsi+Multipath and LIO with 2 targets ZFS pool 8 vdevs, 16 disks - LUNs are 512b bs
## First Run
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
Jobs: 1 (f=1): [w_______] [99.7% done] [0KB/45650KB/0KB /s] [0/1426/0 iops] [eta 00m:01s]
random-writers: (groupid=0, jobs=1): err= 0: pid=17752: Thu Sep 19 13:58:33 2013
write: io=0B, bw=31910KB/s, iops=997, runt=394319msec
slat (usec): min=3, max=330356, avg=36.37, stdev=1219.64
clat (usec): min=1, max=2794.7K, avg=3968.63, stdev=40144.83
lat (usec): min=26, max=2794.8K, avg=4005.00, stdev=40165.46
clat percentiles (usec):
| 1.00th=[ 2], 5.00th=[ 30], 10.00th=[ 35], 20.00th=[ 41],
| 30.00th=[ 47], 40.00th=[ 55], 50.00th=[ 64], 60.00th=[ 88],
| 70.00th=[ 133], 80.00th=[ 227], 90.00th=[10048], 95.00th=[20096],
| 99.00th=[40192], 99.50th=[59648], 99.90th=[179200], 99.95th=[1073152],
| 99.99th=[1925120]
bw (KB /s): min= 130, max=1598113, per=30.17%, avg=38515.36, stdev=111353.77
lat (usec) : 2=0.47%, 4=1.28%, 10=0.03%, 20=1.16%, 50=30.20%
lat (usec) : 100=29.31%, 250=18.35%, 500=2.87%, 750=0.61%, 1000=0.22%
lat (msec) : 2=0.23%, 4=0.08%, 10=5.48%, 20=5.15%, 50=3.97%
lat (msec) : 100=0.40%, 250=0.11%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=92.66%, sys=5.49%, ctx=703451, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=17753: Thu Sep 19 13:58:33 2013
write: io=0B, bw=32294KB/s, iops=1009, runt=389640msec
slat (usec): min=3, max=252177, avg=29.65, stdev=1003.50
clat (usec): min=1, max=2774.7K, avg=3928.01, stdev=40112.43
lat (usec): min=30, max=2774.7K, avg=3957.66, stdev=40128.54
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 45],
| 30.00th=[ 52], 40.00th=[ 59], 50.00th=[ 68], 60.00th=[ 88],
| 70.00th=[ 137], 80.00th=[ 211], 90.00th=[10048], 95.00th=[20096],
| 99.00th=[40192], 99.50th=[60160], 99.90th=[171008], 99.95th=[1073152],
| 99.99th=[1941504]
bw (KB /s): min= 121, max=766554, per=30.34%, avg=38723.64, stdev=94725.06
lat (usec) : 2=0.06%, 4=0.43%, 10=0.02%, 20=0.19%, 50=26.53%
lat (usec) : 100=34.87%, 250=19.73%, 500=2.65%, 750=0.55%, 1000=0.24%
lat (msec) : 2=0.24%, 4=0.10%, 10=5.16%, 20=4.76%, 50=3.75%
lat (msec) : 100=0.51%, 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=93.06%, sys=5.44%, ctx=674762, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=17754: Thu Sep 19 13:58:33 2013
write: io=0B, bw=37384KB/s, iops=1168, runt=336581msec
slat (usec): min=2, max=290852, avg=58.44, stdev=1236.22
clat (usec): min=1, max=2774.8K, avg=3359.07, stdev=40136.03
lat (usec): min=21, max=2774.8K, avg=3417.51, stdev=40157.14
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 30],
| 30.00th=[ 44], 40.00th=[ 52], 50.00th=[ 63], 60.00th=[ 88],
| 70.00th=[ 126], 80.00th=[ 191], 90.00th=[ 9920], 95.00th=[19840],
| 99.00th=[40192], 99.50th=[60160], 99.90th=[150528], 99.95th=[1105920],
| 99.99th=[1925120]
bw (KB /s): min= 139, max=572928, per=36.11%, avg=46092.96, stdev=100534.26
lat (usec) : 2=4.96%, 4=7.82%, 10=0.05%, 20=5.72%, 50=18.74%
lat (usec) : 100=25.77%, 250=21.10%, 500=3.66%, 750=0.50%, 1000=0.19%
lat (msec) : 2=0.18%, 4=0.05%, 10=3.78%, 20=3.68%, 50=3.18%
lat (msec) : 100=0.45%, 250=0.10%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=90.75%, sys=6.35%, ctx=950383, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=17755: Thu Sep 19 13:58:33 2013
write: io=0B, bw=38109KB/s, iops=1190, runt=330182msec
slat (usec): min=2, max=360107, avg=55.90, stdev=1286.20
clat (usec): min=1, max=2804.6K, avg=3296.55, stdev=39643.40
lat (usec): min=25, max=2804.6K, avg=3352.45, stdev=39662.63
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 28],
| 30.00th=[ 47], 40.00th=[ 56], 50.00th=[ 70], 60.00th=[ 91],
| 70.00th=[ 127], 80.00th=[ 183], 90.00th=[ 9920], 95.00th=[19840],
| 99.00th=[40192], 99.50th=[49920], 99.90th=[160768], 99.95th=[1056768],
| 99.99th=[1925120]
bw (KB /s): min= 183, max=570158, per=37.11%, avg=47362.23, stdev=102767.27
lat (usec) : 2=5.98%, 4=9.33%, 10=0.07%, 20=3.67%, 50=14.36%
lat (usec) : 100=29.50%, 250=21.65%, 500=3.32%, 750=0.43%, 1000=0.14%
lat (msec) : 2=0.14%, 4=0.08%, 10=3.86%, 20=3.81%, 50=3.12%
lat (msec) : 100=0.38%, 250=0.09%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=90.79%, sys=6.50%, ctx=982249, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=17756: Thu Sep 19 13:58:33 2013
read : io=0B, bw=1420.6MB/s, iops=45458, runt= 8650msec
slat (usec): min=1, max=140502, avg= 6.76, stdev=255.40
clat (usec): min=1, max=348612, avg=78.71, stdev=1466.16
lat (usec): min=18, max=348617, avg=85.47, stdev=1488.48
clat percentiles (usec):
| 1.00th=[ 20], 5.00th=[ 26], 10.00th=[ 29], 20.00th=[ 34],
| 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 46], 60.00th=[ 51],
| 70.00th=[ 59], 80.00th=[ 74], 90.00th=[ 107], 95.00th=[ 149],
| 99.00th=[ 270], 99.50th=[ 338], 99.90th=[ 652], 99.95th=[ 1064],
| 99.99th=[60160]
bw (MB /s): min= 118, max= 2373, per=25.26%, avg=1433.16, stdev=569.38
lat (usec) : 2=0.06%, 4=0.12%, 10=0.08%, 20=0.65%, 50=56.18%
lat (usec) : 100=31.35%, 250=10.31%, 500=1.06%, 750=0.11%, 1000=0.03%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=83.59%, sys=143.57%, ctx=607137, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=17757: Thu Sep 19 13:58:33 2013
read : io=0B, bw=1461.3MB/s, iops=46761, runt= 8409msec
slat (usec): min=1, max=300877, avg= 8.02, stdev=484.51
clat (usec): min=1, max=399875, avg=74.63, stdev=1646.07
lat (usec): min=17, max=399886, avg=82.65, stdev=1717.89
clat percentiles (usec):
| 1.00th=[ 21], 5.00th=[ 26], 10.00th=[ 29], 20.00th=[ 34],
| 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 60], 80.00th=[ 74], 90.00th=[ 109], 95.00th=[ 155],
| 99.00th=[ 294], 99.50th=[ 366], 99.90th=[ 620], 99.95th=[ 876],
| 99.99th=[11200]
bw (MB /s): min= 46, max= 2610, per=25.83%, avg=1465.74, stdev=653.00
lat (usec) : 2=0.02%, 4=0.05%, 10=0.02%, 20=0.59%, 50=55.31%
lat (usec) : 100=32.17%, 250=10.24%, 500=1.42%, 750=0.11%, 1000=0.02%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=77.33%, sys=144.60%, ctx=599218, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=17758: Thu Sep 19 13:58:33 2013
read : io=0B, bw=1503.5MB/s, iops=48111, runt= 8173msec
slat (usec): min=1, max=35705, avg= 5.96, stdev=58.60
clat (usec): min=1, max=421434, avg=74.61, stdev=1664.07
lat (usec): min=15, max=421488, avg=80.56, stdev=1665.24
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 32],
| 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 46], 60.00th=[ 53],
| 70.00th=[ 61], 80.00th=[ 75], 90.00th=[ 107], 95.00th=[ 145],
| 99.00th=[ 253], 99.50th=[ 318], 99.90th=[ 556], 99.95th=[ 860],
| 99.99th=[31360]
bw (MB /s): min= 45, max= 2106, per=26.58%, avg=1507.92, stdev=511.63
lat (usec) : 2=0.09%, 4=0.13%, 10=0.14%, 20=1.07%, 50=53.53%
lat (usec) : 100=33.35%, 250=10.64%, 500=0.92%, 750=0.08%, 1000=0.02%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=82.91%, sys=144.68%, ctx=621926, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=17759: Thu Sep 19 13:58:33 2013
read : io=0B, bw=1418.5MB/s, iops=45390, runt= 8663msec
slat (usec): min=1, max=20546, avg= 6.82, stdev=47.75
clat (usec): min=1, max=428975, avg=78.55, stdev=1658.64
lat (usec): min=14, max=428979, avg=85.37, stdev=1659.48
clat percentiles (usec):
| 1.00th=[ 2], 5.00th=[ 21], 10.00th=[ 26], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 53],
| 70.00th=[ 62], 80.00th=[ 78], 90.00th=[ 111], 95.00th=[ 149],
| 99.00th=[ 253], 99.50th=[ 310], 99.90th=[ 588], 99.95th=[ 1176],
| 99.99th=[47872]
bw (MB /s): min= 55, max= 2241, per=24.71%, avg=1401.91, stdev=543.80
lat (usec) : 2=0.52%, 4=0.65%, 10=1.02%, 20=2.01%, 50=50.53%
lat (usec) : 100=32.55%, 250=11.69%, 500=0.89%, 750=0.06%, 1000=0.02%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=79.59%, sys=133.53%, ctx=628225, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=5673.9MB/s, minb=1418.5MB/s, maxb=1503.5MB/s, mint=8173msec, maxt=8663msec
WRITE: io=49152MB, aggrb=127641KB/s, minb=31910KB/s, maxb=38109KB/s, mint=330182msec, maxt=394319msec
## Second Run
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
Jobs: 1 (f=1): [___w____] [100.0% done] [0KB/737.8MB/0KB /s] [0/23.6K/0 iops] [eta 00m:00s]
random-writers: (groupid=0, jobs=1): err= 0: pid=20888: Thu Sep 19 14:23:10 2013
write: io=0B, bw=36016KB/s, iops=1125, runt=349372msec
slat (usec): min=2, max=239999, avg=52.55, stdev=1330.25
clat (usec): min=1, max=2166.9K, avg=3494.68, stdev=36092.40
lat (usec): min=22, max=2166.1K, avg=3547.23, stdev=36115.70
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 2],
| 30.00th=[ 32], 40.00th=[ 45], 50.00th=[ 52], 60.00th=[ 72],
| 70.00th=[ 97], 80.00th=[ 155], 90.00th=[ 9920], 95.00th=[20096],
| 99.00th=[49920], 99.50th=[60160], 99.90th=[179200], 99.95th=[946176],
| 99.99th=[1810432]
bw (KB /s): min= 181, max=1020931, per=32.52%, avg=43423.91, stdev=109695.58
lat (usec) : 2=8.88%, 4=13.43%, 10=0.35%, 20=3.24%, 50=21.71%
lat (usec) : 100=23.05%, 250=15.15%, 500=2.43%, 750=0.50%, 1000=0.21%
lat (msec) : 2=0.17%, 4=0.05%, 10=3.05%, 20=3.24%, 50=3.71%
lat (msec) : 100=0.62%, 250=0.13%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=91.20%, sys=4.90%, ctx=1022839, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=20889: Thu Sep 19 14:23:10 2013
write: io=0B, bw=35994KB/s, iops=1124, runt=349582msec
slat (usec): min=2, max=408552, avg=55.80, stdev=1515.66
clat (usec): min=1, max=2390.1K, avg=3492.95, stdev=36348.40
lat (usec): min=22, max=2390.1K, avg=3548.75, stdev=36386.06
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 18],
| 30.00th=[ 41], 40.00th=[ 46], 50.00th=[ 57], 60.00th=[ 77],
| 70.00th=[ 105], 80.00th=[ 161], 90.00th=[10048], 95.00th=[20096],
| 99.00th=[40192], 99.50th=[59648], 99.90th=[160768], 99.95th=[978944],
| 99.99th=[1826816]
bw (KB /s): min= 243, max=740992, per=32.06%, avg=42812.48, stdev=102662.57
lat (usec) : 2=6.39%, 4=8.76%, 10=0.22%, 20=5.38%, 50=23.04%
lat (usec) : 100=24.88%, 250=15.68%, 500=1.86%, 750=0.29%, 1000=0.12%
lat (msec) : 2=0.09%, 4=0.03%, 10=4.79%, 20=4.62%, 50=3.28%
lat (msec) : 100=0.40%, 250=0.11%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.04%, >=2000=0.01%
cpu : usr=91.14%, sys=5.15%, ctx=952348, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=20890: Thu Sep 19 14:23:10 2013
write: io=0B, bw=33916KB/s, iops=1059, runt=371007msec
slat (usec): min=2, max=440505, avg=53.32, stdev=1712.16
clat (usec): min=1, max=2217.1K, avg=3712.93, stdev=36016.23
lat (usec): min=22, max=2217.1K, avg=3766.25, stdev=36064.22
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 19],
| 30.00th=[ 36], 40.00th=[ 45], 50.00th=[ 54], 60.00th=[ 67],
| 70.00th=[ 90], 80.00th=[ 153], 90.00th=[10048], 95.00th=[20096],
| 99.00th=[40192], 99.50th=[60160], 99.90th=[168960], 99.95th=[978944],
| 99.99th=[1810432]
bw (KB /s): min= 204, max=671811, per=30.11%, avg=40196.30, stdev=96106.60
lat (usec) : 2=5.78%, 4=9.29%, 10=0.13%, 20=4.86%, 50=25.48%
lat (usec) : 100=26.95%, 250=11.30%, 500=1.84%, 750=0.36%, 1000=0.11%
lat (msec) : 2=0.11%, 4=0.04%, 10=4.94%, 20=4.49%, 50=3.61%
lat (msec) : 100=0.50%, 250=0.13%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.04%, >=2000=0.01%
cpu : usr=91.02%, sys=4.89%, ctx=850123, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=20891: Thu Sep 19 14:23:10 2013
write: io=0B, bw=33380KB/s, iops=1043, runt=376965msec
slat (usec): min=2, max=468272, avg=47.89, stdev=1580.12
clat (usec): min=1, max=2510.5K, avg=3779.73, stdev=36566.28
lat (usec): min=21, max=2510.6K, avg=3827.62, stdev=36602.10
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 18],
| 30.00th=[ 35], 40.00th=[ 43], 50.00th=[ 51], 60.00th=[ 65],
| 70.00th=[ 88], 80.00th=[ 151], 90.00th=[10048], 95.00th=[20096],
| 99.00th=[40704], 99.50th=[60160], 99.90th=[162816], 99.95th=[978944],
| 99.99th=[1810432]
bw (KB /s): min= 214, max=1087559, per=27.81%, avg=37133.41, stdev=95359.67
lat (usec) : 2=6.45%, 4=9.36%, 10=0.13%, 20=4.67%, 50=27.52%
lat (usec) : 100=24.79%, 250=11.22%, 500=1.53%, 750=0.26%, 1000=0.09%
lat (msec) : 2=0.10%, 4=0.04%, 10=4.75%, 20=4.56%, 50=3.81%
lat (msec) : 100=0.51%, 250=0.13%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.04%, >=2000=0.01%
cpu : usr=91.74%, sys=4.85%, ctx=843154, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=20892: Thu Sep 19 14:23:10 2013
read : io=0B, bw=559539KB/s, iops=17485, runt= 22488msec
slat (usec): min=0, max=99866, avg= 4.35, stdev=159.33
clat (usec): min=1, max=1528.7K, avg=222.58, stdev=9377.56
lat (usec): min=11, max=1528.7K, avg=226.93, stdev=9378.99
clat percentiles (usec):
| 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 18], 20.00th=[ 21],
| 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 24], 60.00th=[ 26],
| 70.00th=[ 28], 80.00th=[ 31], 90.00th=[ 36], 95.00th=[ 42],
| 99.00th=[ 61], 99.50th=[ 87], 99.90th=[49408], 99.95th=[72192],
| 99.99th=[309248]
bw (KB /s): min= 1117, max=4037056, per=29.83%, avg=667541.89, stdev=1310176.92
lat (usec) : 2=0.03%, 4=0.04%, 10=0.01%, 20=13.97%, 50=83.64%
lat (usec) : 100=1.87%, 250=0.09%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.05%, 20=0.06%, 50=0.09%
lat (msec) : 100=0.07%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=90.63%, sys=36.08%, ctx=426096, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=20893: Thu Sep 19 14:23:10 2013
read : io=0B, bw=2186.5MB/s, iops=69967, runt= 5620msec
slat (usec): min=1, max=120254, avg= 5.13, stdev=222.94
clat (usec): min=1, max=439760, avg=49.88, stdev=1363.27
lat (usec): min=17, max=439765, avg=55.02, stdev=1381.71
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 27],
| 30.00th=[ 30], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 39],
| 70.00th=[ 43], 80.00th=[ 51], 90.00th=[ 64], 95.00th=[ 82],
| 99.00th=[ 135], 99.50th=[ 167], 99.90th=[ 314], 99.95th=[ 478],
| 99.99th=[ 8768]
bw (MB /s): min= 165, max= 2893, per=100.00%, avg=2189.93, stdev=987.11
lat (usec) : 2=0.01%, 4=0.02%, 10=0.07%, 20=0.91%, 50=77.76%
lat (usec) : 100=18.48%, 250=2.59%, 500=0.11%, 750=0.02%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=89.45%, sys=180.58%, ctx=574338, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=20894: Thu Sep 19 14:23:10 2013
read : io=0B, bw=1743.5MB/s, iops=55791, runt= 7048msec
slat (usec): min=1, max=330150, avg= 7.31, stdev=545.46
clat (usec): min=1, max=411391, avg=62.17, stdev=1947.24
lat (usec): min=17, max=411403, avg=69.48, stdev=2026.74
clat percentiles (usec):
| 1.00th=[ 21], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 29],
| 30.00th=[ 31], 40.00th=[ 34], 50.00th=[ 37], 60.00th=[ 41],
| 70.00th=[ 46], 80.00th=[ 54], 90.00th=[ 72], 95.00th=[ 95],
| 99.00th=[ 167], 99.50th=[ 205], 99.90th=[ 418], 99.95th=[ 668],
| 99.99th=[21632]
bw (MB /s): min= 83, max= 2487, per=78.96%, avg=1725.83, stdev=940.58
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.49%, 50=74.21%
lat (usec) : 100=20.78%, 250=4.21%, 500=0.20%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=78.52%, sys=152.44%, ctx=601879, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=20895: Thu Sep 19 14:23:10 2013
read : io=0B, bw=1530.9MB/s, iops=48986, runt= 8027msec
slat (usec): min=1, max=60930, avg= 6.68, stdev=156.98
clat (usec): min=1, max=358568, avg=72.85, stdev=1809.87
lat (usec): min=16, max=358573, avg=79.53, stdev=1832.56
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 21], 10.00th=[ 24], 20.00th=[ 26],
| 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 34], 60.00th=[ 37],
| 70.00th=[ 42], 80.00th=[ 50], 90.00th=[ 69], 95.00th=[ 96],
| 99.00th=[ 183], 99.50th=[ 235], 99.90th=[ 588], 99.95th=[10176],
| 99.99th=[70144]
bw (MB /s): min= 44, max= 3566, per=64.94%, avg=1419.38, stdev=990.44
lat (usec) : 2=0.05%, 4=0.05%, 10=0.01%, 20=1.82%, 50=77.23%
lat (usec) : 100=16.19%, 250=4.22%, 500=0.30%, 750=0.04%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%
cpu : usr=88.23%, sys=120.89%, ctx=564572, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=2185.8MB/s, minb=559538KB/s, maxb=2186.5MB/s, mint=5620msec, maxt=22488msec
WRITE: io=49152MB, aggrb=133518KB/s, minb=33379KB/s, maxb=36015KB/s, mint=349372msec, maxt=376965msec