szaydel
9/18/2013 - 7:51 PM
fio test with iscsi+Multipath with 2 targets ZFS pool 8 vdevs, 16 disks - LUNs are 512b bs
fio test with iscsi+Multipath with 2 targets ZFS pool 8 vdevs, 16 disks - LUNs are 512b bs
## First Run
root@cl01:/volumes/p0/test/d0# fio ~/rand.fio
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
Jobs: 1 (f=1): [__w_____] [88.4% done] [0KB/329.2MB/0KB /s] [0/10.6K/0 iops] [eta 00m:21s]
random-writers: (groupid=0, jobs=1): err= 0: pid=6230: Wed Sep 18 12:38:38 2013
write: io=0B, bw=103867KB/s, iops=3245, runt=121144msec
slat (usec): min=2, max=300218, avg=70.02, stdev=1380.25
clat (usec): min=1, max=580238, avg=1156.26, stdev=6759.76
lat (usec): min=24, max=580244, avg=1226.29, stdev=6918.24
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 11],
| 30.00th=[ 45], 40.00th=[ 53], 50.00th=[ 75], 60.00th=[ 102],
| 70.00th=[ 137], 80.00th=[ 181], 90.00th=[ 286], 95.00th=[ 1004],
| 99.00th=[30080], 99.50th=[40192], 99.90th=[80384], 99.95th=[110080],
| 99.99th=[189440]
bw (KB /s): min= 4400, max=1251136, per=32.82%, avg=102895.37, stdev=164960.53
lat (usec) : 2=6.24%, 4=10.46%, 10=0.07%, 20=5.34%, 50=14.33%
lat (usec) : 100=22.71%, 250=28.70%, 500=6.03%, 750=0.87%, 1000=0.25%
lat (msec) : 2=0.18%, 4=0.06%, 10=1.44%, 20=1.61%, 50=1.42%
lat (msec) : 100=0.23%, 250=0.05%, 500=0.01%, 750=0.01%
cpu : usr=74.99%, sys=18.38%, ctx=1009120, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=6231: Wed Sep 18 12:38:38 2013
write: io=0B, bw=81465KB/s, iops=2545, runt=154458msec
slat (usec): min=2, max=357294, avg=70.09, stdev=1663.68
clat (usec): min=1, max=730231, avg=1491.97, stdev=8090.40
lat (usec): min=26, max=730409, avg=1562.06, stdev=8304.76
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 25],
| 30.00th=[ 43], 40.00th=[ 52], 50.00th=[ 66], 60.00th=[ 88],
| 70.00th=[ 123], 80.00th=[ 173], 90.00th=[ 334], 95.00th=[ 9920],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[100864], 99.95th=[132096],
| 99.99th=[238592]
bw (KB /s): min= 3269, max=841524, per=25.24%, avg=79125.90, stdev=129939.74
lat (usec) : 2=5.44%, 4=9.42%, 10=0.10%, 20=4.64%, 50=17.43%
lat (usec) : 100=26.29%, 250=23.74%, 500=5.01%, 750=0.90%, 1000=0.33%
lat (msec) : 2=0.29%, 4=0.09%, 10=2.11%, 20=2.12%, 50=1.67%
lat (msec) : 100=0.30%, 250=0.10%, 500=0.01%, 750=0.01%
cpu : usr=75.63%, sys=15.06%, ctx=929156, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=6232: Wed Sep 18 12:38:38 2013
write: io=0B, bw=78381KB/s, iops=2449, runt=160535msec
slat (usec): min=3, max=260281, avg=32.99, stdev=943.59
clat (usec): min=1, max=528654, avg=1593.75, stdev=8440.13
lat (usec): min=29, max=528666, avg=1626.74, stdev=8506.17
clat percentiles (usec):
| 1.00th=[ 5], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 46],
| 30.00th=[ 53], 40.00th=[ 61], 50.00th=[ 73], 60.00th=[ 97],
| 70.00th=[ 135], 80.00th=[ 181], 90.00th=[ 378], 95.00th=[ 9920],
| 99.00th=[30336], 99.50th=[49920], 99.90th=[110080], 99.95th=[140288],
| 99.99th=[240640]
bw (KB /s): min= 2200, max=1927552, per=24.44%, avg=76611.63, stdev=165451.95
lat (usec) : 2=0.17%, 4=0.79%, 10=0.06%, 20=0.63%, 50=23.40%
lat (usec) : 100=35.69%, 250=25.39%, 500=5.35%, 750=1.12%, 1000=0.44%
lat (msec) : 2=0.46%, 4=0.11%, 10=2.01%, 20=2.06%, 50=1.87%
lat (msec) : 100=0.32%, 250=0.11%, 500=0.01%, 750=0.01%
cpu : usr=82.30%, sys=14.10%, ctx=734878, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=6233: Wed Sep 18 12:38:38 2013
write: io=0B, bw=80416KB/s, iops=2513, runt=156472msec
slat (usec): min=3, max=229155, avg=67.12, stdev=1372.45
clat (usec): min=1, max=510641, avg=1515.56, stdev=8018.23
lat (usec): min=24, max=510655, avg=1582.68, stdev=8154.78
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 13],
| 30.00th=[ 39], 40.00th=[ 50], 50.00th=[ 64], 60.00th=[ 86],
| 70.00th=[ 124], 80.00th=[ 175], 90.00th=[ 334], 95.00th=[ 9920],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[99840], 99.95th=[129536],
| 99.99th=[220160]
bw (KB /s): min= 3646, max=555619, per=25.53%, avg=80027.77, stdev=122604.93
lat (usec) : 2=6.83%, 4=10.77%, 10=0.15%, 20=5.84%, 50=15.52%
lat (usec) : 100=24.39%, 250=23.36%, 500=5.29%, 750=0.91%, 1000=0.29%
lat (msec) : 2=0.23%, 4=0.07%, 10=2.02%, 20=2.14%, 50=1.80%
lat (msec) : 100=0.30%, 250=0.09%, 500=0.01%, 750=0.01%
cpu : usr=77.10%, sys=15.31%, ctx=956985, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=6234: Wed Sep 18 12:38:38 2013
read : io=0B, bw=1008.2MB/s, iops=32259, runt= 12189msec
slat (usec): min=1, max=137889, avg=10.73, stdev=367.05
clat (usec): min=1, max=480000, avg=109.51, stdev=2329.46
lat (usec): min=19, max=480007, avg=120.24, stdev=2358.60
clat percentiles (usec):
| 1.00th=[ 2], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 33],
| 30.00th=[ 39], 40.00th=[ 45], 50.00th=[ 52], 60.00th=[ 61],
| 70.00th=[ 75], 80.00th=[ 102], 90.00th=[ 159], 95.00th=[ 225],
| 99.00th=[ 434], 99.50th=[ 564], 99.90th=[ 1448], 99.95th=[ 8768],
| 99.99th=[99840]
bw (KB /s): min=203648, max=1655104, per=25.77%, avg=989393.87, stdev=490283.96
lat (usec) : 2=0.75%, 4=0.87%, 10=0.33%, 20=1.28%, 50=43.82%
lat (usec) : 100=32.30%, 250=16.62%, 500=3.33%, 750=0.43%, 1000=0.11%
lat (msec) : 2=0.07%, 4=0.02%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=60.19%, sys=100.47%, ctx=721461, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=6235: Wed Sep 18 12:38:38 2013
read : io=0B, bw=959649KB/s, iops=29989, runt= 13112msec
slat (usec): min=1, max=157863, avg=12.00, stdev=489.08
clat (usec): min=1, max=439137, avg=117.11, stdev=2400.72
lat (usec): min=19, max=439143, avg=129.11, stdev=2450.56
clat percentiles (usec):
| 1.00th=[ 8], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 35],
| 30.00th=[ 39], 40.00th=[ 45], 50.00th=[ 52], 60.00th=[ 61],
| 70.00th=[ 76], 80.00th=[ 103], 90.00th=[ 161], 95.00th=[ 227],
| 99.00th=[ 430], 99.50th=[ 564], 99.90th=[ 1640], 99.95th=[ 9920],
| 99.99th=[101888]
bw (KB /s): min=76928, max=1926755, per=25.90%, avg=994134.72, stdev=563350.15
lat (usec) : 2=0.43%, 4=0.56%, 10=0.23%, 20=1.44%, 50=44.45%
lat (usec) : 100=32.02%, 250=16.87%, 500=3.32%, 750=0.42%, 1000=0.10%
lat (msec) : 2=0.08%, 4=0.02%, 10=0.02%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=57.20%, sys=100.18%, ctx=715482, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=6236: Wed Sep 18 12:38:38 2013
read : io=0B, bw=1019.8MB/s, iops=32610, runt= 12058msec
slat (usec): min=1, max=331130, avg=12.19, stdev=701.04
clat (usec): min=1, max=434295, avg=106.94, stdev=2327.24
lat (usec): min=19, max=434297, avg=119.14, stdev=2434.58
clat percentiles (usec):
| 1.00th=[ 8], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 34],
| 30.00th=[ 39], 40.00th=[ 46], 50.00th=[ 54], 60.00th=[ 64],
| 70.00th=[ 78], 80.00th=[ 104], 90.00th=[ 157], 95.00th=[ 219],
| 99.00th=[ 414], 99.50th=[ 548], 99.90th=[ 1256], 99.95th=[ 3632],
| 99.99th=[82432]
bw (MB /s): min= 76, max= 1827, per=27.37%, avg=1025.94, stdev=543.45
lat (usec) : 2=0.35%, 4=0.56%, 10=0.52%, 20=1.67%, 50=41.99%
lat (usec) : 100=33.67%, 250=17.49%, 500=3.13%, 750=0.37%, 1000=0.10%
lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=60.28%, sys=103.92%, ctx=770631, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=6237: Wed Sep 18 12:38:38 2013
read : io=0B, bw=1027.8MB/s, iops=32888, runt= 11956msec
slat (usec): min=1, max=350052, avg=13.99, stdev=694.30
clat (usec): min=1, max=600526, avg=104.01, stdev=1925.47
lat (usec): min=20, max=600727, avg=117.99, stdev=2052.80
clat percentiles (usec):
| 1.00th=[ 2], 5.00th=[ 21], 10.00th=[ 26], 20.00th=[ 33],
| 30.00th=[ 39], 40.00th=[ 46], 50.00th=[ 54], 60.00th=[ 64],
| 70.00th=[ 80], 80.00th=[ 107], 90.00th=[ 161], 95.00th=[ 223],
| 99.00th=[ 418], 99.50th=[ 532], 99.90th=[ 1208], 99.95th=[ 3600],
| 99.99th=[80384]
bw (MB /s): min= 154, max= 1901, per=27.34%, avg=1024.73, stdev=546.73
lat (usec) : 2=0.81%, 4=1.01%, 10=0.64%, 20=2.10%, 50=39.91%
lat (usec) : 100=33.34%, 250=18.38%, 500=3.20%, 750=0.40%, 1000=0.08%
lat (msec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=55.96%, sys=99.09%, ctx=775075, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=3748.7MB/s, minb=959648KB/s, maxb=1027.8MB/s, mint=11956msec, maxt=13112msec
WRITE: io=49152MB, aggrb=313524KB/s, minb=78381KB/s, maxb=103867KB/s, mint=121144msec, maxt=160535msec
## Second Run
root@cl01:/volumes/p0/test/d0# fio ~/rand.fio
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
Jobs: 1 (f=1): [_w______] [93.8% done] [0KB/950.8MB/0KB /s] [0/30.5K/0 iops] [eta 00m:11s]
random-writers: (groupid=0, jobs=1): err= 0: pid=9788: Wed Sep 18 13:08:19 2013
write: io=0B, bw=75785KB/s, iops=2368, runt=166034msec
slat (usec): min=2, max=250166, avg=60.41, stdev=1593.79
clat (usec): min=1, max=680649, avg=1621.21, stdev=9059.26
lat (usec): min=21, max=680658, avg=1681.63, stdev=9240.72
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 19],
| 30.00th=[ 36], 40.00th=[ 45], 50.00th=[ 56], 60.00th=[ 69],
| 70.00th=[ 85], 80.00th=[ 116], 90.00th=[ 249], 95.00th=[10048],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[110080], 99.95th=[150528],
| 99.99th=[268288]
bw (KB /s): min= 2137, max=904128, per=24.24%, avg=73280.22, stdev=134097.36
lat (usec) : 2=5.92%, 4=9.01%, 10=0.17%, 20=5.17%, 50=24.69%
lat (usec) : 100=30.80%, 250=14.26%, 500=2.18%, 750=0.49%, 1000=0.19%
lat (msec) : 2=0.16%, 4=0.07%, 10=2.37%, 20=2.32%, 50=1.76%
lat (msec) : 100=0.33%, 250=0.11%, 500=0.01%, 750=0.01%
cpu : usr=79.14%, sys=10.54%, ctx=881497, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=9789: Wed Sep 18 13:08:19 2013
write: io=0B, bw=75587KB/s, iops=2362, runt=166470msec
slat (usec): min=2, max=565425, avg=47.86, stdev=1562.18
clat (usec): min=1, max=790200, avg=1638.68, stdev=9079.58
lat (usec): min=23, max=790203, avg=1686.54, stdev=9218.48
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 28],
| 30.00th=[ 39], 40.00th=[ 48], 50.00th=[ 57], 60.00th=[ 71],
| 70.00th=[ 89], 80.00th=[ 126], 90.00th=[ 294], 95.00th=[10048],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[105984], 99.95th=[150528],
| 99.99th=[309248]
bw (KB /s): min= 1823, max=977792, per=22.96%, avg=69422.40, stdev=124042.35
lat (usec) : 2=5.96%, 4=9.38%, 10=0.13%, 20=1.95%, 50=24.02%
lat (usec) : 100=32.59%, 250=15.13%, 500=2.55%, 750=0.60%, 1000=0.25%
lat (msec) : 2=0.24%, 4=0.08%, 10=2.45%, 20=2.39%, 50=1.87%
lat (msec) : 100=0.30%, 250=0.10%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=82.33%, sys=10.56%, ctx=824167, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=9790: Wed Sep 18 13:08:19 2013
write: io=0B, bw=84145KB/s, iops=2629, runt=149539msec
slat (usec): min=2, max=510477, avg=60.85, stdev=1469.43
clat (usec): min=1, max=530383, avg=1452.91, stdev=8535.94
lat (usec): min=21, max=530401, avg=1513.75, stdev=8682.67
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 15],
| 30.00th=[ 42], 40.00th=[ 50], 50.00th=[ 68], 60.00th=[ 84],
| 70.00th=[ 106], 80.00th=[ 145], 90.00th=[ 258], 95.00th=[ 9920],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[100864], 99.95th=[140288],
| 99.99th=[261120]
bw (KB /s): min= 2263, max=848064, per=27.27%, avg=82455.89, stdev=143057.55
lat (usec) : 2=6.89%, 4=10.28%, 10=0.21%, 20=3.40%, 50=18.76%
lat (usec) : 100=28.26%, 250=21.91%, 500=3.51%, 750=0.62%, 1000=0.20%
lat (msec) : 2=0.21%, 4=0.07%, 10=1.70%, 20=1.86%, 50=1.70%
lat (msec) : 100=0.32%, 250=0.09%, 500=0.01%, 750=0.01%
cpu : usr=78.17%, sys=11.23%, ctx=1066625, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=9791: Wed Sep 18 13:08:19 2013
write: io=0B, bw=78420KB/s, iops=2450, runt=160456msec
slat (usec): min=2, max=457303, avg=59.37, stdev=1596.25
clat (usec): min=1, max=530237, avg=1565.81, stdev=8740.19
lat (usec): min=22, max=530243, avg=1625.17, stdev=8899.72
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 10],
| 30.00th=[ 32], 40.00th=[ 45], 50.00th=[ 56], 60.00th=[ 70],
| 70.00th=[ 88], 80.00th=[ 121], 90.00th=[ 262], 95.00th=[10048],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[99840], 99.95th=[140288],
| 99.99th=[301056]
bw (KB /s): min= 1514, max=803904, per=25.43%, avg=76871.94, stdev=135816.06
lat (usec) : 2=7.56%, 4=11.07%, 10=0.18%, 20=6.10%, 50=19.90%
lat (usec) : 100=29.76%, 250=15.13%, 500=2.61%, 750=0.51%, 1000=0.15%
lat (msec) : 2=0.17%, 4=0.09%, 10=2.30%, 20=2.30%, 50=1.76%
lat (msec) : 100=0.31%, 250=0.09%, 500=0.01%, 750=0.01%
cpu : usr=79.90%, sys=10.90%, ctx=922328, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=9792: Wed Sep 18 13:08:19 2013
read : io=0B, bw=1082.8MB/s, iops=34647, runt= 11349msec
slat (usec): min=1, max=521109, avg=14.98, stdev=1093.01
clat (usec): min=1, max=1314.4K, avg=96.53, stdev=2985.46
lat (usec): min=16, max=1314.4K, avg=111.51, stdev=3185.64
clat percentiles (usec):
| 1.00th=[ 8], 5.00th=[ 21], 10.00th=[ 24], 20.00th=[ 29],
| 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 48], 60.00th=[ 56],
| 70.00th=[ 69], 80.00th=[ 90], 90.00th=[ 135], 95.00th=[ 187],
| 99.00th=[ 358], 99.50th=[ 474], 99.90th=[ 1192], 99.95th=[ 3088],
| 99.99th=[70144]
bw (MB /s): min= 51, max= 1965, per=26.47%, avg=1082.88, stdev=682.68
lat (usec) : 2=0.35%, 4=0.52%, 10=0.65%, 20=2.69%, 50=48.19%
lat (usec) : 100=30.56%, 250=14.53%, 500=2.07%, 750=0.24%, 1000=0.07%
lat (msec) : 2=0.06%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%
cpu : usr=53.74%, sys=94.19%, ctx=729934, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=9793: Wed Sep 18 13:08:19 2013
read : io=0B, bw=1055.4MB/s, iops=33769, runt= 11644msec
slat (usec): min=1, max=190601, avg= 9.30, stdev=358.50
clat (usec): min=1, max=1440.4K, avg=105.33, stdev=4185.27
lat (usec): min=17, max=1440.4K, avg=114.63, stdev=4200.85
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 19], 10.00th=[ 24], 20.00th=[ 29],
| 30.00th=[ 34], 40.00th=[ 40], 50.00th=[ 46], 60.00th=[ 54],
| 70.00th=[ 66], 80.00th=[ 88], 90.00th=[ 139], 95.00th=[ 197],
| 99.00th=[ 378], 99.50th=[ 486], 99.90th=[ 1048], 99.95th=[ 4448],
| 99.99th=[88576]
bw (MB /s): min= 0, max= 1786, per=26.77%, avg=1095.18, stdev=595.53
lat (usec) : 2=1.08%, 4=1.29%, 10=0.24%, 20=2.79%, 50=49.51%
lat (usec) : 100=28.22%, 250=13.97%, 500=2.44%, 750=0.28%, 1000=0.07%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
cpu : usr=62.43%, sys=93.91%, ctx=732668, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=9794: Wed Sep 18 13:08:19 2013
read : io=0B, bw=1022.7MB/s, iops=32724, runt= 12016msec
slat (usec): min=1, max=370983, avg=11.51, stdev=717.28
clat (usec): min=1, max=1131.1K, avg=106.72, stdev=4118.17
lat (usec): min=18, max=1131.1K, avg=118.22, stdev=4250.97
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 23], 10.00th=[ 26], 20.00th=[ 31],
| 30.00th=[ 35], 40.00th=[ 40], 50.00th=[ 46], 60.00th=[ 54],
| 70.00th=[ 67], 80.00th=[ 90], 90.00th=[ 143], 95.00th=[ 205],
| 99.00th=[ 398], 99.50th=[ 524], 99.90th=[ 1272], 99.95th=[ 3056],
| 99.99th=[71168]
bw (MB /s): min= 6, max= 2433, per=26.05%, avg=1065.48, stdev=666.60
lat (usec) : 2=0.16%, 4=0.24%, 10=0.22%, 20=1.82%, 50=52.23%
lat (usec) : 100=28.01%, 250=14.10%, 500=2.66%, 750=0.34%, 1000=0.08%
lat (msec) : 2=0.07%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=54.55%, sys=94.34%, ctx=713527, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=9795: Wed Sep 18 13:08:19 2013
read : io=0B, bw=1110.7MB/s, iops=35540, runt= 11064msec
slat (usec): min=1, max=160279, avg=11.47, stdev=558.16
clat (usec): min=1, max=838250, avg=98.06, stdev=3048.46
lat (usec): min=18, max=945796, avg=109.53, stdev=3173.04
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 13], 10.00th=[ 22], 20.00th=[ 28],
| 30.00th=[ 32], 40.00th=[ 37], 50.00th=[ 43], 60.00th=[ 51],
| 70.00th=[ 64], 80.00th=[ 85], 90.00th=[ 131], 95.00th=[ 183],
| 99.00th=[ 358], 99.50th=[ 470], 99.90th=[ 1112], 99.95th=[ 3952],
| 99.99th=[94720]
bw (MB /s): min= 77, max= 1972, per=28.42%, avg=1162.45, stdev=690.03
lat (usec) : 2=1.89%, 4=1.61%, 10=0.66%, 20=2.66%, 50=51.22%
lat (usec) : 100=26.31%, 250=13.20%, 500=2.02%, 750=0.26%, 1000=0.06%
lat (msec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=58.49%, sys=99.08%, ctx=738464, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=4090.6MB/s, minb=1022.7MB/s, maxb=1110.7MB/s, mint=11064msec, maxt=12016msec
WRITE: io=49152MB, aggrb=302346KB/s, minb=75586KB/s, maxb=84144KB/s, mint=149539msec, maxt=166470msec
## Third Run
root@cl01:/volumes/p0/test/d0# fio ~/rand.fio
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
Jobs: 2 (f=2): [__ww____] [94.2% done] [0KB/206.4MB/0KB /s] [0/6604/0 iops] [eta 00m:11s]
random-writers: (groupid=0, jobs=1): err= 0: pid=1912: Wed Sep 18 16:10:18 2013
write: io=0B, bw=89301KB/s, iops=2790, runt=140904msec
slat (usec): min=2, max=400485, avg=68.40, stdev=2033.11
clat (usec): min=1, max=597847, avg=1359.76, stdev=8325.19
lat (usec): min=20, max=597854, avg=1428.15, stdev=8597.26
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 18],
| 30.00th=[ 39], 40.00th=[ 45], 50.00th=[ 55], 60.00th=[ 71],
| 70.00th=[ 87], 80.00th=[ 118], 90.00th=[ 209], 95.00th=[ 9920],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[105984], 99.95th=[140288],
| 99.99th=[259072]
bw (KB /s): min= 1257, max=967117, per=30.75%, avg=86931.17, stdev=150668.44
lat (usec) : 2=6.25%, 4=8.78%, 10=0.16%, 20=5.59%, 50=24.48%
lat (usec) : 100=29.14%, 250=16.99%, 500=2.53%, 750=0.42%, 1000=0.13%
lat (msec) : 2=0.14%, 4=0.06%, 10=1.68%, 20=1.70%, 50=1.52%
lat (msec) : 100=0.32%, 250=0.10%, 500=0.01%, 750=0.01%
cpu : usr=76.86%, sys=11.80%, ctx=967473, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=1914: Wed Sep 18 16:10:18 2013
write: io=0B, bw=72723KB/s, iops=2272, runt=173025msec
slat (usec): min=2, max=319830, avg=71.22, stdev=2075.40
clat (usec): min=1, max=599732, avg=1680.92, stdev=8702.60
lat (usec): min=25, max=599742, avg=1752.14, stdev=8966.91
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 15],
| 30.00th=[ 37], 40.00th=[ 46], 50.00th=[ 59], 60.00th=[ 73],
| 70.00th=[ 92], 80.00th=[ 129], 90.00th=[ 294], 95.00th=[10048],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[110080], 99.95th=[140288],
| 99.99th=[259072]
bw (KB /s): min= 1246, max=859305, per=25.08%, avg=70899.64, stdev=122755.20
lat (usec) : 2=7.23%, 4=10.27%, 10=0.08%, 20=5.36%, 50=20.06%
lat (usec) : 100=29.71%, 250=16.39%, 500=2.46%, 750=0.49%, 1000=0.16%
lat (msec) : 2=0.18%, 4=0.08%, 10=2.75%, 20=2.54%, 50=1.79%
lat (msec) : 100=0.33%, 250=0.11%, 500=0.01%, 750=0.01%
cpu : usr=77.85%, sys=10.17%, ctx=934773, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=1915: Wed Sep 18 16:10:18 2013
write: io=0B, bw=70725KB/s, iops=2210, runt=177913msec
slat (usec): min=2, max=120377, avg=39.48, stdev=667.18
clat (usec): min=1, max=759934, avg=1761.73, stdev=8742.47
lat (usec): min=24, max=759961, avg=1801.21, stdev=8775.73
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 2], 10.00th=[ 17], 20.00th=[ 34],
| 30.00th=[ 43], 40.00th=[ 52], 50.00th=[ 63], 60.00th=[ 78],
| 70.00th=[ 100], 80.00th=[ 143], 90.00th=[ 366], 95.00th=[10048],
| 99.00th=[39680], 99.50th=[49920], 99.90th=[99840], 99.95th=[126464],
| 99.99th=[261120]
bw (KB /s): min= 2782, max=882432, per=24.31%, avg=68709.93, stdev=121160.59
lat (usec) : 2=2.70%, 4=5.50%, 10=0.26%, 20=3.15%, 50=26.02%
lat (usec) : 100=32.37%, 250=17.76%, 500=3.29%, 750=0.74%, 1000=0.25%
lat (msec) : 2=0.24%, 4=0.09%, 10=2.52%, 20=2.53%, 50=2.14%
lat (msec) : 100=0.35%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=85.49%, sys=10.26%, ctx=867719, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=1916: Wed Sep 18 16:10:18 2013
write: io=0B, bw=70666KB/s, iops=2208, runt=178062msec
slat (usec): min=2, max=400369, avg=59.78, stdev=2035.78
clat (usec): min=1, max=429622, avg=1743.15, stdev=8809.06
lat (usec): min=21, max=429635, avg=1802.94, stdev=9068.79
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 2],
| 30.00th=[ 31], 40.00th=[ 43], 50.00th=[ 54], 60.00th=[ 67],
| 70.00th=[ 85], 80.00th=[ 120], 90.00th=[ 350], 95.00th=[10048],
| 99.00th=[30080], 99.50th=[49920], 99.90th=[110080], 99.95th=[148480],
| 99.99th=[268288]
bw (KB /s): min= 1171, max=684800, per=23.54%, avg=66536.50, stdev=117099.04
lat (usec) : 2=8.59%, 4=12.40%, 10=0.13%, 20=2.67%, 50=22.31%
lat (usec) : 100=29.31%, 250=13.15%, 500=2.29%, 750=0.52%, 1000=0.21%
lat (msec) : 2=0.21%, 4=0.08%, 10=3.04%, 20=2.78%, 50=1.87%
lat (msec) : 100=0.30%, 250=0.11%, 500=0.01%
cpu : usr=80.00%, sys=9.98%, ctx=867589, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=1917: Wed Sep 18 16:10:18 2013
read : io=0B, bw=1138.5MB/s, iops=36429, runt= 10794msec
slat (usec): min=1, max=220050, avg=13.00, stdev=693.73
clat (usec): min=1, max=479767, avg=93.22, stdev=1776.81
lat (usec): min=17, max=479773, avg=106.22, stdev=1909.85
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 21], 10.00th=[ 25], 20.00th=[ 30],
| 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 48], 60.00th=[ 56],
| 70.00th=[ 68], 80.00th=[ 90], 90.00th=[ 137], 95.00th=[ 199],
| 99.00th=[ 402], 99.50th=[ 532], 99.90th=[ 1112], 99.95th=[ 2672],
| 99.99th=[80384]
bw (MB /s): min= 206, max= 2177, per=27.14%, avg=1140.92, stdev=620.00
lat (usec) : 2=0.37%, 4=0.57%, 10=0.56%, 20=2.79%, 50=47.86%
lat (usec) : 100=30.79%, 250=13.96%, 500=2.51%, 750=0.39%, 1000=0.09%
lat (msec) : 2=0.05%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=57.89%, sys=103.17%, ctx=718973, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=1918: Wed Sep 18 16:10:18 2013
read : io=0B, bw=1076.9MB/s, iops=34459, runt= 11411msec
slat (usec): min=1, max=160267, avg= 9.75, stdev=326.33
clat (usec): min=1, max=778684, avg=102.74, stdev=3179.95
lat (usec): min=18, max=779520, avg=112.49, stdev=3200.08
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 32],
| 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 55],
| 70.00th=[ 66], 80.00th=[ 89], 90.00th=[ 139], 95.00th=[ 203],
| 99.00th=[ 410], 99.50th=[ 532], 99.90th=[ 1208], 99.95th=[ 2320],
| 99.99th=[80384]
bw (MB /s): min= 1, max= 1895, per=25.86%, avg=1086.90, stdev=644.09
lat (usec) : 2=0.28%, 4=0.45%, 10=0.33%, 20=1.28%, 50=51.13%
lat (usec) : 100=29.57%, 250=13.74%, 500=2.63%, 750=0.36%, 1000=0.10%
lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=62.35%, sys=101.72%, ctx=675275, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=1919: Wed Sep 18 16:10:18 2013
read : io=0B, bw=1060.7MB/s, iops=33941, runt= 11585msec
slat (usec): min=1, max=110469, avg=10.86, stdev=280.45
clat (usec): min=1, max=709969, avg=102.34, stdev=2376.47
lat (usec): min=16, max=710167, avg=113.20, stdev=2393.58
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 21], 10.00th=[ 26], 20.00th=[ 30],
| 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 55],
| 70.00th=[ 67], 80.00th=[ 91], 90.00th=[ 147], 95.00th=[ 219],
| 99.00th=[ 454], 99.50th=[ 596], 99.90th=[ 1544], 99.95th=[ 9408],
| 99.99th=[89600]
bw (KB /s): min=64939, max=1952832, per=24.28%, avg=1045128.50, stdev=610240.07
lat (usec) : 2=1.04%, 4=1.40%, 10=0.26%, 20=1.24%, 50=49.86%
lat (usec) : 100=28.44%, 250=13.90%, 500=3.08%, 750=0.49%, 1000=0.13%
lat (msec) : 2=0.08%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=62.21%, sys=96.01%, ctx=704936, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=1921: Wed Sep 18 16:10:18 2013
read : io=0B, bw=1050.1MB/s, iops=33628, runt= 11693msec
slat (usec): min=1, max=110205, avg=10.57, stdev=399.12
clat (usec): min=1, max=1486.8K, avg=104.47, stdev=3793.99
lat (usec): min=16, max=1486.8K, avg=115.04, stdev=3835.13
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 29],
| 30.00th=[ 34], 40.00th=[ 40], 50.00th=[ 47], 60.00th=[ 56],
| 70.00th=[ 70], 80.00th=[ 95], 90.00th=[ 149], 95.00th=[ 213],
| 99.00th=[ 422], 99.50th=[ 564], 99.90th=[ 1192], 99.95th=[ 2256],
| 99.99th=[81408]
bw (MB /s): min= 23, max= 2667, per=27.08%, avg=1138.34, stdev=685.83
lat (usec) : 2=0.19%, 4=0.35%, 10=0.36%, 20=2.17%, 50=49.65%
lat (usec) : 100=28.59%, 250=15.18%, 500=2.82%, 750=0.42%, 1000=0.12%
lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
cpu : usr=56.56%, sys=94.99%, ctx=697460, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=4203.6MB/s, minb=1050.1MB/s, maxb=1138.5MB/s, mint=10794msec, maxt=11693msec
WRITE: io=49152MB, aggrb=282663KB/s, minb=70665KB/s, maxb=89301KB/s, mint=140904msec, maxt=178062msec