szaydel
9/17/2013 - 6:28 PM
fio test with iscsi ZFS pool 8 vdevs, 16 disks - LUNs are 8K bs
fio test with iscsi ZFS pool 8 vdevs, 16 disks - LUNs are 8K bs
## First Run
root@cl01:/volumes/testp/d0# fio ~/rand.fio
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
Jobs: 1 (f=1): [___w____] [100.0% done] [0KB/11872KB/0KB /s] [0/371/0 iops] [eta 00m:00s]
random-writers: (groupid=0, jobs=1): err= 0: pid=13488: Tue Sep 17 11:04:54 2013
write: io=0B, bw=35108KB/s, iops=1097, runt=358404msec
slat (usec): min=2, max=289677, avg=48.10, stdev=1452.87
clat (usec): min=1, max=3328.3K, avg=3591.74, stdev=47962.99
lat (usec): min=23, max=3328.3K, avg=3639.84, stdev=47990.84
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 10],
| 30.00th=[ 41], 40.00th=[ 50], 50.00th=[ 58], 60.00th=[ 72],
| 70.00th=[ 92], 80.00th=[ 126], 90.00th=[ 466], 95.00th=[20096],
| 99.00th=[49920], 99.50th=[70144], 99.90th=[199680], 99.95th=[1335296],
| 99.99th=[2244608]
bw (KB /s): min= 39, max=956928, per=39.05%, avg=47044.08, stdev=117378.40
lat (usec) : 2=6.72%, 4=11.36%, 10=0.12%, 20=5.06%, 50=16.68%
lat (usec) : 100=33.61%, 250=14.96%, 500=1.59%, 750=0.41%, 1000=0.11%
lat (msec) : 2=0.12%, 4=0.05%, 10=2.39%, 20=2.77%, 50=3.20%
lat (msec) : 100=0.62%, 250=0.14%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.02%
cpu : usr=92.12%, sys=4.76%, ctx=924187, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=13489: Tue Sep 17 11:04:54 2013
write: io=0B, bw=32151KB/s, iops=1004, runt=391372msec
slat (usec): min=2, max=192786, avg=46.94, stdev=1236.45
clat (usec): min=1, max=3320.6K, avg=3926.89, stdev=48651.95
lat (usec): min=23, max=3320.7K, avg=3973.82, stdev=48668.02
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 17],
| 30.00th=[ 42], 40.00th=[ 50], 50.00th=[ 59], 60.00th=[ 75],
| 70.00th=[ 97], 80.00th=[ 145], 90.00th=[10048], 95.00th=[19840],
| 99.00th=[40192], 99.50th=[49920], 99.90th=[199680], 99.95th=[1400832],
| 99.99th=[2244608]
bw (KB /s): min= 94, max=864960, per=35.00%, avg=42163.88, stdev=103111.83
lat (usec) : 2=6.48%, 4=9.98%, 10=0.06%, 20=5.93%, 50=16.84%
lat (usec) : 100=31.89%, 250=12.19%, 500=1.36%, 750=0.28%, 1000=0.08%
lat (msec) : 2=0.09%, 4=0.04%, 10=6.00%, 20=5.28%, 50=2.97%
lat (msec) : 100=0.30%, 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.04%, >=2000=0.02%
cpu : usr=92.08%, sys=4.91%, ctx=896320, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=13490: Tue Sep 17 11:04:54 2013
write: io=0B, bw=32683KB/s, iops=1021, runt=385004msec
slat (usec): min=2, max=281921, avg=42.74, stdev=1250.50
clat (usec): min=1, max=3328.3K, avg=3868.21, stdev=48092.74
lat (usec): min=25, max=3328.4K, avg=3910.95, stdev=48111.19
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 2], 20.00th=[ 28],
| 30.00th=[ 43], 40.00th=[ 51], 50.00th=[ 59], 60.00th=[ 71],
| 70.00th=[ 89], 80.00th=[ 133], 90.00th=[ 9920], 95.00th=[19840],
| 99.00th=[40192], 99.50th=[60160], 99.90th=[203776], 99.95th=[1351680],
| 99.99th=[2244608]
bw (KB /s): min= 69, max=574976, per=35.45%, avg=42715.18, stdev=96166.65
lat (usec) : 2=5.59%, 4=8.56%, 10=0.13%, 20=4.60%, 50=19.15%
lat (usec) : 100=35.89%, 250=10.92%, 500=1.54%, 750=0.32%, 1000=0.13%
lat (msec) : 2=0.12%, 4=0.06%, 10=5.06%, 20=4.18%, 50=3.10%
lat (msec) : 100=0.43%, 250=0.14%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.02%
cpu : usr=92.87%, sys=4.89%, ctx=878610, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=13491: Tue Sep 17 11:04:54 2013
write: io=0B, bw=30120KB/s, iops=941, runt=417765msec
slat (usec): min=2, max=451896, avg=43.20, stdev=1656.55
clat (usec): min=1, max=3429.1K, avg=4199.85, stdev=48924.41
lat (usec): min=26, max=3429.1K, avg=4243.05, stdev=48963.67
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 2], 10.00th=[ 16], 20.00th=[ 36],
| 30.00th=[ 44], 40.00th=[ 52], 50.00th=[ 60], 60.00th=[ 71],
| 70.00th=[ 88], 80.00th=[ 129], 90.00th=[10048], 95.00th=[20096],
| 99.00th=[49920], 99.50th=[70144], 99.90th=[222208], 99.95th=[1368064],
| 99.99th=[2244608]
bw (KB /s): min= 112, max=1496566, per=32.29%, avg=38899.90, stdev=125774.20
lat (usec) : 2=3.83%, 4=6.05%, 10=0.03%, 20=2.11%, 50=25.38%
lat (usec) : 100=37.32%, 250=9.51%, 500=1.05%, 750=0.29%, 1000=0.09%
lat (msec) : 2=0.09%, 4=0.04%, 10=5.54%, 20=4.54%, 50=3.32%
lat (msec) : 100=0.54%, 250=0.17%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.05%, >=2000=0.02%
cpu : usr=91.90%, sys=4.80%, ctx=698440, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=13492: Tue Sep 17 11:04:54 2013
read : io=0B, bw=1779.7MB/s, iops=56930, runt= 6907msec
slat (usec): min=1, max=40015, avg= 6.26, stdev=80.15
clat (usec): min=1, max=264318, avg=61.53, stdev=1019.79
lat (usec): min=18, max=264325, avg=67.79, stdev=1023.95
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 23], 10.00th=[ 26], 20.00th=[ 30],
| 30.00th=[ 34], 40.00th=[ 38], 50.00th=[ 42], 60.00th=[ 47],
| 70.00th=[ 52], 80.00th=[ 62], 90.00th=[ 87], 95.00th=[ 121],
| 99.00th=[ 231], 99.50th=[ 282], 99.90th=[ 478], 99.95th=[ 692],
| 99.99th=[14528]
bw (MB /s): min= 439, max= 2258, per=26.28%, avg=1762.02, stdev=568.88
lat (usec) : 2=0.44%, 4=0.31%, 10=0.08%, 20=1.14%, 50=63.54%
lat (usec) : 100=26.95%, 250=6.78%, 500=0.68%, 750=0.05%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=87.07%, sys=156.37%, ctx=591979, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=13493: Tue Sep 17 11:04:54 2013
read : io=0B, bw=1675.1MB/s, iops=53630, runt= 7332msec
slat (usec): min=1, max=219815, avg= 6.77, stdev=353.40
clat (usec): min=1, max=309934, avg=65.42, stdev=1202.70
lat (usec): min=17, max=309936, avg=72.20, stdev=1259.88
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 30],
| 30.00th=[ 34], 40.00th=[ 38], 50.00th=[ 43], 60.00th=[ 48],
| 70.00th=[ 55], 80.00th=[ 69], 90.00th=[ 103], 95.00th=[ 147],
| 99.00th=[ 282], 99.50th=[ 354], 99.90th=[ 612], 99.95th=[ 812],
| 99.99th=[ 8640]
bw (MB /s): min= 545, max= 2179, per=23.51%, avg=1576.00, stdev=513.16
lat (usec) : 2=0.03%, 4=0.11%, 10=0.02%, 20=1.74%, 50=60.82%
lat (usec) : 100=26.71%, 250=9.17%, 500=1.23%, 750=0.11%, 1000=0.03%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=88.64%, sys=145.14%, ctx=637473, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=13494: Tue Sep 17 11:04:54 2013
read : io=0B, bw=1735.2MB/s, iops=55523, runt= 7082msec
slat (usec): min=1, max=109835, avg= 6.60, stdev=191.60
clat (usec): min=1, max=230940, avg=62.95, stdev=1043.67
lat (usec): min=16, max=230947, avg=69.55, stdev=1061.36
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 24], 10.00th=[ 26], 20.00th=[ 30],
| 30.00th=[ 34], 40.00th=[ 38], 50.00th=[ 42], 60.00th=[ 47],
| 70.00th=[ 54], 80.00th=[ 65], 90.00th=[ 93], 95.00th=[ 131],
| 99.00th=[ 249], 99.50th=[ 310], 99.90th=[ 498], 99.95th=[ 676],
| 99.99th=[13376]
bw (MB /s): min= 416, max= 2332, per=25.62%, avg=1717.25, stdev=550.03
lat (usec) : 2=0.05%, 4=0.09%, 10=0.09%, 20=1.34%, 50=61.90%
lat (usec) : 100=27.81%, 250=7.73%, 500=0.89%, 750=0.06%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=86.09%, sys=153.23%, ctx=600709, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=13495: Tue Sep 17 11:04:54 2013
read : io=0B, bw=1840.8MB/s, iops=58882, runt= 6678msec
slat (usec): min=1, max=119179, avg= 6.03, stdev=292.80
clat (usec): min=1, max=266450, avg=59.60, stdev=1154.71
lat (usec): min=18, max=266459, avg=65.63, stdev=1191.26
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 26], 20.00th=[ 30],
| 30.00th=[ 33], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 46],
| 70.00th=[ 51], 80.00th=[ 59], 90.00th=[ 81], 95.00th=[ 110],
| 99.00th=[ 199], 99.50th=[ 247], 99.90th=[ 410], 99.95th=[ 524],
| 99.99th=[ 9920]
bw (MB /s): min= 381, max= 2326, per=27.27%, avg=1828.34, stdev=556.09
lat (usec) : 2=0.31%, 4=0.23%, 10=0.06%, 20=0.58%, 50=66.21%
lat (usec) : 100=26.26%, 250=5.87%, 500=0.42%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=93.01%, sys=159.31%, ctx=572929, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=6703.8MB/s, minb=1675.1MB/s, maxb=1840.8MB/s, mint=6678msec, maxt=7332msec
WRITE: io=49152MB, aggrb=120478KB/s, minb=30119KB/s, maxb=35108KB/s, mint=358404msec, maxt=417765msec
## Second Run
root@cl01:/volumes/p0/test/d0# fio ~/rand.fio
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-writers: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
...
random-readers: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=solarisaio, iodepth=4
fio-2.1.2
Starting 8 processes
random-writers: Laying out IO file(s) (1 file(s) / 12288MB)
random-writers: Laying out IO file(s) (1 file(s) / 12288MB)
random-writers: Laying out IO file(s) (1 file(s) / 12288MB)
random-writers: Laying out IO file(s) (1 file(s) / 12288MB)
random-readers: Laying out IO file(s) (1 file(s) / 12288MB)
Jobs: 8 (f=8): [wwwwrrrr] [0.0% done] [66646KB/252.7MB/0KB /s] [2082/8084/0 iops] [eta 01d:11h:01m:48Jobs: 8 (f=8): [wwwwrrrr] [0.0% done] [27697KB/447.2MB/0KB /s] [865/14.4K/0 iops] [eta 01d:08h:13m:32Jobs: 8 (f=8): [wwwwrrrr] [0.0% done] [70240KB/712.8MB/0KB /s] [2195/22.9K/0 iops] [eta 22h:04m:20s] Jobs: 1 (f=1): [____r___] [99.5% done] [601.4MB/0KB/0KB /s] [19.3K/0/0 iops] [eta 00m:03s]
random-writers: (groupid=0, jobs=1): err= 0: pid=3864: Tue Sep 17 13:56:32 2013
write: io=0B, bw=49994KB/s, iops=1562, runt=251688msec
slat (usec): min=2, max=340260, avg=20.87, stdev=1188.20
clat (usec): min=1, max=14810K, avg=2532.07, stdev=60513.50
lat (usec): min=19, max=14810K, avg=2552.95, stdev=60528.61
clat percentiles (usec):
| 1.00th=[ 20], 5.00th=[ 25], 10.00th=[ 27], 20.00th=[ 30],
| 30.00th=[ 32], 40.00th=[ 35], 50.00th=[ 41], 60.00th=[ 54],
| 70.00th=[ 90], 80.00th=[ 118], 90.00th=[ 207], 95.00th=[ 3152],
| 99.00th=[36608], 99.50th=[80384], 99.90th=[366592], 99.95th=[618496],
| 99.99th=[1908736]
bw (KB /s): min= 117, max=620480, per=35.51%, avg=59956.50, stdev=88047.61
lat (usec) : 2=0.02%, 4=0.58%, 10=0.18%, 20=0.21%, 50=56.85%
lat (usec) : 100=16.77%, 250=17.03%, 500=2.29%, 750=0.44%, 1000=0.21%
lat (msec) : 2=0.23%, 4=0.38%, 10=1.73%, 20=1.38%, 50=0.95%
lat (msec) : 100=0.34%, 250=0.25%, 500=0.09%, 750=0.03%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=89.61%, sys=9.10%, ctx=505686, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=3865: Tue Sep 17 13:56:32 2013
write: io=0B, bw=42209KB/s, iops=1319, runt=298113msec
slat (usec): min=2, max=239927, avg=20.97, stdev=840.42
clat (usec): min=1, max=15073K, avg=3006.61, stdev=61626.08
lat (usec): min=18, max=15073K, avg=3027.58, stdev=61640.93
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 37], 50.00th=[ 42], 60.00th=[ 52],
| 70.00th=[ 89], 80.00th=[ 117], 90.00th=[ 245], 95.00th=[ 7328],
| 99.00th=[45312], 99.50th=[99840], 99.90th=[428032], 99.95th=[675840],
| 99.99th=[1892352]
bw (KB /s): min= 46, max=452672, per=29.75%, avg=50233.55, stdev=77682.08
lat (usec) : 2=0.01%, 4=0.99%, 10=0.48%, 20=0.15%, 50=57.06%
lat (usec) : 100=15.00%, 250=16.43%, 500=2.17%, 750=0.56%, 1000=0.27%
lat (msec) : 2=0.22%, 4=0.50%, 10=2.31%, 20=1.82%, 50=1.11%
lat (msec) : 100=0.42%, 250=0.32%, 500=0.11%, 750=0.03%, 1000=0.02%
lat (msec) : 2000=0.02%, >=2000=0.01%
cpu : usr=91.17%, sys=7.62%, ctx=507827, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=3866: Tue Sep 17 13:56:32 2013
write: io=0B, bw=43394KB/s, iops=1356, runt=289969msec
slat (usec): min=2, max=541670, avg=23.08, stdev=1301.90
clat (usec): min=1, max=14993K, avg=2920.00, stdev=61699.11
lat (usec): min=17, max=14993K, avg=2943.08, stdev=61714.81
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 24], 10.00th=[ 26], 20.00th=[ 30],
| 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 50],
| 70.00th=[ 88], 80.00th=[ 114], 90.00th=[ 231], 95.00th=[ 6752],
| 99.00th=[43776], 99.50th=[100864], 99.90th=[399360], 99.95th=[675840],
| 99.99th=[1892352]
bw (KB /s): min= 91, max=499840, per=30.29%, avg=51138.00, stdev=79461.21
lat (usec) : 2=0.05%, 4=1.13%, 10=0.32%, 20=0.44%, 50=57.80%
lat (usec) : 100=14.95%, 250=15.82%, 500=2.24%, 750=0.50%, 1000=0.22%
lat (msec) : 2=0.20%, 4=0.44%, 10=2.19%, 20=1.71%, 50=1.10%
lat (msec) : 100=0.38%, 250=0.32%, 500=0.11%, 750=0.03%, 1000=0.02%
lat (msec) : 2000=0.02%, >=2000=0.01%
cpu : usr=90.00%, sys=7.77%, ctx=511529, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-writers: (groupid=0, jobs=1): err= 0: pid=3867: Tue Sep 17 13:56:32 2013
write: io=0B, bw=62335KB/s, iops=1947, runt=201861msec
slat (usec): min=2, max=552199, avg=23.95, stdev=1494.81
clat (usec): min=1, max=14853K, avg=2023.92, stdev=59386.86
lat (usec): min=22, max=14853K, avg=2047.87, stdev=59406.06
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 30],
| 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 50],
| 70.00th=[ 81], 80.00th=[ 105], 90.00th=[ 179], 95.00th=[ 524],
| 99.00th=[26496], 99.50th=[59648], 99.90th=[296960], 99.95th=[497664],
| 99.99th=[1925120]
bw (KB /s): min= 321, max=614666, per=45.58%, avg=76959.88, stdev=94528.57
lat (usec) : 2=0.05%, 4=1.52%, 10=0.65%, 20=0.23%, 50=57.45%
lat (usec) : 100=17.69%, 250=14.86%, 500=2.47%, 750=0.57%, 1000=0.26%
lat (msec) : 2=0.22%, 4=0.31%, 10=1.35%, 20=1.10%, 50=0.71%
lat (msec) : 100=0.26%, 250=0.18%, 500=0.08%, 750=0.02%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=89.09%, sys=11.12%, ctx=510713, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=3868: Tue Sep 17 13:56:32 2013
read : io=0B, bw=22823KB/s, iops=713, runt=551328msec
slat (usec): min=1, max=10184, avg= 3.58, stdev=17.22
clat (usec): min=1, max=10331K, avg=5601.64, stdev=56176.63
lat (usec): min=11, max=10331K, avg=5605.22, stdev=56177.61
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 27],
| 70.00th=[ 36], 80.00th=[ 6944], 90.00th=[12864], 95.00th=[19840],
| 99.00th=[50432], 99.50th=[110080], 99.90th=[497664], 99.95th=[782336],
| 99.99th=[2056192]
bw (KB /s): min= 3, max=696960, per=28.09%, avg=25642.02, stdev=63975.61
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=47.29%, 50=26.68%
lat (usec) : 100=0.72%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.05%
lat (msec) : 2=0.04%, 4=0.89%, 10=9.79%, 20=9.54%, 50=3.95%
lat (msec) : 100=0.46%, 250=0.31%, 500=0.13%, 750=0.05%, 1000=0.02%
lat (msec) : 2000=0.02%, >=2000=0.01%
cpu : usr=95.08%, sys=2.46%, ctx=525493, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=3869: Tue Sep 17 13:56:32 2013
read : io=0B, bw=23111KB/s, iops=722, runt=544448msec
slat (usec): min=0, max=327549, avg= 5.23, stdev=644.12
clat (usec): min=1, max=15704K, avg=5530.24, stdev=62206.94
lat (usec): min=11, max=15704K, avg=5535.47, stdev=62215.16
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23],
| 70.00th=[ 29], 80.00th=[ 6880], 90.00th=[12864], 95.00th=[19840],
| 99.00th=[50432], 99.50th=[110080], 99.90th=[473088], 99.95th=[765952],
| 99.99th=[1728512]
bw (KB /s): min= 3, max=533037, per=28.34%, avg=25872.05, stdev=52697.56
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=47.75%, 50=26.63%
lat (usec) : 100=0.32%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.05%
lat (msec) : 2=0.03%, 4=0.97%, 10=9.77%, 20=9.57%, 50=3.87%
lat (msec) : 100=0.46%, 250=0.32%, 500=0.13%, 750=0.04%, 1000=0.02%
lat (msec) : 2000=0.02%, >=2000=0.01%
cpu : usr=95.22%, sys=2.39%, ctx=519929, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=3870: Tue Sep 17 13:56:32 2013
read : io=0B, bw=23068KB/s, iops=720, runt=545479msec
slat (usec): min=0, max=169914, avg= 4.02, stdev=286.15
clat (usec): min=2, max=10663K, avg=5541.07, stdev=56546.92
lat (usec): min=11, max=10663K, avg=5545.09, stdev=56551.06
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15],
| 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 21],
| 70.00th=[ 26], 80.00th=[ 6944], 90.00th=[12992], 95.00th=[20096],
| 99.00th=[51456], 99.50th=[105984], 99.90th=[468992], 99.95th=[757760],
| 99.99th=[2277376]
bw (KB /s): min= 16, max=581888, per=28.35%, avg=25880.54, stdev=53857.31
lat (usec) : 4=0.01%, 10=0.01%, 20=55.13%, 50=19.42%, 100=0.27%
lat (usec) : 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.04%
lat (msec) : 2=0.03%, 4=0.84%, 10=9.54%, 20=9.68%, 50=3.96%
lat (msec) : 100=0.51%, 250=0.32%, 500=0.12%, 750=0.04%, 1000=0.02%
lat (msec) : 2000=0.02%, >=2000=0.01%
cpu : usr=94.88%, sys=2.36%, ctx=518845, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
random-readers: (groupid=0, jobs=1): err= 0: pid=3871: Tue Sep 17 13:56:32 2013
read : io=0B, bw=23154KB/s, iops=723, runt=543452msec
slat (usec): min=0, max=446161, avg= 6.47, stdev=774.69
clat (usec): min=1, max=9225.1K, avg=5519.16, stdev=56539.55
lat (usec): min=11, max=9225.1K, avg=5525.63, stdev=56552.49
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16],
| 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 26],
| 70.00th=[ 38], 80.00th=[ 6624], 90.00th=[13120], 95.00th=[20352],
| 99.00th=[50944], 99.50th=[98816], 99.90th=[468992], 99.95th=[823296],
| 99.99th=[2441216]
bw (KB /s): min= 10, max=572736, per=28.83%, avg=26315.47, stdev=47072.23
lat (usec) : 2=0.01%, 4=0.09%, 10=0.04%, 20=39.75%, 50=31.95%
lat (usec) : 100=1.74%, 250=2.21%, 500=0.37%, 750=0.11%, 1000=0.07%
lat (msec) : 2=0.07%, 4=0.53%, 10=8.39%, 20=9.51%, 50=4.13%
lat (msec) : 100=0.54%, 250=0.29%, 500=0.11%, 750=0.04%, 1000=0.02%
lat (msec) : 2000=0.02%, >=2000=0.01%
cpu : usr=94.94%, sys=2.60%, ctx=516461, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=49152MB, aggrb=91291KB/s, minb=22822KB/s, maxb=23153KB/s, mint=543452msec, maxt=551328msec
WRITE: io=49152MB, aggrb=168834KB/s, minb=42208KB/s, maxb=62334KB/s, mint=201861msec, maxt=298113msec