virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4 ... virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4 fio-2.1.2 Starting 4 processes virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: (groupid=0, jobs=1): err= 0: pid=3738: Thu May 29 11:08:56 2014 read : io=523520KB, bw=859827B/s, iops=13, runt=623479msec slat (usec): min=10, max=740, avg=35.85, stdev=33.62 clat (msec): min=1, max=938, avg=77.79, stdev=66.30 lat (msec): min=1, max=938, avg=77.83, stdev=66.30 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 50], 50.00th=[ 77], 60.00th=[ 104], | 70.00th=[ 139], 80.00th=[ 147], 90.00th=[ 157], 95.00th=[ 163], | 99.00th=[ 188], 99.50th=[ 198], 99.90th=[ 396], 99.95th=[ 562], | 99.99th=[ 938] bw (KB /s): min= 79, max= 2402, per=25.10%, avg=842.11, stdev=354.52 write: io=525056KB, bw=862350B/s, iops=13, runt=623479msec slat (usec): min=18, max=863, avg=44.93, stdev=37.02 clat (msec): min=113, max=1080, avg=226.33, stdev=69.76 lat (msec): min=113, max=1080, avg=226.37, stdev=69.76 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 137], 10.00th=[ 143], 20.00th=[ 151], | 30.00th=[ 161], 40.00th=[ 194], 50.00th=[ 229], 60.00th=[ 262], | 70.00th=[ 285], 80.00th=[ 293], 90.00th=[ 310], 95.00th=[ 322], | 99.00th=[ 359], 99.50th=[ 375], 99.90th=[ 510], 99.95th=[ 685], | 99.99th=[ 1074] bw (KB /s): min= 79, max= 1508, per=25.05%, avg=842.85, stdev=166.32 lat (msec) : 2=0.02%, 4=1.41%, 10=17.13%, 20=0.19%, 50=1.38% lat (msec) : 100=8.86%, 250=49.33%, 500=21.59%, 750=0.08%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.05%, sys=0.16%, ctx=13625, majf=0, minf=21 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 virt: (groupid=0, jobs=1): err= 0: pid=3739: Thu May 29 11:08:56 2014 read : io=523520KB, bw=859997B/s, iops=13, runt=623356msec slat (usec): min=13, max=726, avg=37.34, stdev=32.95 clat (msec): min=2, max=939, avg=77.30, stdev=67.28 lat (msec): min=2, max=939, avg=77.34, stdev=67.28 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 49], 50.00th=[ 70], 60.00th=[ 105], | 70.00th=[ 139], 80.00th=[ 147], 90.00th=[ 155], 95.00th=[ 163], | 99.00th=[ 188], 99.50th=[ 198], 99.90th=[ 529], 99.95th=[ 545], | 99.99th=[ 938] bw (KB /s): min= 100, max= 2985, per=25.14%, avg=843.54, stdev=374.14 write: io=525056KB, bw=862520B/s, iops=13, runt=623356msec slat (usec): min=21, max=843, avg=46.61, stdev=41.72 clat (msec): min=118, max=1080, avg=226.73, stdev=69.39 lat (msec): min=118, max=1081, avg=226.78, stdev=69.39 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 137], 10.00th=[ 143], 20.00th=[ 151], | 30.00th=[ 161], 40.00th=[ 194], 50.00th=[ 231], 60.00th=[ 265], | 70.00th=[ 285], 80.00th=[ 293], 90.00th=[ 310], 95.00th=[ 322], | 99.00th=[ 363], 99.50th=[ 375], 99.90th=[ 449], 99.95th=[ 562], | 99.99th=[ 1074] bw (KB /s): min= 68, max= 1233, per=25.08%, avg=844.03, stdev=148.51 lat (msec) : 4=1.52%, 10=17.36%, 20=0.21%, 50=1.24%, 100=8.62% lat (msec) : 250=49.04%, 500=21.91%, 750=0.07%, 1000=0.02%, 2000=0.01% cpu : usr=0.05%, sys=0.16%, ctx=13639, majf=0, minf=21 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 virt: (groupid=0, jobs=1): err= 0: pid=3740: Thu May 29 11:08:56 2014 read : io=523520KB, bw=859047B/s, iops=13, runt=624045msec slat (usec): min=16, max=788, avg=35.47, stdev=29.41 clat (msec): min=1, max=938, avg=77.71, stdev=66.20 lat (msec): min=1, max=938, avg=77.74, stdev=66.20 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 49], 50.00th=[ 75], 60.00th=[ 106], | 70.00th=[ 139], 80.00th=[ 147], 90.00th=[ 157], 95.00th=[ 165], | 99.00th=[ 192], 99.50th=[ 200], 99.90th=[ 330], 99.95th=[ 400], | 99.99th=[ 938] bw (KB /s): min= 101, max= 2171, per=25.17%, avg=844.41, stdev=359.37 write: io=525056KB, bw=861568B/s, iops=13, runt=624045msec slat (usec): min=16, max=985, avg=44.76, stdev=38.93 clat (msec): min=113, max=1042, avg=226.67, stdev=70.47 lat (msec): min=113, max=1042, avg=226.72, stdev=70.48 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 137], 10.00th=[ 141], 20.00th=[ 151], | 30.00th=[ 161], 40.00th=[ 194], 50.00th=[ 231], 60.00th=[ 265], | 70.00th=[ 285], 80.00th=[ 293], 90.00th=[ 310], 95.00th=[ 322], | 99.00th=[ 359], 99.50th=[ 375], 99.90th=[ 562], 99.95th=[ 693], | 99.99th=[ 1045] bw (KB /s): min= 92, max= 1696, per=25.04%, avg=842.74, stdev=159.86 lat (msec) : 2=0.05%, 4=1.46%, 10=17.30%, 20=0.23%, 50=1.26% lat (msec) : 100=8.62%, 250=48.93%, 500=22.06%, 750=0.07%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=0.06%, sys=0.15%, ctx=13612, majf=0, minf=21 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 virt: (groupid=0, jobs=1): err= 0: pid=3741: Thu May 29 11:08:56 2014 read : io=523520KB, bw=860948B/s, iops=13, runt=622667msec slat (usec): min=17, max=823, avg=35.43, stdev=29.85 clat (msec): min=2, max=629, avg=76.81, stdev=64.70 lat (msec): min=2, max=629, avg=76.85, stdev=64.70 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 49], 50.00th=[ 70], 60.00th=[ 104], | 70.00th=[ 139], 80.00th=[ 147], 90.00th=[ 155], 95.00th=[ 163], | 99.00th=[ 188], 99.50th=[ 198], 99.90th=[ 237], 99.95th=[ 302], | 99.99th=[ 627] bw (KB /s): min= 107, max= 2593, per=25.22%, avg=846.15, stdev=363.59 write: io=525056KB, bw=863474B/s, iops=13, runt=622667msec slat (usec): min=21, max=927, avg=45.85, stdev=42.15 clat (msec): min=111, max=1076, avg=226.88, stdev=70.88 lat (msec): min=111, max=1076, avg=226.93, stdev=70.88 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 137], 10.00th=[ 143], 20.00th=[ 151], | 30.00th=[ 161], 40.00th=[ 194], 50.00th=[ 231], 60.00th=[ 262], | 70.00th=[ 285], 80.00th=[ 293], 90.00th=[ 310], 95.00th=[ 322], | 99.00th=[ 359], 99.50th=[ 375], 99.90th=[ 594], 99.95th=[ 685], | 99.99th=[ 1074] bw (KB /s): min= 159, max= 1409, per=25.09%, avg=844.26, stdev=158.02 lat (msec) : 4=1.40%, 10=17.40%, 20=0.12%, 50=1.44%, 100=8.78% lat (msec) : 250=49.05%, 500=21.69%, 750=0.09%, 1000=0.02%, 2000=0.01% cpu : usr=0.05%, sys=0.16%, ctx=13666, majf=0, minf=20 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=2045.0MB, aggrb=3355KB/s, minb=838KB/s, maxb=840KB/s, mint=622667msec, maxt=624045msec WRITE: io=2051.0MB, aggrb=3365KB/s, minb=841KB/s, maxb=843KB/s, mint=622667msec, maxt=624045msec