virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4 ... virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4 fio-2.1.2 Starting 4 processes virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: Laying out IO file(s) (1 file(s) / 1024MB) virt: (groupid=0, jobs=1): err= 0: pid=988: Thu May 29 08:14:19 2014 read : io=523520KB, bw=2401.8KB/s, iops=37, runt=217979msec slat (usec): min=10, max=84002, avg=87.60, stdev=1061.18 clat (usec): min=4, max=2026.9K, avg=96292.19, stdev=187215.26 lat (usec): min=655, max=2026.1K, avg=96381.82, stdev=187217.67 clat percentiles (usec): | 1.00th=[ 876], 5.00th=[ 6624], 10.00th=[ 9152], 20.00th=[12992], | 30.00th=[16512], 40.00th=[21120], 50.00th=[27264], 60.00th=[38144], | 70.00th=[61696], 80.00th=[115200], 90.00th=[272384], 95.00th=[436224], | 99.00th=[946176], 99.50th=[1236992], 99.90th=[1728512], 99.95th=[2007040], | 99.99th=[2023424] bw (KB /s): min= 81, max=13668, per=27.01%, avg=2594.77, stdev=2720.77 write: io=525056KB, bw=2408.8KB/s, iops=37, runt=217979msec slat (usec): min=16, max=57789, avg=85.20, stdev=781.14 clat (usec): min=614, max=546568, avg=10062.54, stdev=35825.65 lat (usec): min=685, max=546646, avg=10149.82, stdev=35918.03 clat percentiles (usec): | 1.00th=[ 788], 5.00th=[ 860], 10.00th=[ 908], 20.00th=[ 972], | 30.00th=[ 1032], 40.00th=[ 1208], 50.00th=[ 1384], 60.00th=[ 1656], | 70.00th=[ 2096], 80.00th=[ 7072], 90.00th=[18560], 95.00th=[35584], | 99.00th=[173056], 99.50th=[240640], 99.90th=[489472], 99.95th=[501760], | 99.99th=[544768] bw (KB /s): min= 107, max=15171, per=27.43%, avg=2642.88, stdev=2845.73 lat (usec) : 10=0.01%, 250=0.01%, 750=0.32%, 1000=12.59% lat (msec) : 2=21.97%, 4=4.62%, 10=8.06%, 20=16.78%, 50=16.68% lat (msec) : 100=6.56%, 250=6.73%, 500=3.68%, 750=1.10%, 1000=0.42% lat (msec) : 2000=0.43%, >=2000=0.03% cpu : usr=0.23%, sys=0.76%, ctx=15627, majf=0, minf=23 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 virt: (groupid=0, jobs=1): err= 0: pid=989: Thu May 29 08:14:19 2014 read : io=523520KB, bw=2877.6KB/s, iops=44, runt=181934msec slat (usec): min=10, max=11908, avg=68.03, stdev=147.08 clat (usec): min=701, max=2023.3K, avg=79398.75, stdev=112457.96 lat (usec): min=782, max=2023.4K, avg=79468.81, stdev=112460.83 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 16], | 30.00th=[ 22], 40.00th=[ 30], 50.00th=[ 41], 60.00th=[ 57], | 70.00th=[ 81], 80.00th=[ 117], 90.00th=[ 190], 95.00th=[ 277], | 99.00th=[ 515], 99.50th=[ 668], 99.90th=[ 1090], 99.95th=[ 1467], | 99.99th=[ 2024] bw (KB /s): min= 396, max= 7407, per=30.40%, avg=2920.40, stdev=1207.50 write: io=525056KB, bw=2885.1KB/s, iops=45, runt=181934msec slat (usec): min=12, max=36778, avg=87.83, stdev=594.47 clat (usec): min=506, max=590635, avg=9343.91, stdev=36655.18 lat (usec): min=586, max=590691, avg=9433.79, stdev=36711.48 clat percentiles (usec): | 1.00th=[ 756], 5.00th=[ 836], 10.00th=[ 884], 20.00th=[ 964], | 30.00th=[ 1048], 40.00th=[ 1240], 50.00th=[ 1368], 60.00th=[ 1576], | 70.00th=[ 1880], 80.00th=[ 2416], 90.00th=[13760], 95.00th=[29056], | 99.00th=[185344], 99.50th=[268288], 99.90th=[501760], 99.95th=[501760], | 99.99th=[593920] bw (KB /s): min= 120, max= 8743, per=30.51%, avg=2939.44, stdev=1488.43 lat (usec) : 750=0.47%, 1000=12.55% lat (msec) : 2=23.96%, 4=5.33%, 10=5.98%, 20=12.13%, 50=15.87% lat (msec) : 100=10.20%, 250=10.11%, 500=2.81%, 750=0.41%, 1000=0.09% lat (msec) : 2000=0.07%, >=2000=0.01% cpu : usr=0.26%, sys=0.95%, ctx=16038, majf=0, minf=23 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 virt: (groupid=0, jobs=1): err= 0: pid=990: Thu May 29 08:14:19 2014 read : io=523520KB, bw=3448.7KB/s, iops=53, runt=151830msec slat (usec): min=12, max=23438, avg=68.79, stdev=262.71 clat (usec): min=633, max=670928, avg=67035.72, stdev=77806.45 lat (usec): min=694, max=670997, avg=67106.56, stdev=77808.31 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 16], | 30.00th=[ 22], 40.00th=[ 29], 50.00th=[ 38], 60.00th=[ 51], | 70.00th=[ 71], 80.00th=[ 102], 90.00th=[ 163], 95.00th=[ 235], | 99.00th=[ 375], 99.50th=[ 424], 99.90th=[ 586], 99.95th=[ 635], | 99.99th=[ 668] bw (KB /s): min= 430, max= 6424, per=36.32%, avg=3489.35, stdev=1174.14 write: io=525056KB, bw=3458.2KB/s, iops=54, runt=151830msec slat (usec): min=12, max=96640, avg=104.44, stdev=1437.07 clat (usec): min=560, max=579610, avg=6978.63, stdev=29592.65 lat (usec): min=585, max=579737, avg=7085.12, stdev=29793.22 clat percentiles (usec): | 1.00th=[ 780], 5.00th=[ 844], 10.00th=[ 892], 20.00th=[ 964], | 30.00th=[ 1064], 40.00th=[ 1240], 50.00th=[ 1352], 60.00th=[ 1496], | 70.00th=[ 1752], 80.00th=[ 2008], 90.00th=[ 2800], 95.00th=[21120], | 99.00th=[160768], 99.50th=[220160], 99.90th=[288768], 99.95th=[444416], | 99.99th=[577536] bw (KB /s): min= 110, max= 9580, per=36.32%, avg=3499.55, stdev=1528.11 lat (usec) : 750=0.31%, 1000=12.25% lat (msec) : 2=27.42%, 4=5.85%, 10=5.09%, 20=10.63%, 50=16.58% lat (msec) : 100=10.39%, 250=9.30%, 500=2.06%, 750=0.13% cpu : usr=0.30%, sys=1.14%, ctx=16066, majf=0, minf=23 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 virt: (groupid=0, jobs=1): err= 0: pid=991: Thu May 29 08:14:19 2014 read : io=523520KB, bw=2963.7KB/s, iops=46, runt=176645msec slat (usec): min=9, max=61082, avg=73.41, stdev=676.38 clat (usec): min=157, max=932711, avg=77441.36, stdev=94713.21 lat (usec): min=635, max=932770, avg=77516.85, stdev=94721.28 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 16], | 30.00th=[ 23], 40.00th=[ 31], 50.00th=[ 43], 60.00th=[ 59], | 70.00th=[ 83], 80.00th=[ 118], 90.00th=[ 192], 95.00th=[ 273], | 99.00th=[ 457], 99.50th=[ 537], 99.90th=[ 734], 99.95th=[ 775], | 99.99th=[ 930] bw (KB /s): min= 250, max= 6770, per=31.25%, avg=3002.12, stdev=1187.48 write: io=525056KB, bw=2972.4KB/s, iops=46, runt=176645msec slat (usec): min=13, max=37000, avg=84.78, stdev=498.88 clat (usec): min=618, max=589379, avg=8706.75, stdev=35445.51 lat (usec): min=665, max=589483, avg=8793.56, stdev=35499.78 clat percentiles (usec): | 1.00th=[ 764], 5.00th=[ 844], 10.00th=[ 892], 20.00th=[ 964], | 30.00th=[ 1048], 40.00th=[ 1224], 50.00th=[ 1368], 60.00th=[ 1560], | 70.00th=[ 1848], 80.00th=[ 2192], 90.00th=[11328], 95.00th=[26240], | 99.00th=[173056], 99.50th=[242688], 99.90th=[501760], 99.95th=[544768], | 99.99th=[585728] bw (KB /s): min= 122, max= 9426, per=31.33%, avg=3017.98, stdev=1417.27 lat (usec) : 250=0.01%, 750=0.37%, 1000=12.70% lat (msec) : 2=24.91%, 4=5.43%, 10=5.76%, 20=11.23%, 50=15.47% lat (msec) : 100=10.55%, 250=10.39%, 500=2.76%, 750=0.38%, 1000=0.04% cpu : usr=0.28%, sys=0.96%, ctx=15983, majf=0, minf=21 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=2045.0MB, aggrb=9606KB/s, minb=2401KB/s, maxb=3448KB/s, mint=151830msec, maxt=217979msec WRITE: io=2051.0MB, aggrb=9634KB/s, minb=2408KB/s, maxb=3458KB/s, mint=151830msec, maxt=217979msec Disk stats (read/write): sda: ios=32710/32906, merge=0/54, ticks=2617904/1714191, in_queue=4340204, util=100.00%