virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
fio-2.1.2
Starting 4 processes
virt: Laying out IO file(s) (1 file(s) / 1024MB)
virt: Laying out IO file(s) (1 file(s) / 1024MB)
virt: Laying out IO file(s) (1 file(s) / 1024MB)
virt: Laying out IO file(s) (1 file(s) / 1024MB)

virt: (groupid=0, jobs=1): err= 0: pid=2195: Wed May 28 22:12:50 2014
  read : io=523520KB, bw=2600.4KB/s, iops=40, runt=201329msec
    slat (usec): min=23, max=24586, avg=65.89, stdev=306.38
    clat (usec): min=305, max=1765.7K, avg=84912.67, stdev=124153.30
     lat (usec): min=367, max=1765.8K, avg=84979.16, stdev=124150.29
    clat percentiles (usec):
     |  1.00th=[  780],  5.00th=[ 6944], 10.00th=[ 9536], 20.00th=[14144],
     | 30.00th=[19840], 40.00th=[28032], 50.00th=[40704], 60.00th=[57600],
     | 70.00th=[82432], 80.00th=[125440], 90.00th=[209920], 95.00th=[309248],
     | 99.00th=[593920], 99.50th=[790528], 99.90th=[1204224], 99.95th=[1286144],
     | 99.99th=[1761280]
    bw (KB  /s): min=   82, max=12416, per=25.85%, avg=2688.32, stdev=1545.40
  write: io=525056KB, bw=2607.1KB/s, iops=40, runt=201329msec
    slat (usec): min=31, max=140675, avg=132.77, stdev=1945.34
    clat (usec): min=346, max=1355.5K, avg=13280.27, stdev=57149.27
     lat (usec): min=404, max=1355.6K, avg=13413.69, stdev=57202.63
    clat percentiles (usec):
     |  1.00th=[  358],  5.00th=[  374], 10.00th=[  434], 20.00th=[  446],
     | 30.00th=[  644], 40.00th=[  852], 50.00th=[ 1272], 60.00th=[ 1320],
     | 70.00th=[ 1496], 80.00th=[ 5728], 90.00th=[18048], 95.00th=[63232],
     | 99.00th=[257024], 99.50th=[382976], 99.90th=[831488], 99.95th=[946176],
     | 99.99th=[1351680]
    bw (KB  /s): min=  121, max=10709, per=25.96%, avg=2708.14, stdev=1769.64
    lat (usec) : 500=12.91%, 750=6.04%, 1000=3.25%
    lat (msec) : 2=16.32%, 4=1.65%, 10=7.59%, 20=12.91%, 50=14.75%
    lat (msec) : 100=9.90%, 250=10.51%, 500=3.19%, 750=0.66%, 1000=0.20%
    lat (msec) : 2000=0.13%
  cpu          : usr=0.11%, sys=0.54%, ctx=16504, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0
virt: (groupid=0, jobs=1): err= 0: pid=2196: Wed May 28 22:12:50 2014
  read : io=523520KB, bw=2947.6KB/s, iops=46, runt=177615msec
    slat (usec): min=24, max=59936, avg=81.73, stdev=987.31
    clat (usec): min=149, max=1054.1K, avg=74995.11, stdev=93418.23
     lat (usec): min=369, max=1054.2K, avg=75077.47, stdev=93411.06
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    8], 10.00th=[   10], 20.00th=[   16],
     | 30.00th=[   22], 40.00th=[   31], 50.00th=[   42], 60.00th=[   57],
     | 70.00th=[   80], 80.00th=[  116], 90.00th=[  180], 95.00th=[  260],
     | 99.00th=[  437], 99.50th=[  529], 99.90th=[  840], 99.95th=[  979],
     | 99.99th=[ 1057]
    bw (KB  /s): min=  113, max= 6898, per=29.26%, avg=3043.36, stdev=1217.82
  write: io=525056KB, bw=2956.2KB/s, iops=46, runt=177615msec
    slat (usec): min=33, max=140655, avg=128.77, stdev=2069.57
    clat (usec): min=258, max=1000.6K, avg=11590.37, stdev=57029.08
     lat (usec): min=403, max=1000.7K, avg=11719.76, stdev=57077.03
    clat percentiles (usec):
     |  1.00th=[  362],  5.00th=[  378], 10.00th=[  434], 20.00th=[  446],
     | 30.00th=[  612], 40.00th=[  748], 50.00th=[ 1224], 60.00th=[ 1304],
     | 70.00th=[ 1352], 80.00th=[ 1528], 90.00th=[ 7776], 95.00th=[55040],
     | 99.00th=[244736], 99.50th=[362496], 99.90th=[913408], 99.95th=[929792],
     | 99.99th=[1003520]
    bw (KB  /s): min=  140, max= 7409, per=29.16%, avg=3042.19, stdev=1466.35
    lat (usec) : 250=0.01%, 500=13.49%, 750=6.57%, 1000=3.19%
    lat (msec) : 2=19.84%, 4=1.70%, 10=5.92%, 20=9.95%, 50=14.89%
    lat (msec) : 100=10.81%, 250=10.45%, 500=2.73%, 750=0.31%, 1000=0.13%
    lat (msec) : 2000=0.02%
  cpu          : usr=0.14%, sys=0.59%, ctx=16858, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0
virt: (groupid=0, jobs=1): err= 0: pid=2197: Wed May 28 22:12:50 2014
  read : io=523520KB, bw=2923.2KB/s, iops=45, runt=179092msec
    slat (usec): min=20, max=99838, avg=91.84, stdev=1411.97
    clat (usec): min=160, max=1512.4K, avg=75755.06, stdev=105522.71
     lat (usec): min=382, max=1512.9K, avg=75847.54, stdev=105514.14
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    8], 10.00th=[   10], 20.00th=[   15],
     | 30.00th=[   21], 40.00th=[   29], 50.00th=[   40], 60.00th=[   56],
     | 70.00th=[   76], 80.00th=[  112], 90.00th=[  186], 95.00th=[  269],
     | 99.00th=[  469], 99.50th=[  586], 99.90th=[ 1156], 99.95th=[ 1287],
     | 99.99th=[ 1516]
    bw (KB  /s): min=  124, max= 6144, per=29.37%, avg=3055.29, stdev=1223.87
  write: io=525056KB, bw=2931.8KB/s, iops=45, runt=179092msec
    slat (usec): min=35, max=140660, avg=114.41, stdev=1768.12
    clat (usec): min=345, max=1441.6K, avg=11547.93, stdev=62451.29
     lat (usec): min=415, max=1441.7K, avg=11663.01, stdev=62476.14
    clat percentiles (usec):
     |  1.00th=[  362],  5.00th=[  378], 10.00th=[  434], 20.00th=[  446],
     | 30.00th=[  596], 40.00th=[  756], 50.00th=[ 1224], 60.00th=[ 1304],
     | 70.00th=[ 1352], 80.00th=[ 1544], 90.00th=[ 8896], 95.00th=[37632],
     | 99.00th=[232448], 99.50th=[350208], 99.90th=[995328], 99.95th=[1044480],
     | 99.99th=[1433600]
    bw (KB  /s): min=   80, max= 9325, per=29.37%, avg=3063.24, stdev=1532.25
    lat (usec) : 250=0.01%, 500=13.56%, 750=6.50%, 1000=3.08%
    lat (msec) : 2=19.73%, 4=1.62%, 10=6.32%, 20=10.52%, 50=14.89%
    lat (msec) : 100=10.77%, 250=9.75%, 500=2.72%, 750=0.27%, 1000=0.13%
    lat (msec) : 2000=0.14%
  cpu          : usr=0.14%, sys=0.59%, ctx=16985, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0
virt: (groupid=0, jobs=1): err= 0: pid=2198: Wed May 28 22:12:50 2014
  read : io=523520KB, bw=2629.9KB/s, iops=41, runt=199069msec
    slat (usec): min=25, max=99063, avg=89.77, stdev=1365.24
    clat (usec): min=112, max=1392.1K, avg=83373.43, stdev=118987.34
     lat (usec): min=369, max=1392.1K, avg=83463.84, stdev=118977.09
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    7], 10.00th=[   10], 20.00th=[   15],
     | 30.00th=[   21], 40.00th=[   28], 50.00th=[   40], 60.00th=[   57],
     | 70.00th=[   81], 80.00th=[  122], 90.00th=[  206], 95.00th=[  310],
     | 99.00th=[  603], 99.50th=[  734], 99.90th=[  979], 99.95th=[ 1156],
     | 99.99th=[ 1401]
    bw (KB  /s): min=   64, max= 9708, per=26.35%, avg=2740.70, stdev=1540.11
  write: io=525056KB, bw=2637.6KB/s, iops=41, runt=199069msec
    slat (usec): min=38, max=140657, avg=121.47, stdev=1860.80
    clat (usec): min=349, max=1002.9K, avg=13698.39, stdev=66153.66
     lat (usec): min=405, max=1002.9K, avg=13820.49, stdev=66192.16
    clat percentiles (usec):
     |  1.00th=[  362],  5.00th=[  378], 10.00th=[  434], 20.00th=[  446],
     | 30.00th=[  652], 40.00th=[  876], 50.00th=[ 1272], 60.00th=[ 1320],
     | 70.00th=[ 1448], 80.00th=[ 2992], 90.00th=[15552], 95.00th=[36096],
     | 99.00th=[321536], 99.50th=[489472], 99.90th=[962560], 99.95th=[995328],
     | 99.99th=[1003520]
    bw (KB  /s): min=   71, max= 9836, per=26.41%, avg=2755.14, stdev=1757.17
    lat (usec) : 250=0.02%, 500=12.84%, 750=5.83%, 1000=3.12%
    lat (msec) : 2=17.58%, 4=1.73%, 10=7.50%, 20=12.41%, 50=14.97%
    lat (msec) : 100=9.86%, 250=9.86%, 500=3.19%, 750=0.78%, 1000=0.25%
    lat (msec) : 2000=0.06%
  cpu          : usr=0.12%, sys=0.53%, ctx=16540, majf=0, minf=22
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=2045.0MB, aggrb=10401KB/s, minb=2600KB/s, maxb=2947KB/s, mint=177615msec, maxt=201329msec
  WRITE: io=2051.0MB, aggrb=10431KB/s, minb=2607KB/s, maxb=2956KB/s, mint=177615msec, maxt=201329msec

Disk stats (read/write):
    dm-0: ios=32841/33299, merge=0/0, ticks=2623746/506809, in_queue=3130698, util=100.00%, aggrios=32855/33392, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md127: ios=32855/33392, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=16426/33225, aggrmerge=1/168, aggrticks=1311820/306619, aggrin_queue=1618332, aggrutil=98.91%
  sda: ios=8494/33223, merge=0/171, ticks=464540/232964, in_queue=697442, util=96.18%
  sdb: ios=24359/33228, merge=2/166, ticks=2159100/380274, in_queue=2539222, util=98.91%