virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
virt: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
fio-2.1.2
Starting 4 processes

virt: (groupid=0, jobs=1): err= 0: pid=2177: Wed May 28 22:07:58 2014
  read : io=523520KB, bw=28983KB/s, iops=452, runt= 18063msec
    slat (usec): min=23, max=4451, avg=42.52, stdev=61.50
    clat (usec): min=136, max=26872, avg=4360.12, stdev=1103.80
     lat (msec): min=2, max=26, avg= 4.40, stdev= 1.10
    clat percentiles (usec):
     |  1.00th=[ 3824],  5.00th=[ 3888], 10.00th=[ 3920], 20.00th=[ 3952],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4640], 95.00th=[ 4960],
     | 99.00th=[ 9792], 99.50th=[10304], 99.90th=[17024], 99.95th=[21888],
     | 99.99th=[26752]
    bw (KB  /s): min=25600, max=33280, per=25.02%, avg=29007.28, stdev=1840.52
  write: io=525056KB, bw=29068KB/s, iops=454, runt= 18063msec
    slat (usec): min=26, max=5046, avg=48.33, stdev=57.35
    clat (msec): min=3, max=29, avg= 4.36, stdev= 1.10
     lat (msec): min=3, max=29, avg= 4.41, stdev= 1.11
    clat percentiles (usec):
     |  1.00th=[ 3856],  5.00th=[ 3920], 10.00th=[ 3952], 20.00th=[ 3984],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4640], 95.00th=[ 4896],
     | 99.00th=[ 9920], 99.50th=[10560], 99.90th=[16320], 99.95th=[21376],
     | 99.99th=[29056]
    bw (KB  /s): min=25447, max=31744, per=25.02%, avg=29091.58, stdev=1503.68
    lat (usec) : 250=0.01%
    lat (msec) : 4=24.84%, 10=74.26%, 20=0.82%, 50=0.08%
  cpu          : usr=1.10%, sys=4.57%, ctx=16802, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0
virt: (groupid=0, jobs=1): err= 0: pid=2178: Wed May 28 22:07:58 2014
  read : io=523520KB, bw=28986KB/s, iops=452, runt= 18061msec
    slat (usec): min=20, max=4734, avg=44.14, stdev=65.97
    clat (usec): min=134, max=34582, avg=4367.22, stdev=1102.36
     lat (msec): min=2, max=34, avg= 4.41, stdev= 1.10
    clat percentiles (usec):
     |  1.00th=[ 3824],  5.00th=[ 3888], 10.00th=[ 3920], 20.00th=[ 3984],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4704], 95.00th=[ 4960],
     | 99.00th=[ 9920], 99.50th=[10304], 99.90th=[16512], 99.95th=[17024],
     | 99.99th=[34560]
    bw (KB  /s): min=25804, max=33280, per=25.03%, avg=29016.61, stdev=1835.93
  write: io=525056KB, bw=29071KB/s, iops=454, runt= 18061msec
    slat (usec): min=26, max=2297, avg=49.25, stdev=29.79
    clat (msec): min=3, max=28, avg= 4.35, stdev= 1.07
     lat (msec): min=3, max=28, avg= 4.40, stdev= 1.07
    clat percentiles (usec):
     |  1.00th=[ 3824],  5.00th=[ 3888], 10.00th=[ 3920], 20.00th=[ 3984],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4192],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4640], 95.00th=[ 4896],
     | 99.00th=[ 9920], 99.50th=[10304], 99.90th=[16192], 99.95th=[18816],
     | 99.99th=[28288]
    bw (KB  /s): min=25447, max=31936, per=25.03%, avg=29099.78, stdev=1497.17
    lat (usec) : 250=0.01%
    lat (msec) : 4=25.34%, 10=73.77%, 20=0.83%, 50=0.04%
  cpu          : usr=1.23%, sys=4.71%, ctx=16888, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0
virt: (groupid=0, jobs=1): err= 0: pid=2179: Wed May 28 22:07:58 2014
  read : io=523520KB, bw=28983KB/s, iops=452, runt= 18063msec
    slat (usec): min=15, max=4262, avg=41.83, stdev=65.79
    clat (usec): min=128, max=35128, avg=4352.56, stdev=1194.36
     lat (msec): min=1, max=35, avg= 4.39, stdev= 1.19
    clat percentiles (usec):
     |  1.00th=[ 3824],  5.00th=[ 3888], 10.00th=[ 3920], 20.00th=[ 3952],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4192],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4640], 95.00th=[ 4896],
     | 99.00th=[ 9792], 99.50th=[10432], 99.90th=[17280], 99.95th=[20864],
     | 99.99th=[35072]
    bw (KB  /s): min=25676, max=33402, per=25.02%, avg=29002.72, stdev=1797.83
  write: io=525056KB, bw=29068KB/s, iops=454, runt= 18063msec
    slat (usec): min=22, max=1784, avg=47.23, stdev=24.88
    clat (usec): min=296, max=35165, avg=4367.18, stdev=1113.83
     lat (msec): min=1, max=35, avg= 4.41, stdev= 1.11
    clat percentiles (usec):
     |  1.00th=[ 3856],  5.00th=[ 3920], 10.00th=[ 3952], 20.00th=[ 3984],
     | 30.00th=[ 4048], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4640], 95.00th=[ 4960],
     | 99.00th=[ 9792], 99.50th=[10176], 99.90th=[16320], 99.95th=[20608],
     | 99.99th=[35072]
    bw (KB  /s): min=25223, max=32127, per=25.02%, avg=29093.50, stdev=1608.39
    lat (usec) : 250=0.01%, 500=0.01%
    lat (msec) : 2=0.01%, 4=23.93%, 10=75.24%, 20=0.73%, 50=0.07%
  cpu          : usr=1.07%, sys=4.55%, ctx=16766, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0
virt: (groupid=0, jobs=1): err= 0: pid=2180: Wed May 28 22:07:58 2014
  read : io=523520KB, bw=28985KB/s, iops=452, runt= 18062msec
    slat (usec): min=24, max=5553, avg=44.79, stdev=80.00
    clat (usec): min=138, max=34979, avg=4358.30, stdev=1106.07
     lat (msec): min=2, max=35, avg= 4.40, stdev= 1.10
    clat percentiles (usec):
     |  1.00th=[ 3824],  5.00th=[ 3888], 10.00th=[ 3920], 20.00th=[ 3984],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4704], 95.00th=[ 4960],
     | 99.00th=[ 9792], 99.50th=[10304], 99.90th=[16192], 99.95th=[19584],
     | 99.99th=[35072]
    bw (KB  /s): min=25243, max=33280, per=25.02%, avg=29005.64, stdev=1815.27
  write: io=525056KB, bw=29070KB/s, iops=454, runt= 18062msec
    slat (usec): min=27, max=4550, avg=50.17, stdev=52.18
    clat (usec): min=372, max=34869, avg=4354.62, stdev=1175.13
     lat (msec): min=3, max=34, avg= 4.41, stdev= 1.17
    clat percentiles (usec):
     |  1.00th=[ 3856],  5.00th=[ 3888], 10.00th=[ 3952], 20.00th=[ 3984],
     | 30.00th=[ 4016], 40.00th=[ 4080], 50.00th=[ 4128], 60.00th=[ 4192],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4640], 95.00th=[ 4896],
     | 99.00th=[ 9920], 99.50th=[10432], 99.90th=[18304], 99.95th=[22144],
     | 99.99th=[35072]
    bw (KB  /s): min=25377, max=32000, per=25.02%, avg=29094.31, stdev=1546.49
    lat (usec) : 250=0.01%, 500=0.01%
    lat (msec) : 4=25.07%, 10=74.04%, 20=0.82%, 50=0.05%
  cpu          : usr=1.02%, sys=4.93%, ctx=16748, majf=0, minf=22
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8180/w=8204/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=2045.0MB, aggrb=115932KB/s, minb=28983KB/s, maxb=28986KB/s, mint=18061msec, maxt=18063msec
  WRITE: io=2051.0MB, aggrb=116272KB/s, minb=29068KB/s, maxb=29071KB/s, mint=18061msec, maxt=18063msec

Disk stats (read/write):
    dm-1: ios=32531/32589, merge=0/0, ticks=141395/170612, in_queue=312036, util=99.54%, aggrios=32720/32831, aggrmerge=0/12, aggrticks=142412/171944, aggrin_queue=314244, aggrutil=99.45%
  sdc: ios=32720/32831, merge=0/12, ticks=142412/171944, in_queue=314244, util=99.45%