Thank you, good afternoon.
My name is Richard Jones and I work at Red Hat and I'm going to be talking today about libguestfs
which is a C library and a suite of tools
including virt-builder, virt-sysprep, virt-sparsify and much more
but before I start ...
I just wanted to give you an idea of what exactly I'm talking about today
So I'm going to build a Fedora 20 complete virtual machine
It'll take about 20 seconds or so
This is starting from a template and cloning it
giving it a random seed and root password
and we can use tools like guestfish to open this disk image
and make automated changes inside it
For example I can .. if you wanted to
have a virtual machine that always started up with a particular message
you could use something like guestfish or our C API to make these
automated changes to it.
So that's what libguestfs basically does.
I'm going to start running a little script
looks a bit complicated but all it is, is a while-loop
which goes round and round
and it's going to keep running virt-builder while I'm talking
Let's set that going.
And we'll come back to that in just 10 minutes.
What does virt-builder do?
Virt-builder can build you a CentOS 6 guest
It can change the format of the output disk image
or the size using virt-resize
using libguesfs to go in there and resize the partitions
and the logical volumes correctly
so the guest is already resized when it starts running.
It can change the hostname, timezone
You can install packages.
You can edit configuration files in many different ways.
You can run scripts at various points during the build
including when it's building or if you prefer, in this case,
when it first boots.
Virt-builder does not install guests from scratch
and that's really what I'm going to talk about today
It doesn't run Anaconda or debian-installer
because that would take far too long.
Instead it clones clean, cryptographically signed
compressed templates, and it customizes them for you.
Libguestfs can safely and securely modify disk images
without requiring root.
Virt-builder comes with a handful of templates
but what I'm really going to talk about today is
how you can prepare your own.
Over the years we have come up with a good way
to prepare and distribute small templates.
This is the method that we use now.
Virt-install runs the regular O.S. installer
so it does run Anaconda or debian-installer
or Windows installers and so on.
You can completely automate it using preseed
or kickstart.
You could also use Oz here instead.
Virt-sysprep unconfigures the guest.
Using libguestfs it's going to go in there
It can remove things like ssh keys
which you don't want to have when you clone a guest
you don't want them to all have the same ssh key
It can do things like removing log files
You don't want to give away how your guest was created
so it's a good idea to scrub things like log files
It can get rid of persistent network configuration
There are about 20 different operations that virt-sysprep can do
Virt-sparsify is the next one
When you've installed a guest
virt-sparsify can go in there and make that guest disk image
"sparse" or "thin-provisioned"
Using libguestfs again it goes and looks deep inside
the filesystem.
It can find unused space in deleted files
inside unused swap
in deleted logical volumes or partitions
that are not used.
It can find blocks which although not zeroes
in the disk image and actually not used.
It can give that space back to the operating system
by making the disk image sparse.
As you can see it is very simple to use.
The final step is xz.
We really like xz for a few reasons
xz is nearly best in class compression
It's not perhaps the very best compression
but it's way better than gzip or bzip2
Importantly xz preserves sparseness
If you've gone to the trouble of running virt-sparsify
you don't want to decompress your cloned disk image
and find that your sparseness is gone
it's all fully allocated again
The other thing about xz is it has a brilliant API
and a brilliant file format
It's all well-documented
and it allows us to do all sorts of clever stuff
We have a sister project called nbdkit
It's an NBD (Network Block Device) server
It allows you to serve an xz-compressed
disk image, presented as uncompressed
to someone consuming NBD
but it doesn't do that by uncompressing the whole xz
image before it starts.
It just uncompresses the bits you're reading at the time.
It can do this because the xz API allows random access
to the compressed file.
The other thing we've done with xz:
virt-builder includes a custom uncompressor
which is way faster than unxz or xzcat
On my laptop it's four times faster
When you see virt-builder running
it has an uncompress step that takes 10 seconds
that normally takes 40 seconds
if you just used xzcat
So what's the result of all this?
I'm going to take just one row of this, Fedora 20
These are all in megabytes
The Fedora 20 container is 6 gigabytes
that doesn't really mean much
when you're installing a guest you just plug in a number
in this case it was 6 gigabytes
Most of that is empty space, but 826 megabytes
is used by the Fedora core packages
After sparsification and xz compression
it's down to 174 megabytes.
This isn't the most minimal disk image ever.
It's possible to go much smaller.
However, we have done this without doing any nasty tricks
like deleting files from the guest.
We don't delete the documentation.
We don't remove the package manager.
The only things we change are configuration files
you're allowed to change those
and we've scrubbed log files.
So we've got a trustworthy
faithful representation of that guest.
We can distribute this over broadband.
Virt-builder caches the images so you only have
to download it once.
You can create 1000s of F20 guests from a single download.
I set virt-builder running earlier on
It's created 24 guests
and it's working on guest number 25 at the moment.
You can see that uncompression step took ~9 seconds.
Virt-builder spits out images every 15 .. 20 .. 25 seconds
it really depends on exactly how you configure the guest.
If you install packages it takes a bit longer.
If you use virt-resize that takes a bit longer too.
But a basic guest usually about 20 seconds or so.
Finally, this is my last slide
libguestfs.org is the website
We have many more tools using libguestfs
We have a stable C API
we have bindings in about a dozen programming languages
Almost certainly your Linux distro includes libguestfs
so it's just an "apt-get" or "yum" install or use your package manager
and it'll just work for you.
Join us on the IRC channel [#libguestfs on FreeNode]
or the mailing list [see libguestfs.org]
Questions
Q: Can you use virt-builder with a base image
to create an overlay, which is much more efficient
and much smaller?
A: It doesn't right now, but it will do in about 1 month.
It is the most requested feature.
Q: virt-diff: Is it the ordinary diff, or a custom diff
built on top of libguestfs?
A: It is a special tool, but it does use regular diff.
It takes two disk images and it uses the libguestfs API
to iterate over the filesystems in both images
that's the custom code
but when it finds a file that has changed it runs
diff on that file and presents you with diff output.
Q: lrzip and thin provisioning: the results were a little bit better
A: You're correct that lrzip .. xz is not absolute the best
compression tool out there. There are ones which
will give you a megabyte or two better than xz.
But it's still extremely good compression.
Because it has such a well-defined API
it's useful in many other ways.
Q: Will libguestfs manipulate any sort of guest image?
A: Yes. We use Linux kernel code and Linux utilities
If you could plug the disk into a Linux computer
and Linux could read it, then we can read it.
Q: You support qcow2
A: Yes, we use KVM to decode the qcow2 layer.
Q: Do you support formats like VHD?
A: Yes we support anything that qemu supports
and qemu supports VHD, VMDK, raw.
We support everything that qemu & Linux supports.
Q: Do you plan to get templates from other sources?
A: Yes. We don't want to supply templates.
We only have a few as you can see.
What we want to do is use the cloud images
that distros are already publishing.
All we're going to do is add metadata.
The distros are going to host those.
There's a Ceph talk coming up next and we can read images directly from Ceph.
http://libguestfs.org/