Welcome to my talk about loop mounting and going beyond ordinary loop mounting.
Hopefully most of you here know what loop mounting is
but in case you don't I'm just going to give you a quick demo
of what I'm going to call traditional, old fashioned loop mounting.
I've got a file on my laptop here, a Fedora disk image.
It's 6 GB in size.
I use the command "losetup" to associate files with Linux kernel block devices.
I don't have any loops set up at the moment.
But I can associate this file with a Linux kernel block device called /dev/loop0.
And I can look at partitions.
And I can mount them -- that's loop mounting.
That's just ordinary loop mounting, it's not what we're going to be talking about today although conceptually it's similar.
Traditional loop mounting is fine if it's a plain file.
It falls over very quickly.
What happens if you've got a compressed disk image?
This isn't a disk image that I've compressed, this is how cloud images are distributed by the Fedora community.
And this is one that I just downloaded.
From the Fedora website in exactly this format. I didn't modify it in any way.
It's XZ-compressed.
And of course you could turn that into a loop device.
And what would happen if you'd end up with a device which contained XZ-compressed data.
And it is my contention today that this is not what you meant by loop mounting this.
What you'd want it to do is you'd like it to transparently uncompress so you can see the data inside.
And we can do that.
Using a command called "nbdkit" which I'm going to talk about in a minute.
And that creates a process.
And now I use a command to associate that process with a loop device.
It's a different command, "nbd-client", but it's conceptually similar.
And you see there the size -- 4096 MB, 4 GB.
So that's not the compressed size, that's the uncompressed size.
What this is doing is it's transparently uncompressing. It's not uncompressing the whole thing.
It's only uncompressing the blocks that are being used when you request them.
And I can mount that.
And I can look at it, and I can even create files.
So that is what I'm going to be talking about today.
How does this differ from loop mounting?
In both cases we've got a kernel module, on the left hand side "loop.ko".
And it's configured using a command line utility, "losetup".
And you use that to create Linux kernel block devices.
Like /dev/loop0.
On the right hand side I've got a kernel module, "nbd.ko".
It's configured using a command line client "nbd-client".
And it creates loop devices, Linux kernel block devices, "/dev/nbd0" etc.
But the back end as you can see is a little bit different.
On the left hand side the back end is talking over internal Linux kernel APIs like the VFS.
To the file which is associated with the loop device.
On the right hand side we've got a user process running.
This is critical: We've got a user process in this case "nbdkit", but I should say that other NBD servers are available.
Other very very good NBD servers are available.
The kernel is talking to that process using
a TCP port or a Unix domain socket as you require.
I'm going to demonstrate in this talk nbdkit which is an NBD server which I wrote
with a guy called Eric Blake
who's a brilliant free software hacker.
Nbdkit is slightly different from other NBD servers
in that we have a plugin API. It's a stable API.
Which means you can write a plugin now
or you could have written a plugin back in 2013 when we started the project
and it would still be compilable with nbdkit now
and it will still be compilable in the future.
We're not going to break plugins at the source level.
It also has an ABI guarantee.
That means you can compile your plugin
and you can distribute it separately from nbdkit
as a binary
and load it into nbdkit at some point later. We're not going to break that even as we evolve nbdkit
and we evolve the API we don't break source or binary compatibility.
If you don't want to write a plugin -- and I'm going to show you in a minute how you can write a plugin simply
if you don't want to write a plugin, many other plugins are available.
I've listed the ones which are in nbdkit 1.10 on here.
Some of these plugins aren't quite like the others.
These are plugins like Perl, Python and they are gateways to writing plugins
in non-C languages.
So you can write plugins in scripting languages,
even in shell script, if you're not very happy with writing C plugins.
The other concept that nbdkit has is "filters".
You can think of a plugin as like a data source.
It's like a source of disk images.
But filters apply modifications or changes to
that data source.
An example here is the partition plugin. If your source is a whole disk image, a partitioned disk image,
but you only want to serve one of the partitions over NBD,
you can apply the partition filter,
which selects a partition.
Each running nbdkit instance must have exactly one plugin running in it.
But it can have zero or any number of filters.
In this case I've selected the "file" plugin.
So my source is a local file.
But as it's a compressed file I'm going to apply the "xz" filter on top
to transparently uncompress it.
And then I'm going to apply the "partition" filter
to select a partition.
And then I'm going to apply the "cow" filter
because I want to make a writable overlay which I can write out to a qcow2 file later.
This is how you would express all that on the nbdkit command line.
So you put "nbdkit", the name of the program.
The list of the filters. You might think of these filters as being in reverse order.
They're in reverse order from the distance they are from the plugin.
Or another way to think about it is: When an NBD request comes into the server,
it travels through the filters in this order.
At the bottom I've got the name of the plugin.
And then any parameters that the plugin needs. Obviously the file plugin needs to know which file you want to serve.
So I give it the disk name ... the file name.
And filters may also require parameters as well.
In this case the partition filter wants to know which partition you want to serve,
so you have to give that as a parameter.
Now ... I wanted to demonstrate actually writing a plugin,
and I want to do it very quickly so I don't bore you,
and I was trying to think what could I do to demonstrate
to demonstrate writing a plugin?
I thought I'd write a test device,
so I'm going to write a Linux kernel device,
to test the "badblocks" command.
It's quite a young audience here,
and we haven't used the badblocks command for a really long time,
perhaps since we've had IDE disks in the early 90s.
But before then old grey-haired people will remember RLL and MFM disks?
Everyone's looking a bit confused. Floppy disks?
In those systems, when there was an error on the surface of the disk
that would appear at the filesystem layer.
So you had to run the badblocks command first,
to find these bad sectors,
and it would produce a list of blocks which were bad,
and it would pass that over to mkfs, and mkfs could work around this by, ...
So that's the badblocks command.
And this is the device I'm going to write to test that.
It's going to be a big virtual device.
It's going to have a bad sector somewhere in it.
And the idea is whenever the kernel requests,
or reads from,
that bad sector, so whenever my request contains the bad sector,
it's going to return an error.
But any other place in the disk that it tries to read,
it's just going to return some data.
So a nice and simple demo, let's write that now.
What's a good language for writing Linux kernel block devices in?
bash
yup, bash
The first thing nbdkit is going to do is it's going to send me a request for the size of the disk.
So I'm just going to return any size -- it doesn't matter -- 64 MB is fine.
And then nbdkit will send me a request any time there's a read.
The request is called "pread".
And the parameters for that: $1 is the literal string "pread".
$2 is a handle, which we're not using here.
$3 is the size in bytes.
And $4 is the offset in bytes of the request.
Error case. The error case is if my request contains the bad sector or bad byte.
So I'm going to put the bad byte at 100,000.
So if my offset is less than the bad byte and the offset + size is bigger than the bad byte,
that means that the bad byte is in the request.
Agreed?
Has anyone done pair programming where you have people looking over your shoulder?
Have you done pair programming with 100 people looking over your shoulder?
My offset less than the bad byte at 100,000,
and ...
and the size, sorry the offset + the size,
offset is $4 plus the size [$3] if that's greater than the
bad byte,
(I hope I've got the right number of 000's there)
so this is my error case ...
So I just have to echo the error number that I want.
EIO
and just something that goes into syslog.
And I have to send that to stderr.
And I have to exit with an error code.
So that's the error case.
The other case is where I'm reading somewhere else in the disk.
I have to return a block of size bytes back to nbdkit.
I'm going to return just zeroes, it doesn't matter what I return.
So if I use "dd",
from /dev/zero,
and I want to return exactly bytes $3 size
So if I set the block size [bs] to $3,
and I set the count to 1, that should return that number of zero [bytes], right?
So that's my complete bash script Linux block device.
So to [test] that I'm just going to run it.
So run "nbdkit", name of the plugin which is "sh", and the "sh" plugin needs the name of the script which I've just written.
The moment of truth here.
If I associate that ... so the good thing there is the size,
64 megabytes,
remember we set the size to 64 MB?
So that's good.
And now if I run "badblocks".
That worked.
Now you might say why has that printed out 4 numbers where there was only 1 bad block?
And the reason for this is the badblocks command reads the disk in 4K chunks,
and when it hits a bad 4K chunk,
it wants the output to be in 1K chunks,
it says there must be 4 bad [1K] blocks.
It doesn't go in any deeper,
and try to work out which of the blocks is bad,
it's just the way badblocks works.
So it's good that badblocks, we've proven here, doesn't have any bugs,
even though nobody's used it since 1992.
You don't have to write plugins.
You can use existing plugins.
We've got loads of them, I don't know what to demonstrate, so I'm going to demonstrate 2 at random.
The "floppy" plugin, and the "memory" plugin which is a RAM disk.
The "floppy" plugin first.
Real simple.
"nbdkit", name of the plugin which is "floppy", any directory. This happens to be the directory of the source code of nbdkit.
And same old nbd-client command to associate that with a loop device.
And it should pop up in a second. There it goes.
That popped up.
That is a floppy disk image,
which contains the source of nbdkit.
And what exactly happened there?
I took a filesystem from my host,
I turned it into a floppy disk, a FAT formatted, MBR partitioned disk image,
and then I loop mounted it on my host again.
Why is that useful?
One thing you can genuinely use this for is to export disk images really easily to virtual machines.
Or to containers,
some container systems let you [import] a disk image which gets loop mounted inside the container.
When this is really useful is actually not in the loop mounting case.
That's when you're creating a PXE client, [correction:] PXE server,
and your PXE client needs to be given a root filesystem,
and the traditional way you do that is you create a massive initramfs,
that you TFTP over to the client at boot time.
Which is slow, TFTP is unreliable, it's not encrypted, etc.
NBD is encrypted and authenticated,
it's a super-efficient protocol,
and it's a much better protocol [to use]
because it only fetches the bits it needs to read,
so you can have a much bigger root filesystem. So it's a useful thing to do.
My next demonstration is ...
a RAM disk.
Linux of course has a RAM disk driver inside [the kernel].
It is however much more convenient to be able to
write RAM disks in userspace.
We've written a simple RAM disk called the "memory" plugin,
which is implemented using a sparse array,
so it's not limited by the size of RAM,
(in virtual size)
you can actually create really massive disks.
In this case I'm going to create the most massive disk that you can create.
The biggest that Linux supports,
until we eventually move to 128 bit block sizes,
this is 2^63-1, it's the largest signed 64 bit integer you can have.
How big is that -- 2^63 - 1 -- in terms of disks?
I went on to Amazon to try to work out how much it would cost you to buy that many disks.
And it turns out that's €300 million.
I was very disappointed that Amazon doesn't let you create an order
for €300 million that I could have screenshotted here
the field isn't big enough, it doesn't let you do that,
you can't do that.
But anyway it's €300 million on Amazon,
but we can create it here much more cheaply.
Associate it with a loop device.
You can see the size there is just massive.
I'm going to use GPT for partitioning, because MBR is limited to 2 TB.
All defaults.
That's what the partition size looks like.
8 exabytes [EB]. It's actually 8 EB minus 1 byte.
I'm going to use btrfs, but what are my other choices?
I could have used ext4 ... could I?
What's the limit on filesystem [size] on ext4?
Nobody knows. It's 1 EB.
We'd have 7 EB wasted.
XFS is possible, but XFS has quite a high metadata overhead.
Actually that's unfair on XFS.
XFS has a really nice, low metadata overhead,
but it's about 1%.
1% of 8 EB is too big for my laptop.
So I'm going to use btrfs.
You can see there, btrfs is an absolute champ.
It totally just creates an 8 EB filesystem.
And I can mount it.
And I can ... we've got 8 EB ...
I'm just going to "chown" this so I can go in there and show you.
I played around with this [before].
I missed that question I'm afraid.
(inaudible question)
The question was: How many bugs in anything
do you hit when you try to use the very last block which is only 511 bytes long?
The answer is you definitely hit bugs in qemu.
Qemu can't handle that case.
You can create btrfs subvolume[s].
What is it? "btrfs filesystem df", I think?
And it just works great.
And the next thing is when I click to the next slide ... that's gone.
This software I'm using will kill nbdkit,
everything's destroyed and it goes away.
So it's great for testing.
Other things that are useful for testing. There are some plugins there which are very useful for testing.
And some filters I'm going to talk about now.
Which are super-useful if you're testing filesystems or the limits of filesystems.
The first filter I'm going to talk about which is useful for testing is the "delay" filter.
You can inject delays ...
into the nbdkit request. You can specify the number of seconds,
or the number of milliseconds.
This is useful if you were testing, say, a distributed filesystem.
You want to test it all on one machine,
but you want to simulate the effects of having a really remote node,
that has a long delay,
you just inject delays into that device
to simulate that.
So it's a very simple filter.
This filter's also a lot of fun.
It's the "error" filter, and it injects errors.
Obvious use for testing here.
There's two ways to use this.
The first way is we want this particular error [EIO] and we want a generalized error rate of 10%.
This means that at random 10% of [requests] are going to fail.
However I think the second way of doing this is more useful for
most people.
Here we're saying the error rate is 100% so
100% of requests are going to fail reliably all the time.
However it's gated ...
on the error file. Now what that means is
if that error file doesn't exist,
or you delete it, no errors are injected,
the error filter is turned off.
When you create that file,
the error filter is turned on. This is while nbdkit is running,
so it's checking that error file [on every request].
And that's super-useful for testing because obviously you can inject errors
when you want them to be injected
and then turn off error injection and see if your filesystem recovers or whatever it's supposed to do.
And the third filter which is a very simple filter but also useful is the
"log" filter. You give it the name of a log file
and it writes the log file in that format.
In the next demonstration I'm going to show you
we're going to have some graphical visualization of what happens
inside filesystems when you do things like creating filesystems.
It's important to note that nbdkit is not a graphical tool.
Nbdkit knows nothing about graphics or anything like that.
What's actually happening here is we're using the log filter,
we're writing a log file,
we've got a second graphical program, a program I wrote for this talk,
which is tailing that log file and then
creating the visualizations which you'll see.
So nbdkit is not a graphical program, it's just a command line tool / server.
Let's have a look at what it looks like to create a filesystem.
Slightly long nbdkit command line here,
but hopefully you should be able to understand what's going on.
We're creating ... the "memory" plugin, so we're creating a RAM disk.
64 megabytes
We're using that log [filter] to create the log file which we're going to tail with a second process.
And we're inserting delays. Now the delays are just so it slows it down a little bit,
to make it a little bit easier to see.
Otherwise everything goes past too quickly.
So I'll run nbdkit.
And this is my second program which is going to visualize things.
Same old command to associate the nbdkit instance
with a loop device.
Now hopefully you could see that.
Little black flashes going on.
Those are reads.
What's happening there is because we've created a Linux kernel block device,
the kernel, udev, are looking at that and saying "is there an LVM PV [physical volume] there?"
"Is there a filesystem there? Is there a partition there I should know about?"
It's a RAM disk so it's empty, but it has to check.
Now let's partition it. I'm going to use GPT.
All defaults.
GPT works by creating a partition table at the beginning of the disk and a secondary or backup PT at the end of the disk.
Those are represented in red. Those are writes.
And you probably also saw little black flashes there. We've created another Linux kernel device,
and again udev has to check it.
Let's create a filesystem in there.
The big thing that happens there is this lump of blue that happens at the beginning. [Not shown because of a technical problem in the video]
Blue in this diagram represents discards.
Modern mkfs always issues a big discard or trim over the entire partition.
The reason for that is it makes SSDs work more efficiently.
Other notable features: The red bar here is some kind of metadata.
I'm in a Storage [Track] room full of filesystem experts,
so hopefully you know better than I do what's going on here. But that's probably an inode table.
Big lump of red here. Could be the journal, maybe?
Little red dots. I think those are backup superblocks.
If you notice there are 4 red dots and 4 backup superblocks.
Let's mount that.
I'm not touching the laptop here but something funny happens in a second.
Here it goes.
Do you see it's writing?
We've just mounted the disk but it's writing to it.
This is lazy blockgroup initialization, it's another feature of modern filesystems.
Because disks are really big these days, and writing to them (relative to the size of the disk) is really slow,
so you wouldn't want your mkfs to sit there,
for hours on end writing all the blockgroup metadata.
And in any case why would you do that? Because you can't
use all of those blockgroups for writing because writing is so slow
compared to the size of the disks.
So it makes so much more sense for filesystems to defer all this [work] to the kernel,
so when the disk is mounted the kernel sees there are uninitialized block groups
blockgroup metadata
and it creates that in the background.
It doesn't matter anyway because you can't write to those new blockgroups
faster than they're being created
so it's fine.
So let's mount this.
It's mounted so I'm going to chown it to make it convenient for me to put some files on there.
Let's again copy the nbdkit source code.
You see that nothing actually is written until I "sync".
We know this, right?
When you write to a disk,
the writes don't hit the disk immediately, they get stored in memory for a bit
and they get written a few seconds later,
unless you do a "sync" which forces that write.
And of course when you delete that directory,
even when I "sync",
it's not going to [update] that.
You know why this is. You know when you delete files it doesn't really delete them, it simply marks them in the block group
as being unused,
and later on those blocks get reused for other files you create.
But there is a command -- for modern filesystems -- we can use
to actually tell it to discard them
and that's the "fstrim" command.
That issues a discard request to the filesystem.
This is my final demo.
That was a nice one showing a single filesystem,
but I think more interesting is when you run multiple copies of nbdkit
to create multiple devices.
And this is the longest nbdkit command line that you'll probably ever see.
There are only two important changes here.
The first one is ...
Previously I was only running one copy of nbdkit.
So I could have it listen on TCP port 10809
which is the default port for NBD.
However I'm going to be running 5 copies of nbdkit this time, and they can't all be listening on the same port.
So I'm going to be using a Unix domain socket and that's the purpose of the -U option here.
And the second change is I'm using the error filter.
I'm using this in the way we described before
where you set the error rate to 100%,
but we gate this on the presence or absence of an error file.
So the error filter is turned off because that error file doesn't exist.
But it gets turned on later on.
I'm going to start 5 copies of nbdkit.
I'll just show you what's going on on the filesystem here.
We've got 5 log files as you'd expect.
Those are going to be tailed by the graphical viewer.
We've got 5 sockets. There are 5 copies of nbdkit hiding behind those Unix domain sockets.
Let me run the graphical viewer.
5 devices this time.
Hopefully not too small. OK.
And now I'm going to associate the 5 nbdkits with the 5 devices.
Now I'm going to create a RAID 5 array.
I'm going to use the first 4 disks as data disks,
and the last disk as a hot spare.
Let's get that going.
You can see what's happening here is it's reading the first 3 disks
and creating a parity disk on the 4th disk.
People who know about RAID will be thinking: "Why's that parity disk not being striped
over all of the data disks?"
The reason is because these disks are so small, they're 64 MB,
the stripe size is actually [larger] than the entire disk.
So there's 1 parity disk and there are 3 data disks.
Let's have a look at the kernel messages which will be interesting in a minute.
Let's partition that as before.
All defaults.
And we can create a filesystem on there as well.
This looks a lot like it did last time,
except there's no trim.
Now the MD device in the kernel doesn't believe you can send discard requests to devices.
I guess because they've been burned [by faulty hardware] in the past.
There is a way to do this, by setting a kernel command line flag,
which is something weird like
[raid456.devices_handle_discard_safely=Y]
However as this is quite literally my work laptop,
I don't have that on my kernel command line
so it's not issuing discards to the underlying devices.
And I can mount this.
Let's chown it so I can write to it.
Let's create some files in there.
And the interesting thing is what happens when I inject an error into this?
Well you can see what happened there, quite dramatically.
It detected first of all that the error occured
on the second disk
and the second disk is called "/dev/nbd1" here because I'm starting the disk numbering from "/dev/nbd0".
You can see also it started to do a recovery.
So it started to read from the remaining good disks,
and it created an extra parity disk on the hot spare.
It took a little bit of time. We're injecting delays here.
Although we're injecting delays so it's a bit slower than normal,
you can imagine how it would be if this wasn't a 64 MB disk
but this was 6.4 TB or larger.
Recovery on RAID 5 takes a really long time,
and unfortunately the way that RAID 5 works is that if you then get another disk failing,
at certain points during the recovery,
you can lose all your data.
That's why we don't use RAID 5 in production,
certainly on larger systems these days.
However it's still a good demo.
I should just note that when I clicked the "Error" button there
the graphical tool didn't start injecting errors.
All that happened was the graphical tool created a file called "error2".
And then nbdkit notices that the file exists and starts to inject errors on that disk.
Now although all that dramatic stuff happened in the background,
the actual filesystem is fine.
There are no errors on the filesystem or array level.
The dramatic stuff happened below there.
And of course I can inject more errors on a second disk.
And now we're running in degraded mode.
This is the minimum that this RAID array can support
without actually failing.
Although there was another error
I'm still just about OK, although if there was another error
if I clicked another button, two things would happen:
You'd see errors appearing at the filesystem level.
And the second thing that would happen is I'd have to reboot my laptop.
Because you cannot ... and I could not work out how to do this ...
You cannot then unmount a RAID array
that's in that state. It's just impossible
and I've no idea why. There's lots about this on stackoverflow.
I don't want to reboot my laptop in the middle of the talk
so I'm not going to do that.
It's probably a kernel bug or something, I don't know.
Instead what I'm going to do is umount the filesystem
and stop the RAID array.
[end of subtitles]