Exploding Sparse Files - Your Lesson For the Day

I ran a backup today - something I haven't done in a while. And I noticed that it was taking a really, really long time on one particular file: /var/lib/docker/devicemapper/devicemapper/data. The file is 100GiB - which is really bizarre, because the partition it's on is only 12GiB. How is that even possible?

It turns out there are several things going on here that I didn't understand. This is what I saw:

# ls -lh /var/lib/docker/devicemapper/devicemapper/
total 34M
-rw------- 1 root root 100G Oct 25 22:02 data
-rw------- 1 root root 2.0G Nov 20 12:33 metadata
# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/sda8                16G   14G  1.7G  89% /
/dev/mapper/home_crypt  237G  211G   24G  91% /home

/var/ isn't a separate partition: it's a part of the root filesystem, which is only 12GiB. But when I used rsync to back up /var/ to an external hard drive, it copied 100GiB onto that external drive anyway. After I'd done freaking out - I even MD5summed the files and they matched - I asked my favourite group of Linux experts what was going on. And they came through quickly. That's what's known as a "sparse file," which essentially means that it appears to be 100GiB, but is actually much smaller because its size is a bunch of zeroes reported as metadata. I'm not explaining that well: try Wikipedia on the subject. That helped me. Somewhat.

If you want to see how much space the file is really taking up, use the -s switch to ls:

# ls -lhs /var/lib/docker/devicemapper/devicemapper/
total 34M
 33M -rw------- 1 root root 100G Oct 25 22:02 data
696K -rw------- 1 root root 2.0G Nov 20 12:33 metadata

Look to the left: both files are taking up a LOT less space than ls reports without this new switch. So ... I've been using Linux and ls for 23 years, and I'm just now finding out that it's not really reporting the amount of space used on disk? Wow. Which has all kinds of unfortunate implications (aside from apparently turning your partition into a Tardis, ie. it's bigger on the inside than the outside ...). This seriously messes with any kind of size calculations - and obviously mucks with backups because most of them will copy it as a 100GiB file. Happily, as someone was helpful enough to point out on TLUG Talk, rsync has a --sparse switch to "handle sparse files efficiently." I was lucky: the media I was backing up to was big enough to handle an extra 100GiB (and also lucky that I noticed).

How to figure out you have a sparse file?

# stat /var/lib/docker/devicemapper/devicemapper/data
  File: '/var/lib/docker/devicemapper/devicemapper/data'
  Size: 107374182400    Blocks: 66664      IO Block: 4096   regular file
Device: 808h/2056d      Inode: 802522      Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-11-20 10:53:56.741662721 -0500
Modify: 2017-10-25 22:02:26.256396194 -0400
Change: 2017-10-25 22:02:26.256396194 -0400
 Birth: -

stat reports both the size and the number of blocks: technically, all the information you need is there. But since block size varies between hard drives and isn't a power of 10, I find this not at all obvious.

# file /var/lib/docker/devicemapper/devicemapper/data
/var/lib/docker/devicemapper/devicemapper/data: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

file isn't about size, it's just about the type of file. (And that's kind of interesting in itself, but that's more of a Docker thing for another blog post.) So ls -slh seems to be about the only practical answer. Also, keep an eye on the "total" in the output of ls: if it doesn't reflect the size of the files, you may be looking at sparse files. And may I suggest adding --sparse to any rsync scripts you have?

Another question for which I don't have a full answer yet: where are you most likely to encounter sparse files? Because I'd like to know to keep my eyes open. Apparently you don't see them often - this is the first time they've bit me in 23 years. The answer (although Wikipedia doesn't go into a lot of detail) appears to be "around anything that makes file systems." But even this isn't entirely accurate: I've used VirtualBox a lot and have something like 30 images on my hard drive, many of which present themselves inside VirtualBox as being 8GiB. But I've looked at most of the ones I know are that size, and it appears VB isn't using sparse files.

UPDATE: This only got weirder when I ran my next backup. The backup for the Docker file ran faster, but much slower than it should have for a 33MiB file. And when I examined the new backup file on disk, the actual disk space used was 21MiB. I was concerned and ran md5sum against both files - but they matched. Sparse files confuse the crap out of me.