04 December 2013

On quickly creating sparse and large files

I've been using dd(1) for as long as I can remember to create sparse files but for some reason I keep forgetting how to use dd's argument options so I set out to find something better... and found it: truncate(1):

From the man page: "Shrink or extend the size of each FILE to the specified size. A FILE argument that does not exist is created." -looking good:

root@debian64:~# time truncate -s 10T testfile

real0m0.003s
user0m0.000s
sys 0m0.004s

root@debian64:~# ls -l testfile
-rw-r--r-- 1 root root 10995116277760 Dec4 10:27 testfile

root@debian64:~# du -hs testfile
0 testfile

Awesome!
So then I set out to find me something to quickly allocate 'real' files:

fallocate(1):
"fallocate  is used to preallocate blocks to a file.  For filesystems which support the fallocate system call, this is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks.  This is much faster than creating a file by filling it with zeros."

And it truly is fast:


root@debian64:~# time fallocate -l 1G testfile

real    0m0.004s
user    0m0.000s
sys     0m0.000s

root@debian64:~# du -hs testfile
1.1G    testfile

root@debian64:~# ls -l testfile
-rw-r--r-- 1 root root 1073741824 Dec  4 10:34 testfile