Into the Pool

This step is the perfect opportunity to test the handling of multiple block devices, also known as pooling. Additionally, it's important to protect not only the metadata, but also the user data, via RAID 1 against faults on individual block devices and accidental data corruption.

To add an extra device to the filesystem with LVM tools, you would first need to define a new block device or at least a partition for use as a physical volume. This physical volume would then need to be added to a volume group. Then, a filesystem could be enlarged on the logical volume or created and mounted on a new logical volume.

Single Step

The btrfs tool handles the whole job in one step using the device add parameter. This command adds an unused hard disk to the root filesystem:

btrfs device add /dev/vde /

The block devices do not all have to be the same size. Depending on the RAID level, which can be different for the data and metadata, the potential overall capacity can vary for different hard disk sizes. As shown in Figure 10, no space is as yet assigned on the hard disk just added. You could use the enlarged filesystem now, but a rebalance is recommended to distribute the data and metadata evenly over all block devices.

Figure 10: Adding more block devices does not redistribute the existing data; however, the space on the extra hard disk is available immediately.

For the sake of completeness, I should mention that Btrfs also can be installed on a LVM logical volume or a Linux software RAID. Especially when dealing with logical volumes, the convenience of growing and shrinking on the fly is something you do not want to do without. Using btrfs filesystem resize helps do this.

As of kernel 3.3, performing a rebalance can also change the RAID level of the data, metadata, or both on an already existing volume. To test these new features, as well as numerous improvements in speed and error handling, the other tests in this article use a 3.4 kernel from the Ubuntu Mainline Kernel archive [7].

Uniform distribution of the data and metadata in this case means duplication to achieve redundancy via RAID 1. For this purpose, all of the data must be moved. As of kernel 3.3, you can specify whether only certain parts of a filesystem are balanced or whether its RAID configuration is changed. A complete, and often time-consuming, full balance is no longer always necessary.

A Question of Balance

The following command converts a non-redundant root filesystem to a redundant one:

btrfs fi balance start -dconvert=raid1 -mconvert=raid1 /

The -d switch here determines the modification type of the data, whereas -m applies the action to the metadata.

As you can see in Figure 11, the data and metadata are now evenly distributed over both block devices, but you can also see that, despite the frugal use of metadata, 9GB was assigned for this. A specific balance of the metadata should provide a solution here. The following command only optimizes the metadata chunks:

btrfs fi balance start -m /

Figure 11: Balancing, including conversion to RAID 1, is handled by a single command. Afterward, you can see the more uniform distribution and an improvement in the footprint.

Unfortunately, issuing this command on the test system made the system unresponsive and logged messages about hanging Btrfs kernel tasks on the console.

After a few hours of waiting and a reset of the system, everything started up again, and the balance process was obviously completed successfully. If a system crash occurs during balancing, the process is automatically restarted in the background after the system boots. As already mentioned, you can display the current progress and pause or completely cancel the balance process.

Figure 12 shows only 2GB is assigned per block device for metadata, and 7GB of space per hard disk has been released. For a RAID 1 with two replicas, these values must be taken times two because all data and metadata are mirrored. A simple df -h / therefore shows that twice as much disk space is occupied. This value also includes the redundant metadata.

Figure 12: Another rebalance, this time only of the metadata, recovers 7GB of 9GB for general use – per block device.

To compensate for this, the sum of all block devices on the volume is displayed as filesystem size. How the value of remaining free space was computed is unclear; in fact, I was unable to fathom it despite intense research. Some simple math shows that about 8GB is missing:

root@maibutter:~# df -h /
Filesystem Size Used Avail Use% Mounted on /dev/m/maibutter-root 60G 4.8G 47G 10% /

The number of replicas of a file on RAID 1 and of a stripe on RAID 10 is always two, regardless of the number of disks in the pool. The option of creating more than two copies will be available in the future. RAID 5 and RAID 6 will probably not be available before kernel 3.6.

When using multiple block devices for a Btrfs filesystem, note that simply loading the Btrfs kernel modules for the kernel is not sufficient to detect all the devices belonging to a filesystem. To do this, you need to scan all the block devices (you can also specify certain devices) using btrfs device scan. For this step to work for a root filesystem, the command must be run in the startup environment (i.e., initrd on your system). Ubuntu 12.04, which I used here, has already integrated this. It is then also irrelevant which device the mount specifies to mount the filesystem.