Docker under Ubuntu 16.04, set up for direct-lvm storage

Recently i wanted to install docker on my ubuntu server the right way and use direct-lvm as storage layer for the devicemapper. For production environments this is the preferred method. Unfortunately it was not working out of the box, which is why i am writing this and maybe prevent some users from getting as much headaches as i was getting.

Preparing LVM

First step is to make some space in our volume group or even make a dedicated one for docker. If you want to do so, you have to create a new partition. You may have many partitions inside one volume group but you cannot have multiple volume groups on one partition.
I am not a fan of partitions, so i choose to use my existing volume group and reduce the the size of my logical volume in order to have some space for the docker's storage.

The command vgdisplay shows my existing volume groups and its properties. Interesting for us is the VG Size, which shows the overall volume group size, and the Alloc PE / Size, which shows the allocated size by existing logical volumes.

# vgdisplay
  --- Volume group ---
  VG Name               ubuntu-vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               36,80 GiB
  PE Size               4,00 MiB
  Total PE              9422
  Alloc PE / Size       9416 / 36,78 GiB
  Free  PE / Size       6 / 24,00 MiB
  VG UUID               e5LZI7-Uo19-9LBg-3Mcj-YgXm-traf-bGdgS8

In this example of my virtual host nearly all allocatable size is allocated already, so next step is to reduce the logical volume's size so there is enough space to build the docker's storage withing our existing volume group.
We cannot resize mounted logical volumes and since our root filesystem is on the only logical volume we have to boot into a rescue environment and perform our changes from there.

After booting into the rescue system i can reduce my local volume as follows:

#$ lvreduce --resizefs -L 20G /dev/ubuntu-vg/root
fsck von util-linux 2.27.1
/dev/mapper/ubuntu--vg-root: 226401/2281104 Dateien (0.1% nicht zusammenhängend), 1417926/9118720 Blöcke
resize2fs 1.42.13 (17-May-2015)
Die Größe des Dateisystems auf /dev/mapper/ubuntu--vg-root wird auf 5242880 (4k) Blöcke geändert.
Das Dateisystem auf /dev/mapper/ubuntu--vg-root is nun 5242880 (4k) Blöcke lang.

  Size of logical volume ubuntu-vg/root changed from 34,79 GiB (8905 extents) to 20,00 GiB (5120 extents).
  Logical volume root successfully resized.

I decided to reduce my root filesystem to 20G, which leaves me about 15G of free space for my docker storage. The --resizefs automatically resizes the underlying filesystem to its new size, so we don't have to do this manually. To figure out, which logical volume you have to resize, use the lvdisplay command. It shows all available logical volumes within a volume group:

#$ lvdisplay ubuntu-vg
  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/root
  LV Name                root
  VG Name                ubuntu-vg
  LV UUID                4KUUhN-PHJ2-5Q26-82Du-os3m-HusW-uKE2eI
  LV Write Access        read/write
  LV Creation host, time ubuntu, 2016-06-30 08:18:44 +0000
  LV Status              available
  # open                 0
  LV Size                20,00 GiB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0
   
  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/swap_1
  LV Name                swap_1
  VG Name                ubuntu-vg
  LV UUID                s4o2Jm-F0Sz-SysU-x8f7-eJaX-RRoJ-7piRMB
  LV Write Access        read/write
  LV Creation host, time ubuntu, 2016-06-30 08:18:44 +0000
  LV Status              available
  # open                 0
  LV Size                2,00 GiB
  Current LE             511
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:1

With this we have successfully prepared our volume group ubuntu-vg to hold our new docker storage.

Install docker

Next we have to install docker. Since docker is supported very well by most distributions we have a recent version in the ubuntu repository. So all we have to do is

#$ sudo apt-get install docker.io
...

After successful installation we can see the docker daemon is up and running:

sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Di 2016-07-12 15:25:26 CEST; 21s ago
     Docs: https://docs.docker.com
 Main PID: 4993 (docker)
   CGroup: /system.slice/docker.service
           └─4993 /usr/bin/docker daemon -H fd://

Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.400038358+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.415527884+02:00" level=info msg="Firewalld running: false"
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.457709818+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.519826610+02:00" level=warning msg="Your kernel does not support swap memory limit."
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560283918+02:00" level=info msg="Loading containers: start."
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560353891+02:00" level=info msg="Loading containers: done."
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560365431+02:00" level=info msg="Daemon has completed initialization"
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560382383+02:00" level=info msg="Docker daemon" commit=20f81dd execdriver=native-0.2 graphdriver=aufs version=1.10.3
Jul 12 15:25:26 docker-lvm systemd[1]: Started Docker Application Container Engine.
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.587885630+02:00" level=info msg="API listen on /var/run/docker.sock"

For convenience i put my user into the docker group so i don't have to sudo all the docker commands:

#$ sudo usermod -aG docker <username>

after log out - log in my user is able to perform docker commands since it has access to docker's communication socket file:

#$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.0-28-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: docker-lvm
ID: HQAE:HBS4:6RIR:7T4F:PTRU:3JBZ:ZVDM:T4LH:H24F:XUVW:VL22:SHKT
WARNING: No swap limit support

What we can see now is that docker is using aufs as storage driver, which seems to be fine in development environments but also lacks performance, which is why we want to use direct-lvm. To do so we follow the instructions from the docker's website.

configuring logical volume for direct-lvm

Since we already have a volume group we don't need to create one. therefore we skip steps 1,2, 3 and 4 and resume creating a logical volume named thinpool. Since we already have a logical volume in our volume group we obviously cannot use 95% of the volume group's size for our thinpool. Instead for the thinpool itself we take 95% of whats left and for the thinpool metadata 25% of whats left of whats left (:

#$ sudo lvcreate --wipesignatures y -n thinpool ubuntu-vg -l 95%FREE
  Logical volume "thinpool" created.
#$ sudo lvcreate --wipesignatures y -n thinpoolmeta ubuntu-vg -l 25%FREE
  Logical volume "thinpoolmeta" created.

Next we convert our pool into a thinpool:

#$ sudo lvconvert -y --zero n -c 512K --thinpool ubuntu-vg/thinpool --poolmetadata ubuntu-vg/thinpoolmeta
  WARNING: Converting logical volume ubuntu-vg/thinpool and ubuntu-vg/thinpoolmeta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted ubuntu-vg/thinpool to thin pool.

Next we configure autoextension for the thinpool:

#$ sudo mkdir -p /etc/lvm/profile
#$ sudo tee /etc/lvm/profile/docker-thinpool.profile <<EOF
activation {
    thin_pool_autoextend_threshold=80
    thin_pool_autoextend_percent=20
}
EOF

afterwards we enable the profile:

#$ sudo lvchange --metadataprofile docker-thinpool ubuntu-vg/thinpool
  Logical volume "thinpool" changed.

To verify that the thinpool is monitored, do:

    $# sudo lvs -o+seg_monitor
  LV       VG        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Monitor  
  root     ubuntu-vg -wi-ao---- 20,00g                                                              
  swap_1   ubuntu-vg -wi-ao----  2,00g                                                              
  thinpool ubuntu-vg twi-a-t--- 14,07g             0,00   0,02                             monitored

All we have to do now is to tell the docker-daemon to use this storage for its images and containers. Since Ubuntu 16.04 is now using systemd the /etc/default/docker file is not applying anymore. The easiest way to configure the new storage is to create a file named /etc/docker/daemon.json:

#$ sudo tee /etc/docker/daemon.json <<EOF
{
     "storage-driver": "devicemapper",
     "storage-opts": [
         "dm.thinpooldev=/dev/mapper/ubuntu--vg-thinpool",
         "dm.use_deferred_removal=true"
     ]
 }
EOF

Now we stop the docker-daemon, remove the folder /var/lib/docker and start the daemon again:

$# sudo systemctl stop docker
$# sudo rm -rf /var/lib/docker
$# sudo systemctl start docker

Congratulations! Now we have an up and running docker-daemon with a direct-lvm storage layer.

The headache

After all this i was very upset, that after reboot the docker-daemon was not coming up.

#$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Di 2016-07-12 16:22:33 CEST; 1min 30s ago
     Docs: https://docs.docker.com
  Process: 1880 ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
 Main PID: 1880 (code=exited, status=1/FAILURE)

Jul 12 16:22:33 docker-lvm systemd[1]: Starting Docker Application Container Engine...
Jul 12 16:22:33 docker-lvm docker[1880]: time="2016-07-12T16:22:33.304838820+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devicemapper: Non existing device ubuntu--vg-thinpool"
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 16:22:33 docker-lvm systemd[1]: Failed to start Docker Application Container Engine.
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Unit entered failed state.
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Start request repeated too quickly.
Jul 12 16:22:33 docker-lvm systemd[1]: Failed to start Docker Application Container Engine.

The status messages indicated that my thinpool was not available. lvdisplay said the same:

#$ sudo lvdisplay ubuntu-vg/thinpool
  --- Logical volume ---
  LV Name                thinpool
  VG Name                ubuntu-vg
  LV UUID                Or2cLx-ENFq-sZqh-kxUw-0FEQ-rEzi-vXCi3C
  LV Write Access        read/write
  LV Creation host, time docker-lvm, 2016-07-12 15:57:06 +0200
  LV Pool metadata       thinpool_tmeta
  LV Pool data           thinpool_tdata
  LV Status              NOT available
  LV Size                14,07 GiB
  Current LE             3601
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

LV Status NOT available

So i tried to activate it:

#$ sudo lvchange -ay ubuntu-vg/thinpool
  /usr/sbin/thin_check: execvp failed: Datei oder Verzeichnis nicht gefunden
  Check of pool ubuntu-vg/thinpool failed (status:2). Manual repair required!

After some research i found that the tools for check and repair are configured in /etc/lvm/lvm.conf and there i found the following paragraph:

        # Configuration option global/thin_check_executable.
        # The full path to the thin_check command.
        # LVM uses this command to check that a thin metadata device is in a
        # usable state. When a thin pool is activated and after it is
        # deactivated, this command is run. Activation will only proceed if
        # the command has an exit status of 0. Set to "" to skip this check.
        # (Not recommended.) Also see thin_check_options.
        # (See package device-mapper-persistent-data or thin-provisioning-tools)
        # This configuration option has an automatic default value.
        # thin_check_executable = "/usr/sbin/thin_check"

So i looked for the packages device-mapper-persistent-data and thin-provisioning-tools and i found that there is a package thin-provisioning-tools, which is not installed:

#$ sudo apt-cache policy thin-provisioning-tools
thin-provisioning-tools:
  Installiert:           (keine)
  Installationskandidat: 0.5.6-1ubuntu1
  Versionstabelle:
     0.5.6-1ubuntu1 500
        500 http://de.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

After installing it i could finally activate my docker storage again:

#$ sudo lvchange -ay ubuntu-vg/thinpool

And also after reboot the docker's logical volume was activated and the docker-daemon was started successfully.

Docker under Ubuntu 16.04, set up for direct-lvm storage
Share this