HOWTO migrate LXC containers using old good LVM as backing storage to ZFS storage back-end and safe tons od disk space in process. This guide assumes that you already configured LXC to use ZFS for backend storage and created ZFS pool and dataset for it.It is probably not optimal, i guess it would be simpler just create new ZFS dataset manually and feed it to LXC, but I do not know all inner working of LXC, so I pick a safer path.

  1. Stop container:
    # lxc-stop -n test
  2. Mount container storage to /mnt. You can get LVM path from lxc.rootfs.path variable in config file for the container /var/lib/lxc/test/config
    # mount /dev/mapper/lcxhost--data--vg-test /mnt 
       root@lcx ~# ls /mnt
   bin/  boot/  dev/  etc/  home/  lib/  lib64/  lost+found/  media/  mnt/  opt/  proc/  root/  run/  sbin/  selinux/  srv/  sys/  tmp/  usr/  var/
  3. Create new LXC container to migrate data
    # lxc-create -B zfs -n test-new -t debian 
     root@metaview ~# zfs list
 NAME                     USED  AVAIL     REFER  MOUNTPOINT
 data                     207M   724G      192K  /data
 data/lxc                 205M   724G      192K  /data/lxc
 data/lxc/test-new   204M   724G      204M  /var/lib/lxc/test-new/rootfs
  4. Mount new root dataset. It will be mounted to /var/lib/lxc/test-new/rootfs
    # zfs mount data/lxc/test-new 
     root@metaview ~# ls  /var/lib/lxc/switchmap-new/rootfs/
 bin@  boot/  dev/  etc/  home/  lib@  lib32@  lib64@  libx32@  media/  mnt/  opt/  proc/  root/  run/  sbin@  selinux/  srv/  sys/  tmp/  usr/  var/
  5. Remove everything on mew root dataset. We do not need it 
     # rm -r  /var/lib/lxc/switchmap-new/rootfs/
  rm: cannot remove '/var/lib/lxc/switchmap-new/rootfs/': Device or resource busy
 # root@metaview ~# ls -a /var/lib/lxc/switchmap-new/rootfs/
 ./  ../
  6. Copy data from old container to new dataset
    # cp -av /mnt/* /var/lib/lxc/test-new/rootfs 
       root@metaview ~# ls  /var/lib/lxc/switchmap-new/rootfs/
   bin/  boot/  dev/  etc/  home/  lib/  lib64/  lost+found/  media/  mnt/  opt/  proc/  root/  run/  sbin/  selinux/  srv/  sys/  tmp/  usr/  var/
  7. Bit of cleanup # rmdir /var/lib/lxc/test-new/rootfs/lost+found/
  8. Unmount everything 
    # umount  /var/lib/lxc/test-new/rootfs/
# umount /tmp
  9. Check diff between old and new LXC containers and add missing parts to new one. At this stage you also need to sort out IP allocation but it depends on method you are using, so I’m not going to cover it. 
     # diff /var/lib/lxc/test/config  /var/lib/lxc/test-new/config

 2,3c2
 < # Parameters passed to the template: --release=stretch
 < # Template script checksum (SHA-1): 2ad4d9cfe8988ae453172bd4fe3b06cf91756168
 ---
 > # Parameters passed to the template:
 10a10
 > lxc.net.0.hwaddr = 00:16:3e:7f:1f:f0
 13,15d12
 < lxc.net.0.hwaddr = 00:16:3e:da:7c:69
 < lxc.rootfs.path = /dev/mapper/lcxhost--data--vg-test
 <
 17a15,16
 > # lxc.bdev.zfs.root=data/lxc
 > lxc.rootfs.path = zfs:data/lxc/test-new
 24c23
 < lxc.uts.name = test
 ---
 > lxc.uts.name = test-new
 26,30c25
 <
 <
 < lxc.start.auto = 1
 < lxc.start.delay = 15
 < lxc.start.order = 3000
 ---
 > lxc.pty.max = 1024

    As you can see in this particular case I need to add lxc.start.* stuff to config of new container.

  10. Let’s toss containers around using lxc-copy
    1. Backup old one
      # lxc-copy --name=test --newname=test-old
    2. Destroy originals
      # lxc-destroy test
    3. Copy new to give it old name.
      # lxc-copy --name=test-new --newname=test
    4. Destroy copy of new
      # lxc-destroy test-new
  11. Start container with new backend storage.
    # lxc-start test

Updated: