Copied on 03 Jun 2022 from here

FreeBSD on ZFS on Raspberry Pi

I’ve done a bit of work the last few days getting FreeBSD set up “properly” with a ZFS root on my RPi4. (The recommended way to install FreeBSD on that platform is to dd an image to an SD card or USB stick, and the image contains a UFS filesystem.) I wrote a few scripts for myself, which I figured might be useful to share with other folks.

My work is based on this great tutorial but contains a few fixes – the big ones being setting up zfs datasets in the same way that zfsboot in bsdinstall does it, as well as adding a loader config option which the system seems to not boot without nowadays (kern.geom.label.disk_ident.enable=0, also from zfsboot). With 13.1 you also don’t need to update u-boot – at least as of this writing; they can always release new board revisions in the future…

The process is:

  • optional: if pi is brand new, use raspbian or whatever to make sure your eeprom is updated

  • download FreeBSD-13.1-RELEASE-arm64-aarch64-RPI.img from a mirror (e.g., https://download.freebsd.org/ftp/releases/arm64/aarch64/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-arm64-aarch64-RPI.img.xz)

  • optional: dd that image onto your card and make sure it boots

  • use dd.sh to write that image to an SD card and grow the partitions

  • mdconfig and mount the 13.1 image above onto /mnt

  • use zfs.sh to blow away the UFS filesystem, create a new ZFS filesystem, and copy the data back over from the mounted image

Some notes:

  • the scripts require a FreeBSD host; I did it in a VM, passing my SD card reader USB directly through, worked fine

  • the scripts assume your sd card is at /dev/da1 and will destroy anything in that device without asking – if that’s not your card, you need to change everywhere da1 appears!!

  • the original directions say gpart modify -i 1 -t freebsd-zfs da1s2 but I didn’t add that to my scripts – not sure if I did it by hand and forgot or if it isn’t necessary :\

  • the image has /home as an actual directory, you may want to move the contents to /usr/home and symlink /home there, since /usr/home is its own dataset

And now… the scripts! They also live at https://gist.github.com/jwatzman/8a53dcdd3084d595c6d5918f4a2a0527.

dd.sh

#!/bin/sh

set -e
set -x

dd if=FreeBSD-13.1-RELEASE-arm64-aarch64-RPI.img of=/dev/da1 bs=4M iflag=direct oflag=direct status=progress
gpart resize -i 2 da1
gpart resize -i 1 da1s2

zfs.sh

#!/bin/sh

if [ ! -f /mnt/COPYRIGHT ]
then
	echo "Forgot to mount disk image"
	exit 1
fi

set -e
set -x

zpool destroy zpi || true
zpool create -O compress=lz4 -O atime=off -o altroot=/zpialt -m none zpi da1s2

zfs create -o mountpoint=none zpi/ROOT
zfs create -o mountpoint=/ zpi/ROOT/default
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zpi/tmp
zfs create -o mountpoint=/usr -o canmount=off zpi/usr
zfs create zpi/usr/home
zfs create -o setuid=off zpi/usr/ports
zfs create zpi/usr/src
zfs create -o mountpoint=/var -o canmount=off zpi/var
zfs create -o exec=off -o setuid=off zpi/var/audit
zfs create -o exec=off -o setuid=off zpi/var/crash
zfs create -o exec=off -o setuid=off zpi/var/log
zfs create -o atime=on zpi/var/mail
zfs create -o setuid=off zpi/var/tmp

zfs set mountpoint=/zpi zpi

chmod 1777 /zpialt/tmp
chmod 1777 /zpialt/var/tmp

zpool set bootfs=zpi/ROOT/default zpi

zfs set canmount=noauto zpi/ROOT/default

cd /mnt
tar cf - . | ( cd /zpialt && tar xvf - )

echo 'zfs_load="YES"' >> /zpialt/boot/loader.conf
echo 'kern.geom.label.disk_ident.enable=0' >> /zpialt/boot/loader.conf
#echo 'kern.geom.label.gptid.enable=0' >> /zpialt/boot/loader.conf
sed -i '' '/boot_serial/ s/^/#/' /zpialt/boot/loader.conf
sed -i '' '/beastie_disable/ s/^/#/' /zpialt/boot/loader.conf

echo 'zfs_enable="YES"' >> /zpialt/etc/rc.conf

sed -i '' /ufs/d /zpialt/etc/fstab

zpool export zpi

Hope this is useful to someone…!

Updated: