HOW TO: FreeBSD remote-bootable crypto setup

Intro

Recently, a friend sponsored a dedicated machine for our homeless ass.

Previously, we only ever had one dedicated machine on the internet and that was probably about a decade ago.

It was completely unencrypted.

This was not going to cut it this time – We wanted security that's up to snuff with our current abilities and needs. One of the key ingredients to that was going to encrypt as much as we could.

The problem being that the ISP the new machine resides with doesn't offer access to the machines serial console, so the only easy options to input any passphrases for encryption would be:

A) Storing the passphrase(s) in plaintext files on an unencrypted drive.

B) Requesting the ISP to connect a KVM switch in order to be able to boot. every.single. time.

Option A would end up making the whole setup pointless unless you also set up something like swatd to delete the passphrases (and keys) if the machine detects any tampering, because anyone getting access to the hardware can just disconnect it from the network, reboot it into a root shell and just mount your encrypted drives, gaining full access.

swatd in turn requires sensors that might or might not be avail- and dependable. All in all extremely

Option B would technically work, but requires some shite java client and makes you depend on ISP admins for every single boot, compounding every single downtime you experience.

We also think it's a bad idea because keyboard input is trivial to eavesdrop on by the ISP admins – even if the java shite uses proper encryption.

On Linux, there's an established pattern for this:

An initramfs with a statically linked dropbear SSH daemon through which you can load up the LUKS + LVM configuration and input any passphrases – Don't ask us exactly how it works, we stopped using Linux before routinely using disk encryption.

Debian even has a dedicated package to do just that: dropbear-initramfs

No such luck on FreeBSD.

We didn't find any How-To's or specialized packages, so…

What is to be done?

Clearly, the FreeBSD community needs to establish a similar pattern to the one that's become commonplace in the Linux world.

There even is a semi-maintained collection of shell scripts to build something similar with a manually compiled and statically linked dropbear: freebsd-remote-crypto

Thing is, it has problems. For one, it's not very generic and assumes a specific setup with two drives which didn't work well with our plans. For another, it doesn't automate the building of dropbear, which we'd have to manually maintain – why even do this when we have a perfectly working SSHD in the base system and only have to worry about base-system updates that way?

Saving a couple megabytes on disk doesn't seem like a good justification for the extra effort needed to keep this setup up to date.

Its code however did serve as good reading, first and foremost because it taught us about reboot -r, which is a critical component of our setup.

So we set out to create a minimal unencrypted FreeBSD install that can be accessed via the built-in SSH daemon and used to boot "through" to a fully encrypted system.

Pulling off a One Wheel Manual

Lucky for us, FreeBSD is simple and doing a manual install is almost trivial. This is going to help us, because we need to set up not one, but two FreeBSDs on our machine, one just for booting and then the encrypted system we actually want to use.

Before we start diving in, let's clear some things up.

  • This guide assumes that you'll initially be booting into a FreeBSD live image, supplied via network or whatever your ISP offers. Usually this is done through a web interface.

  • We simplified this guide as much as we could, so you don't have to burn precious brain cycles by having to figure out which commands are relevant and important – they all are.

  • Nevertheless, we're showing how to do this for UFS and ZFS. Filesystem-specific sections are all clearly marked in their headings as UFS or ZFS.

  • We'll be using the more established geli for encryption, not gbde – if you want that, you'll have to adapt the crypto setup described here yourself – should™ be rather trivial tho.

  • As this guide has been adopted from the much more complex setup on our server, some mistakes might have snuck in – please do tell if you stumble onto any!

So, without further ado…

Let's set up that boot system…

If you want to minimize effort, you can probably skip ahead to Post-install configuration by just installing on a single 5G partition using bsdinstall. We're still telling you how to do this manually, because it will enable you to automate this sort of setup.

Since we'll be working on a mostly read-only filesystem, we'll put the data we need into /tmp, which is a tmpfs living in memory. You'll need a couple hundred MB of free memory as we'll be putting a bit over 200MB of data on it.

Prerequisites

As the image we have at our particular ISP does not contain the data archives needed to set up a base system, we'll also install the common set of CA root certificates, so we can safely fetch those archives over HTTPS.

If you can't install packages on your particular live image, you'll have to resort back to plain-text HTTP and verify that the SHA-256 checksums match the ones in the MANIFEST file.

pkg install ca_root_nss # needed for HTTPS to not fail
cd /tmp
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz
  • If your live image contains kernel.txz and base.txz, you can obviously use those instead of having to fetch the files. In that case, you can skip everything but the cd /tmp.
  • Additionally, it probably goes without saying, but you should adjust these URLs to the right CPU architecture and release you want to install – if you're unsure, browse through the directory listing.

Boot system partitioning

We'll be creating a GPT partition table, create one 512K-sized partition for the bootloader, which will then be installed with gpart bootcode. 512K was chosen to make sure any variation of the bootloader fits while also making sure we don't go over the weird maximum size of 535K or something over which the boot will just fail.

After that, we're creating one 5G-sized partition to hold the minimal FreeBSD we'll use to boot into the encrypted system, set up a filesystem on it and mount it to /mnt.

UFS

gpart create -s gpt ada0

gpart add -s 512K -t freebsd-boot -l boot
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

gpart add -s 5G -t freebsd-ufs -l xboot # will hold the entire first FreeBSD

newfs -U -j gpt/xboot
mount gpt/xboot /mnt

ZFS

gpart create -s gpt ada0

gpart add -s 512k -t freebsd-boot -l boot
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

gpart add -s 5G -t freebsd-zfs -l xboot

zpool create -o altroot=/mnt xboot gpt/xboot
zfs create -o mountpoint=/ xboot/root # create a single dataset, technically not required but considered good form and needed if you want snapshots
zpool set bootfs=xboot/root xboot # we're not 100% sure this is needed, but it sure don't hurt.

Note that the zpool is automatically mounted, with the altroot parameter setting a temporary mountpoint.

Boot system install

Next up, we'll extract the kernel.txz and base.txz, which is all the data needed to create a working base system.

tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz

Don't reboot just yet tho, because we still need to do some…

Post-install configuration

We still have to do some things to make the system actually boot and be reachable via ssh.

First, to make it boot, we'll need to create

loader.conf

This file goes to /mnt/boot/.

UFS
vfs.root.mountfrom="ufs:gpt/xboot"

Actually, we're not 100% sure whether this is needed if you use UFS, feel free to report back if you try booting without it.

ZFS
# load the zfs kernel module
zfs_load="YES"

# tell the bootloader to boot from the xboot/root dataset
vfs.root.mountfrom="zfs:xboot/root"

SSH

Lastly, we'll have to set up the network configuration, make sure the SSH daemon starts up and is accessible for us.

For that we need an…

rc.conf

This file goes to /mnt/etc/.

hostname="alkahest" # your choice

# Disable kernel dumps so the 5G disk space won't get completely
# filled at crash, which would break booting.
dumpdev="NO"

# Network configuration, the values that go here are prescribed by
# your ISP. You can look them up with "ifconfig" and
# "route show default".
ifconfig_re0="inet 176.9.110.174 netmask 255.255.255.224"
defaultrouter="176.9.110.161"

# automatically start the SSH daemon
sshd_enable="YES"
Account creation

Now that we have the first FreeBSD set up, let's add a user so we can actually log in via SSH.

Switch into the newly installed system:

chroot /mnt

and add a user who's part of the wheel group (which allows the user to switch to root via su) by running adduser:

$ adduser
Username: fnord
Full name: 
Uid (Leave empty for default): 
Login group [fnord]: 
Login group is fnord. Invite fnord into other groups? []: wheel
Login class [default]: 
Shell (sh csh tcsh git-shell fish bash rbash nologin) [sh]: 
Home directory [/home/fnord]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: 
Username   : fnord
Password   : <disabled>
Full Name  : 
Uid        : 1001
Class      : 
Groups     : fnord wheel
Home       : /home/fnord
Home Mode  : 
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y

Next, add your SSH public key on the local machine you're doing this from to the created users' ~/.ssh/authorized_keys:

mkdir /home/fnord/.ssh

# in vi: press i, middle-mouse paste your pubkey, press ESC,
# then type :x and press enter to save and quit.
vi /home/fnord/.ssh/authorized_keys

# change ownership and permissions of the involved files
chown -R fnord /home/fnord/.ssh
chmod 700 /home/fnord/.ssh
chmod 644 /home/fnord/.ssh/authorized_keys

Lastly, you might want to set a root password:

passwd

You can skip this step, but it will lead to any user in the wheel group being able to su to root without any password check.

Congratulations, you just did a completely manual FreeBSD installation – Welcome to the club of leet kids.

Finishing up

Now, reboot and hope like hell the system actually boots.

We recommend having a ping on the IP address of the remote machine running before you reboot, so you can see how the machine goes down and (hopefully) becomes reachable again after a minute or two.

If everything went well, the SSHD will be reachable a few seconds after the machine starts responding to ping again.

If you can't ssh into the machine, reboot into your ISPs rescue system and try to figure out where you went wrong – or bug me in the comments, on the fediverse or on IRC. ;P

Encrypted system setup

After doing all this, we finally get to set up the encrypted system we actually want to run and use.

ssh into your created user and su to root.

Partitioning

For this guide, we'll go with simplicity and just create one big ZFS pool with all the space still free on ada0:

UFS

gpart add -t freebsd-ufs -l tank ada0

Yeah yeah, "tank" is ZFS terminology, deal with it.

ZFS

gpart add -t freebsd-zfs -l tank ada0

Crypto setup

Going on, let's create a keyfile so the cryptography will be working with more entropy than just the passphrase:

dd if=/dev/random bs=512 count=1 of=/boot/crypto.key

This gives you 512 byte (or 4096 bit) of entropy.

We honestly don't know if anything past 256 bits of that is actually used, feel free to weigh in if you know!

Next, we'll create the actual crypto device.

# load the kernel module for hardware-accelerated crypto.
# only do this if the CPU in question supports it.
# might be a different module for non-intel CPUs.
kldload aesni

geli load # same as kldload geom_eli

# We're using AES-XTS because it does encryption *and* authentication
# while being supported by AESNI. Otherwise you can forego either
# authentication by using something like AES-CBC or forego full
# hardware acceleration by supplying an authentication algorithm
# with the -a parameter.
geli init -e AES-XTS -l 256 -K /boot/tank.key gpt/tank
geli attach -k /boot/tank.key gpt/tank

If you have some experience with geli, you might have noticed that we're foregoing setting the boot flag when initializing our geli device. This is critically important – if you set the boot flag, not only will the boot setup hang at the passphrase prompt when booting – any FreeBSD live image, like the ISP rescue image, will too! This means that in order to boot anything, you'll have to boot into a live Linux to destroy any partitions containing geli devices and redo your encryption setup.

If it worked, the device /dev/gpt/tank.eli now exists and we can create and mount our filesystem on the encrypted device.

UFS

newfs -U -j gpt/tank.eli
mount gpt/tank.eli /mnt

ZFS

zpool create -o altroot=/mnt tank gpt/tank.eli
zfs create -o mountpoint=/ tank/root

Encrypted setup install

Now, we basically have to rinse and repeat what we did for the boot install:

pkg install ca_root_nss
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz

tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz

cp -p /boot/loader.conf /mnt/boot/

cp -p /etc/rc.conf /mnt/etc/

chroot /mnt # switch into the encrypted install

adduser # add user 'fnord'

mkdir /home/fnord/.ssh

vi /home/fnord/.ssh/authorized_keys # add your public key

chown -R fnord /home/fnord/.ssh
chmod 700 /home/fnord/.ssh
chmod 644 /home/fnord/.ssh/authorized_keys

passwd

Booting through

The last thing on our plate is writing a script that attaches the encrypted device and boots through to it.

We have split this into multiple files/stages to make it easier to repair and debug the encrypted system from the boot system.

Just put them wherever it's convenient for you. In our setup, they live in /root/.

We have…

xboot.sh

This is a wrapper script, that just calls the other scripts in the right order to boot through into the encrypted system.

#!/bin/sh

sh xboot-pre.sh # make sure relevant kernel modules are loaded
sh xboot-crypto.sh # attach all geli devices
sh xboot-finish.sh # reboot into the zpool

xboot-pre.sh

This script just makes sure we have the kernel modules we need loaded.

#!/bin/sh

kldload aesni
kldload geom_eli

echo "loaded relevant kernel modules. you can (probably) safely ignore errors above this."

xboot-crypto.sh

#!/bin/sh

echo "Holy passphrase:"
read passphrase
echo $passphrase > /tmp/passphrase

set -e # show errors
set -x # show commands as they're being run

geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank

rm -f /tmp/passphrase

set +x

echo "Attached all geli devices."

If you have just one geli device, you technically don't need to write the passphrase into a file; In that case you can just do geli attach -k /boot/tank.key gpt/tank. The reason for writing the passphrase into a file is a comfort feature for when you have multiple geli devices (and possibly multiple different keys) using the same passphrase.

xboot-finish.sh

This is where the magic happens. kenv let's us set the root file system of the next boot, while reboot -r let's us boot into it from the currently loaded kernel.

UFS
#!/bin/sh

kenv vfs.root.mountfrom="ufs:gpt/tank.eli"
reboot -r
ZFS
#!/bin/sh

kenv vfs.root.mountfrom="zfs:tank/root"
reboot -r

Full flow

Now that we have everything together, let's go over how the complete boot process plays out.

  • The machine starts
  • The unencrypted system boots, connects to the network and starts sshd
  • You ssh into the unencrypted system
  • You su to root
  • You execute xboot.sh and input your passphrase to reboot into the encrypted system

Phew. That was a lot of work. Pat yourself on the back for being a good cookie. Here's a gif of red pandas being adorable:

But honestly, we should probably talk about…

Pitfalls

There are a couple.

First and foremost, the system we use to boot is neither encrypted nor signed, meaning it can be freely manipulated by anyone with hardware access. This includes the kernel that ends up running the encrypted system.

As, to the best of our knowledge, FreeBSD doesn't currently support Secure Boot, this sadly isn't something we can get around – in the future it would be nice if we could adapt this pattern to have a signed system so the machine at least refuses to boot a compromised system. For now, you can use freebsd-update IDS on the boot system to check whether files have been tampered with.

Also, the pattern described in here uses the "legacy" boot mechanism instead of UEFI, so there'll be some changes needed for that when Secure Boot rolls around on FreeBSD.

If you use ZFS, only the dataset supplied in vfs.root.mountfrom is mounted, but not any children – even if you supply the whole pool instead of a specific dataset (at least as of 12.1).

So if you have any other datasets that need to be mounted at boot time, those have to go into /etc/fstab. The lines to mount them look like this:

tank/root/usr         /usr           zfs      rw   0   0

Another thing is that you have to maintain two systems, even if the first one is base-only. Make sure to check out the freebsd-update manual and read up on its -b flag if you want to update the unencrypted system from within the encrypted one.

An advanced setup

As a bonus, we're throwing in the actual scripts and configs we used to set our system up for inspiration. This does some fancy/harebrained things like HDD+SSD hybrid mirrors and sets up the unencrypted and the encrypted system in one go.

It very probably isn't something you just want to throw at your machine - running install.sh very definitely will shred your current system. You have been warned. ¯\_(ツ)_/¯

ada0 is an SSD, ada1 and ada2 are normal HDDs.

install.sh

#!/bin/sh

echo "Do this in /tmp and have enough memory, or else!"

set -x

kldload zfs
gmirror load
geli load

# cat with "set -e" enabled will cancel execution if the cat'ed files don't exist
set -e

echo "loader.conf?"
cat loader.conf
echo "---------------"

echo "rc.conf?"
cat rc.conf
echo "---------------"

###

pkg install ca_root_nss

###

echo "generating keyfiles…"
dd if=/dev/random bs=512 count=1 of=swap.key
dd if=/dev/random bs=512 count=1 of=fastread.key
dd if=/dev/random bs=512 count=1 of=l2arc.key
dd if=/dev/random bs=512 count=1 of=log.key
dd if=/dev/random bs=512 count=1 of=septic.key
dd if=/dev/random bs=512 count=1 of=tank.key

###
echo "Holy Passphrase:"
read passphrase
echo $passphrase >> passphrase

###

set +e
gpart destroy -F ada0
gpart destroy -F ada1
gpart destroy -F ada2
set -e

gpart create -s gpt ada0
gpart create -s gpt ada1
gpart create -s gpt ada2

###

gpart add -t freebsd-boot -s 512K -l gptzfsboot0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

gpart add -t freebsd-boot -s 512K -l gptzfsboot1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

gpart add -t freebsd-boot -s 512K -l gptzfsboot2 ada2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2

###

gpart add -t freebsd-swap -s 2G -l swap ada0

geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K swap.key gpt/swap

###

gpart add -t freebsd-zfs -s 5G -l xboot0 ada0
gpart add -t freebsd-zfs -s 5G -l xboot1 ada1
gpart add -t freebsd-zfs -s 5G -l xboot2 ada2

zpool create -fo altroot=/mnt xboot mirror gpt/xboot0 gpt/xboot1 gpt/xboot2
zfs create -o mountpoint=/ xboot/root
zpool set bootfs=xboot/root xboot

fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz

tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz

cp -p loader.conf /mnt/boot/
cp -p rc.conf /mnt/etc/
cp -p *.key /mnt/boot/

sync

echo "xboot system built, unmounting"

zpool export xboot

###

gpart add -t freebsd-ufs -s 80G -l fastread0 ada0
gpart add -t freebsd-ufs -s 80G -l fastread1 ada1
gpart add -t freebsd-ufs -s 80G -l fastread2 ada2

geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread0
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread2

geli attach -j passphrase -k fastread.key gpt/fastread0
geli attach -j passphrase -k fastread.key gpt/fastread1
geli attach -j passphrase -k fastread.key gpt/fastread2

gmirror label -b prefer fastread gpt/fastread0.eli gpt/fastread1.eli gpt/fastread2.eli
gmirror configure -p 255 fastread gpt/fastread0.eli

###

gpart add -t freebsd-zfs -l l2arc ada0

geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K l2arc.key gpt/l2arc

###

gpart add -t freebsd-ufs -s 50G -l log1 ada1
gpart add -t freebsd-ufs -s 50G -l log2 ada2

geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K log.key gpt/log1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K log.key gpt/log2

geli attach -j passphrase -k log.key gpt/log1
geli attach -j passphrase -k log.key gpt/log2

gmirror label log gpt/log1.eli gpt/log2.eli

newfs -U -j /dev/mirror/log

###

gpart add -t freebsd-ufs -s 500G -l septic1 ada1
gpart add -t freebsd-ufs -s 500G -l septic2 ada2

geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K septic.key gpt/septic1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K septic.key gpt/septic2

geli attach -j passphrase -k septic.key gpt/septic1
geli attach -j passphrase -k septic.key gpt/septic2

gmirror label septic gpt/septic1.eli gpt/septic2.eli

newfs -U -j /dev/mirror/septic

###

gpart add -t freebsd-zfs -l tank1 ada1
gpart add -t freebsd-zfs -l tank2 ada2

geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K tank.key gpt/tank1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K tank.key gpt/tank2

geli attach -j passphrase -k tank.key gpt/tank1
geli attach -j passphrase -k tank.key gpt/tank2

zpool create -fo altroot=/mnt tank mirror gpt/tank1.eli gpt/tank2.eli

zfs create -o mountpoint=/              tank/root
zfs create                              tank/root/usr
zfs create -o               setuid=off  tank/root/usr/ports
zfs create -o exec=off -o   setuid=off  tank/root/usr/src
zfs create                              tank/root/var
zfs create -o exec=off -o   setuid=off  tank/root/var/crash
zfs create -o               setuid=off  tank/root/var/db
zfs create -o exec=off -o   setuid=off  tank/root/var/empty
zfs create -o exec=off -o   setuid=off  tank/root/var/run

###

tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz

sync

zfs set readonly=on tank/root/var/empty

echo "If this boots, we're done for today."

xboot-pre.sh

#!/bin/sh

kldload aesni
kldload geom_eli
kldload geom_mirror

echo "loaded relevant kernel modules. you can (probably) safely ignore errors above this."

xboot-crypto.sh

#!/bin/sh

echo "Holy passphrase:"
read passphrase
echo $passphrase > /tmp/passphrase

set -e # show errors
set -x # show commands as they're being run

geli attach -j /tmp/passphrase -k /boot/swap.key gpt/swap

geli attach -j /tmp/passphrase -k /boot/l2arc.key gpt/l2arc

geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank1
geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank2

geli attach -j /tmp/passphrase -k /boot/log.key gpt/log1
geli attach -j /tmp/passphrase -k /boot/log.key gpt/log2

geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread0
geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread1
geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread2

geli attach -j /tmp/passphrase -k /boot/septic.key gpt/septic1
geli attach -j /tmp/passphrase -k /boot/septic.key gpt/septic2

rm -f /tmp/passphrase

set +x

echo "Attached all geli devices."

xboot-finish.sh

#!/bin/sh

set -x
set -e

kenv vfs.root.mountfrom="zfs:tank/root"
reboot -r

xboot-mount.sh

#!/bin/sh

# this script is mostly for rescue/debug purposes, not called by xboot.sh!

zpool import -o altroot=/mnt tank

mount /dev/mirror/log /mnt/var/log
mount /dev/mirror/fastread /mnt/mnt/fastread
mount /dev/mirror/septic /mnt/mnt/septic