HOW TO: FreeBSD remote-bootable crypto setup
Intro ¶
Recently, a friend sponsored a dedicated machine for our homeless ass.
Previously, we only ever had one dedicated machine on the internet and that was probably about a decade ago.
It was completely unencrypted.
This was not going to cut it this time – We wanted security that's up to snuff with our current abilities and needs. One of the key ingredients to that was going to encrypt as much as we could.
The problem being that the ISP the new machine resides with doesn't offer access to the machines serial console, so the only easy options to input any passphrases for encryption would be:
A) Storing the passphrase(s) in plaintext files on an unencrypted drive.
B) Requesting the ISP to connect a KVM switch in order to be able to boot. every.single. time.
Option A would end up making the whole setup pointless unless you also set up something like swatd to delete the passphrases (and keys) if the machine detects any tampering, because anyone getting access to the hardware can just disconnect it from the network, reboot it into a root shell and just mount your encrypted drives, gaining full access.
swatd in turn requires sensors that might or might not be avail- and dependable. All in all extremely
Option B would technically work, but requires some shite java client and makes you depend on ISP admins for every single boot, compounding every single downtime you experience.
We also think it's a bad idea because keyboard input is trivial to eavesdrop on by the ISP admins – even if the java shite uses proper encryption.
On Linux, there's an established pattern for this:
An initramfs with a statically linked dropbear SSH daemon through which you can load up the LUKS + LVM configuration and input any passphrases – Don't ask us exactly how it works, we stopped using Linux before routinely using disk encryption.
Debian even has a dedicated package to do just that: dropbear-initramfs
No such luck on FreeBSD.
We didn't find any How-To's or specialized packages, so…
What is to be done? ¶
Clearly, the FreeBSD community needs to establish a similar pattern to the one that's become commonplace in the Linux world.
There even is a semi-maintained collection of shell scripts to build something similar with a manually compiled and statically linked dropbear: freebsd-remote-crypto
Thing is, it has problems. For one, it's not very generic and assumes a specific setup with two drives which didn't work well with our plans. For another, it doesn't automate the building of dropbear, which we'd have to manually maintain – why even do this when we have a perfectly working SSHD in the base system and only have to worry about base-system updates that way?
Saving a couple megabytes on disk doesn't seem like a good justification for the extra effort needed to keep this setup up to date.
Its code however did serve as good reading, first and foremost because it taught us about reboot -r
, which is a critical component of our setup.
So we set out to create a minimal unencrypted FreeBSD install that can be accessed via the built-in SSH daemon and used to boot "through" to a fully encrypted system.
Pulling off a One Wheel Manual ¶
Lucky for us, FreeBSD is simple and doing a manual install is almost trivial. This is going to help us, because we need to set up not one, but two FreeBSDs on our machine, one just for booting and then the encrypted system we actually want to use.
Before we start diving in, let's clear some things up.
- This guide assumes that you'll initially be booting into a FreeBSD live image, supplied via network or whatever your ISP offers. Usually this is done through a web interface.
- We simplified this guide as much as we could, so you don't have to burn precious brain cycles by having to figure out which commands are relevant and important – they all are.
- Nevertheless, we're showing how to do this for UFS and ZFS. Filesystem-specific sections are all clearly marked in their headings as UFS or ZFS.
- We'll be using the more established
geli
for encryption, notgbde
– if you want that, you'll have to adapt the crypto setup described here yourself – should™ be rather trivial tho. - As this guide has been adopted from the much more complex setup on our server, some mistakes might have snuck in – please do tell if you stumble onto any!
So, without further ado…
Let's set up that boot system… ¶
If you want to minimize effort, you can probably skip ahead to Post-install configuration by just installing on a single 5G partition using bsdinstall. We're still telling you how to do this manually, because it will enable you to automate this sort of setup.
Since we'll be working on a mostly read-only filesystem, we'll put the data we need into /tmp
, which is a tmpfs living in memory. You'll need a couple hundred MB of free memory as we'll be putting a bit over 200MB of data on it.
Prerequisites ¶
As the image we have at our particular ISP does not contain the data archives needed to set up a base system, we'll also install the common set of CA root certificates, so we can safely fetch those archives over HTTPS.
If you can't install packages on your particular live image, you'll have to resort back to plain-text HTTP and verify that the SHA-256 checksums match the ones in the MANIFEST
file.
pkg install ca_root_nss # needed for HTTPS to not fail
cd /tmp
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz
- If your live image contains
kernel.txz
andbase.txz
, you can obviously use those instead of having to fetch the files. In that case, you can skip everything but thecd /tmp
. - Additionally, it probably goes without saying, but you should adjust these URLs to the right CPU architecture and release you want to install – if you're unsure, browse through the directory listing.
Boot system partitioning ¶
We'll be creating a GPT partition table, create one 512K-sized partition for the bootloader, which will then be installed with gpart bootcode
. 512K was chosen to make sure any variation of the bootloader fits while also making sure we don't go over the weird maximum size of 535K or something over which the boot will just fail.
After that, we're creating one 5G-sized partition to hold the minimal FreeBSD we'll use to boot into the encrypted system, set up a filesystem on it and mount it to /mnt
.
UFS ¶
gpart create -s gpt ada0
gpart add -s 512K -t freebsd-boot -l boot
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
gpart add -s 5G -t freebsd-ufs -l xboot # will hold the entire first FreeBSD
newfs -U -j gpt/xboot
mount gpt/xboot /mnt
ZFS ¶
gpart create -s gpt ada0
gpart add -s 512k -t freebsd-boot -l boot
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart add -s 5G -t freebsd-zfs -l xboot
zpool create -o altroot=/mnt xboot gpt/xboot
zfs create -o mountpoint=/ xboot/root # create a single dataset, technically not required but considered good form and needed if you want snapshots
zpool set bootfs=xboot/root xboot # we're not 100% sure this is needed, but it sure don't hurt.
Note that the zpool is automatically mounted, with the
altroot
parameter setting a temporary mountpoint.
Boot system install ¶
Next up, we'll extract the kernel.txz
and base.txz
, which is all the data needed to create a working base system.
tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz
Don't reboot just yet tho, because we still need to do some…
Post-install configuration ¶
We still have to do some things to make the system actually boot and be reachable via ssh.
First, to make it boot, we'll need to create
loader.conf ¶
This file goes to /mnt/boot/
.
UFS ¶
vfs.root.mountfrom="ufs:gpt/xboot"
Actually, we're not 100% sure whether this is needed if you use UFS, feel free to report back if you try booting without it.
ZFS ¶
# load the zfs kernel module
zfs_load="YES"
# tell the bootloader to boot from the xboot/root dataset
vfs.root.mountfrom="zfs:xboot/root"
SSH ¶
Lastly, we'll have to set up the network configuration, make sure the SSH daemon starts up and is accessible for us.
For that we need an…
rc.conf ¶
This file goes to /mnt/etc/
.
hostname="alkahest" # your choice
# Disable kernel dumps so the 5G disk space won't get completely
# filled at crash, which would break booting.
dumpdev="NO"
# Network configuration, the values that go here are prescribed by
# your ISP. You can look them up with "ifconfig" and
# "route show default".
ifconfig_re0="inet 176.9.110.174 netmask 255.255.255.224"
defaultrouter="176.9.110.161"
# automatically start the SSH daemon
sshd_enable="YES"
Account creation ¶
Now that we have the first FreeBSD set up, let's add a user so we can actually log in via SSH.
Switch into the newly installed system:
chroot /mnt
and add a user who's part of the wheel
group (which allows the user to switch to root via su
) by running adduser
:
$ adduser
Username: fnord
Full name:
Uid (Leave empty for default):
Login group [fnord]:
Login group is fnord. Invite fnord into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh git-shell fish bash rbash nologin) [sh]:
Home directory [/home/fnord]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]:
Username : fnord
Password :
Full Name :
Uid : 1001
Class :
Groups : fnord wheel
Home : /home/fnord
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
Next, add your SSH public key on the local machine you're doing this from to the created users' ~/.ssh/authorized_keys
:
mkdir /home/fnord/.ssh
# in vi: press i, middle-mouse paste your pubkey, press ESC,
# then type :x and press enter to save and quit.
vi /home/fnord/.ssh/authorized_keys
# change ownership and permissions of the involved files
chown -R fnord /home/fnord/.ssh
chmod 700 /home/fnord/.ssh
chmod 644 /home/fnord/.ssh/authorized_keys
Lastly, you might want to set a root password:
passwd
You can skip this step, but it will lead to any user in the wheel
group being able to su
to root without any password check.
Congratulations, you just did a completely manual FreeBSD installation – Welcome to the club of leet kids.
Finishing up ¶
Now, reboot
and hope like hell the system actually boots.
We recommend having a ping on the IP address of the remote machine running before you reboot, so you can see how the machine goes down and (hopefully) becomes reachable again after a minute or two.
If everything went well, the SSHD will be reachable a few seconds after the machine starts responding to ping again.
If you can't ssh
into the machine, reboot into your ISPs rescue system and try to figure out where you went wrong – or bug me in the comments, on the fediverse or on IRC. ;P
Encrypted system setup ¶
After doing all this, we finally get to set up the encrypted system we actually want to run and use.
ssh
into your created user and su
to root.
Partitioning ¶
For this guide, we'll go with simplicity and just create one big ZFS pool with all the space still free on ada0
:
UFS ¶
gpart add -t freebsd-ufs -l tank ada0
Yeah yeah, "tank" is ZFS terminology, deal with it.
ZFS ¶
gpart add -t freebsd-zfs -l tank ada0
Crypto setup ¶
Going on, let's create a keyfile so the cryptography will be working with more entropy than just the passphrase:
dd if=/dev/random bs=512 count=1 of=/boot/crypto.key
This gives you 512 byte (or 4096 bit) of entropy.
We honestly don't know if anything past 256 bits of that is actually used, feel free to weigh in if you know!
Next, we'll create the actual crypto device.
# load the kernel module for hardware-accelerated crypto.
# only do this if the CPU in question supports it.
# might be a different module for non-intel CPUs.
kldload aesni
geli load # same as kldload geom_eli
# We're using AES-XTS because it does encryption *and* authentication
# while being supported by AESNI. Otherwise you can forego either
# authentication by using something like AES-CBC or forego full
# hardware acceleration by supplying an authentication algorithm
# with the -a parameter.
geli init -e AES-XTS -l 256 -K /boot/tank.key gpt/tank
geli attach -k /boot/tank.key gpt/tank
If you have some experience with geli, you might have noticed that we're foregoing setting the boot flag when initializing our geli device. This is critically important – if you set the boot flag, not only will the boot setup hang at the passphrase prompt when booting – any FreeBSD live image, like the ISP rescue image, will too! This means that in order to boot anything, you'll have to boot into a live Linux to destroy any partitions containing geli devices and redo your encryption setup.
If it worked, the device /dev/gpt/tank.eli
now exists and we can create and mount our filesystem on the encrypted device.
UFS ¶
newfs -U -j gpt/tank.eli
mount gpt/tank.eli /mnt
ZFS ¶
zpool create -o altroot=/mnt tank gpt/tank.eli
zfs create -o mountpoint=/ tank/root
Encrypted setup install ¶
Now, we basically have to rinse and repeat what we did for the boot install:
pkg install ca_root_nss
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz
tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz
cp -p /boot/loader.conf /mnt/boot/
cp -p /etc/rc.conf /mnt/etc/
chroot /mnt # switch into the encrypted install
adduser # add user 'fnord'
mkdir /home/fnord/.ssh
vi /home/fnord/.ssh/authorized_keys # add your public key
chown -R fnord /home/fnord/.ssh
chmod 700 /home/fnord/.ssh
chmod 644 /home/fnord/.ssh/authorized_keys
passwd
Booting through ¶
The last thing on our plate is writing a script that attaches the encrypted device and boots through to it.
We have split this into multiple files/stages to make it easier to repair and debug the encrypted system from the boot system.
Just put them wherever it's convenient for you. In our setup, they live in /root/
.
We have…
xboot.sh ¶
This is a wrapper script, that just calls the other scripts in the right order to boot through into the encrypted system.
#!/bin/sh
sh xboot-pre.sh # make sure relevant kernel modules are loaded
sh xboot-crypto.sh # attach all geli devices
sh xboot-finish.sh # reboot into the zpool
xboot-pre.sh ¶
This script just makes sure we have the kernel modules we need loaded.
#!/bin/sh
kldload aesni
kldload geom_eli
echo "loaded relevant kernel modules. you can (probably) safely ignore errors above this."
xboot-crypto.sh ¶
#!/bin/sh
echo "Holy passphrase:"
read passphrase
echo $passphrase > /tmp/passphrase
set -e # show errors
set -x # show commands as they're being run
geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank
rm -f /tmp/passphrase
set +x
echo "Attached all geli devices."
If you have just one geli device, you technically don't need to write the passphrase into a file; In that case you can just do
geli attach -k /boot/tank.key gpt/tank
. The reason for writing the passphrase into a file is a comfort feature for when you have multiple geli devices (and possibly multiple different keys) using the same passphrase.
xboot-finish.sh ¶
This is where the magic happens. kenv
let's us set the root file system of the next boot, while reboot -r
let's us boot into it from the currently loaded kernel.
UFS ¶
#!/bin/sh
kenv vfs.root.mountfrom="ufs:gpt/tank.eli"
reboot -r
ZFS ¶
#!/bin/sh
kenv vfs.root.mountfrom="zfs:tank/root"
reboot -r
Full flow ¶
Now that we have everything together, let's go over how the complete boot process plays out.
- The machine starts
- The unencrypted system boots, connects to the network and starts sshd
- You
ssh
into the unencrypted system - You
su
to root - You execute
xboot.sh
and input your passphrase to reboot into the encrypted system
Phew. That was a lot of work. Pat yourself on the back for being a good cookie. Here's a gif of red pandas being adorable:
But honestly, we should probably talk about…
Pitfalls ¶
There are a couple.
First and foremost, the system we use to boot is neither encrypted nor signed, meaning it can be freely manipulated by anyone with hardware access. This includes the kernel that ends up running the encrypted system.
As, to the best of our knowledge, FreeBSD doesn't currently support Secure Boot, this sadly isn't something we can get around – in the future it would be nice if we could adapt this pattern to have a signed system so the machine at least refuses to boot a compromised system. For now, you can use freebsd-update IDS
on the boot system to check whether files have been tampered with.
Also, the pattern described in here uses the "legacy" boot mechanism instead of UEFI, so there'll be some changes needed for that when Secure Boot rolls around on FreeBSD.
If you use ZFS, only the dataset supplied in vfs.root.mountfrom
is mounted, but not any children – even if you supply the whole pool instead of a specific dataset (at least as of 12.1).
So if you have any other datasets that need to be mounted at boot time, those have to go into /etc/fstab
. The lines to mount them look like this:
tank/root/usr /usr zfs rw 0 0
Another thing is that you have to maintain two systems, even if the first one is base-only. Make sure to check out the freebsd-update
manual and read up on its -b
flag if you want to update the unencrypted system from within the encrypted one.
An advanced setup ¶
As a bonus, we're throwing in the actual scripts and configs we used to set our system up for inspiration. This does some fancy/harebrained things like HDD+SSD hybrid mirrors and sets up the unencrypted and the encrypted system in one go.
It very probably isn't something you just want to throw at your machine - running install.sh very definitely will shred your current system.
You have been warned. ¯\_(ツ)_/¯
ada0
is an SSD,ada1
andada2
are normal HDDs.
install.sh ¶
#!/bin/sh
echo "Do this in /tmp and have enough memory, or else!"
set -x
kldload zfs
gmirror load
geli load
# cat with "set -e" enabled will cancel execution if the cat'ed files don't exist
set -e
echo "loader.conf?"
cat loader.conf
echo "---------------"
echo "rc.conf?"
cat rc.conf
echo "---------------"
###
pkg install ca_root_nss
###
echo "generating keyfiles…"
dd if=/dev/random bs=512 count=1 of=swap.key
dd if=/dev/random bs=512 count=1 of=fastread.key
dd if=/dev/random bs=512 count=1 of=l2arc.key
dd if=/dev/random bs=512 count=1 of=log.key
dd if=/dev/random bs=512 count=1 of=septic.key
dd if=/dev/random bs=512 count=1 of=tank.key
###
echo "Holy Passphrase:"
read passphrase
echo $passphrase >> passphrase
###
set +e
gpart destroy -F ada0
gpart destroy -F ada1
gpart destroy -F ada2
set -e
gpart create -s gpt ada0
gpart create -s gpt ada1
gpart create -s gpt ada2
###
gpart add -t freebsd-boot -s 512K -l gptzfsboot0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart add -t freebsd-boot -s 512K -l gptzfsboot1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gpart add -t freebsd-boot -s 512K -l gptzfsboot2 ada2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2
###
gpart add -t freebsd-swap -s 2G -l swap ada0
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K swap.key gpt/swap
###
gpart add -t freebsd-zfs -s 5G -l xboot0 ada0
gpart add -t freebsd-zfs -s 5G -l xboot1 ada1
gpart add -t freebsd-zfs -s 5G -l xboot2 ada2
zpool create -fo altroot=/mnt xboot mirror gpt/xboot0 gpt/xboot1 gpt/xboot2
zfs create -o mountpoint=/ xboot/root
zpool set bootfs=xboot/root xboot
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz
fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz
tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz
cp -p loader.conf /mnt/boot/
cp -p rc.conf /mnt/etc/
cp -p *.key /mnt/boot/
sync
echo "xboot system built, unmounting"
zpool export xboot
###
gpart add -t freebsd-ufs -s 80G -l fastread0 ada0
gpart add -t freebsd-ufs -s 80G -l fastread1 ada1
gpart add -t freebsd-ufs -s 80G -l fastread2 ada2
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread0
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread2
geli attach -j passphrase -k fastread.key gpt/fastread0
geli attach -j passphrase -k fastread.key gpt/fastread1
geli attach -j passphrase -k fastread.key gpt/fastread2
gmirror label -b prefer fastread gpt/fastread0.eli gpt/fastread1.eli gpt/fastread2.eli
gmirror configure -p 255 fastread gpt/fastread0.eli
###
gpart add -t freebsd-zfs -l l2arc ada0
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K l2arc.key gpt/l2arc
###
gpart add -t freebsd-ufs -s 50G -l log1 ada1
gpart add -t freebsd-ufs -s 50G -l log2 ada2
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K log.key gpt/log1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K log.key gpt/log2
geli attach -j passphrase -k log.key gpt/log1
geli attach -j passphrase -k log.key gpt/log2
gmirror label log gpt/log1.eli gpt/log2.eli
newfs -U -j /dev/mirror/log
###
gpart add -t freebsd-ufs -s 500G -l septic1 ada1
gpart add -t freebsd-ufs -s 500G -l septic2 ada2
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K septic.key gpt/septic1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K septic.key gpt/septic2
geli attach -j passphrase -k septic.key gpt/septic1
geli attach -j passphrase -k septic.key gpt/septic2
gmirror label septic gpt/septic1.eli gpt/septic2.eli
newfs -U -j /dev/mirror/septic
###
gpart add -t freebsd-zfs -l tank1 ada1
gpart add -t freebsd-zfs -l tank2 ada2
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K tank.key gpt/tank1
geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K tank.key gpt/tank2
geli attach -j passphrase -k tank.key gpt/tank1
geli attach -j passphrase -k tank.key gpt/tank2
zpool create -fo altroot=/mnt tank mirror gpt/tank1.eli gpt/tank2.eli
zfs create -o mountpoint=/ tank/root
zfs create tank/root/usr
zfs create -o setuid=off tank/root/usr/ports
zfs create -o exec=off -o setuid=off tank/root/usr/src
zfs create tank/root/var
zfs create -o exec=off -o setuid=off tank/root/var/crash
zfs create -o setuid=off tank/root/var/db
zfs create -o exec=off -o setuid=off tank/root/var/empty
zfs create -o exec=off -o setuid=off tank/root/var/run
###
tar -C /mnt -xvpf kernel.txz
tar -C /mnt -xvpf base.txz
sync
zfs set readonly=on tank/root/var/empty
echo "If this boots, we're done for today."
xboot-pre.sh ¶
#!/bin/sh
kldload aesni
kldload geom_eli
kldload geom_mirror
echo "loaded relevant kernel modules. you can (probably) safely ignore errors above this."
xboot-crypto.sh ¶
#!/bin/sh
echo "Holy passphrase:"
read passphrase
echo $passphrase > /tmp/passphrase
set -e # show errors
set -x # show commands as they're being run
geli attach -j /tmp/passphrase -k /boot/swap.key gpt/swap
geli attach -j /tmp/passphrase -k /boot/l2arc.key gpt/l2arc
geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank1
geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank2
geli attach -j /tmp/passphrase -k /boot/log.key gpt/log1
geli attach -j /tmp/passphrase -k /boot/log.key gpt/log2
geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread0
geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread1
geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread2
geli attach -j /tmp/passphrase -k /boot/septic.key gpt/septic1
geli attach -j /tmp/passphrase -k /boot/septic.key gpt/septic2
rm -f /tmp/passphrase
set +x
echo "Attached all geli devices."
xboot-finish.sh ¶
#!/bin/sh
set -x
set -e
kenv vfs.root.mountfrom="zfs:tank/root"
reboot -r
xboot-mount.sh ¶
#!/bin/sh
# this script is mostly for rescue/debug purposes, not called by xboot.sh!
zpool import -o altroot=/mnt tank
mount /dev/mirror/log /mnt/var/log
mount /dev/mirror/fastread /mnt/mnt/fastread
mount /dev/mirror/septic /mnt/mnt/septic