Recently, a friend sponsored a dedicated machine for our homeless ass.
Previously, we only ever had one dedicated machine on the internet and that was probably about a decade ago.
It was completely unencrypted.
This was not going to cut it this time – We wanted security that's up to snuff with our current abilities and needs. One of the key ingredients to that was going to encrypt as much as we could.
The problem being that the ISP the new machine resides with doesn't offer access to the machines serial console, so the only easy options to input any passphrases for encryption would be:
A) Storing the passphrase(s) in plaintext files on an unencrypted drive.
B) Requesting the ISP to connect a KVM switch in order to be able to boot. every.single. time.
Option A would end up making the whole setup pointless unless you also set up something like swatd to delete the passphrases (and keys) if the machine detects any tampering, because anyone getting access to the hardware can just disconnect it from the network, reboot it into a root shell and just mount your encrypted drives, gaining full access.
swatd in turn requires sensors that might or might not be avail-
and dependable. All in all extremely
Option B would technically work, but requires some shite java client and makes you depend on ISP admins for every single boot, compounding every single downtime you experience.
We also think it's a bad idea because keyboard input is trivial to eavesdrop on by the ISP admins – even if the java shite uses proper encryption.
On Linux, there's an established pattern for this:
An initramfs with a statically linked dropbear SSH daemon through which you can load up the LUKS + LVM configuration and input any passphrases – Don't ask us exactly how it works, we stopped using Linux before routinely using disk encryption.
Debian even has a dedicated package to do just that:
No such luck on FreeBSD.
We didn't find any How-To's or specialized packages, so…
What is to be done?
Clearly, the FreeBSD community needs to establish a similar pattern to the one that's become commonplace in the Linux world.
There even is a semi-maintained collection of shell scripts to build something similar with a manually compiled and statically linked dropbear: freebsd-remote-crypto
Thing is, it has problems. For one, it's not very generic and assumes a specific setup with two drives which didn't work well with our plans. For another, it doesn't automate the building of dropbear, which we'd have to manually maintain – why even do this when we have a perfectly working SSHD in the base system and only have to worry about base-system updates that way?
Saving a couple megabytes on disk doesn't seem like a good justification for the extra effort needed to keep this setup up to date.
Its code however did serve as good reading, first and foremost
because it taught us about
reboot -r, which is a critical
component of our setup.
So we set out to create a minimal unencrypted FreeBSD install that can be accessed via the built-in SSH daemon and used to boot "through" to a fully encrypted system.
Pulling off a One Wheel Manual
Lucky for us, FreeBSD is simple and doing a manual install is almost trivial. This is going to help us, because we need to set up not one, but two FreeBSDs on our machine, one just for booting and then the encrypted system we actually want to use.
Before we start diving in, let's clear some things up.
This guide assumes that you'll initially be booting into a FreeBSD live image, supplied via network or whatever your ISP offers. Usually this is done through a web interface.
We simplified this guide as much as we could, so you don't have to burn precious brain cycles by having to figure out which commands are relevant and important – they all are.
Nevertheless, we're showing how to do this for UFS and ZFS. Filesystem-specific sections are all clearly marked in their headings as UFS or ZFS.
We'll be using the more established geli for encryption, not gbde – if you want that, you'll have to adapt the crypto setup described here yourself – should™ be rather trivial tho.
As this guide has been adopted from the much more complex setup on our server, some mistakes might have snuck in – please do tell if you stumble onto any!
So, without further ado…
Let's set up that boot system…
If you want to minimize effort, you can probably skip ahead to Post-install configuration by just installing on a single 5G partition using bsdinstall. We're still telling you how to do this manually, because it will enable you to automate this sort of setup.
Since we'll be working on a mostly read-only filesystem,
we'll put the data we need into
/tmp, which is a tmpfs
living in memory. You'll need a couple hundred MB of free
memory as we'll be putting a bit over 200MB of data on it.
As the image we have at our particular ISP does not contain the data archives needed to set up a base system, we'll also install the common set of CA root certificates, so we can safely fetch those archives over HTTPS.
If you can't install packages on your particular live image,
you'll have to resort back to plain-text HTTP and verify that
the SHA-256 checksums match the ones in the
pkg install ca_root_nss # needed for HTTPS to not fail cd /tmp fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz
- If your live image contains kernel.txz and base.txz, you can
obviously use those instead of having to fetch the files.
In that case, you can skip everything but the
- Additionally, it probably goes without saying, but you should adjust these URLs to the right CPU architecture and release you want to install – if you're unsure, browse through the directory listing.
Boot system partitioning
We'll be creating a GPT partition table, create one 512K-sized
partition for the bootloader, which will then be installed with
gpart bootcode. 512K was chosen to make sure any variation
of the bootloader fits while also making sure we don't go over
the weird maximum size of 535K or something over which the boot
will just fail.
After that, we're creating one 5G-sized partition to hold the
minimal FreeBSD we'll use to boot into the encrypted system,
set up a filesystem on it and mount it to
gpart create -s gpt ada0 gpart add -s 512K -t freebsd-boot -l boot gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 gpart add -s 5G -t freebsd-ufs -l xboot # will hold the entire first FreeBSD newfs -U -j gpt/xboot mount gpt/xboot /mnt
gpart create -s gpt ada0 gpart add -s 512k -t freebsd-boot -l boot gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 gpart add -s 5G -t freebsd-zfs -l xboot zpool create -o altroot=/mnt xboot gpt/xboot zfs create -o mountpoint=/ xboot/root # create a single dataset, technically not required but considered good form and needed if you want snapshots zpool set bootfs=xboot/root xboot # we're not 100% sure this is needed, but it sure don't hurt.
Note that the zpool is automatically mounted, with the
altrootparameter setting a temporary mountpoint.
Boot system install
Next up, we'll extract the
base.txz, which is all
the data needed to create a working base system.
tar -C /mnt -xvpf kernel.txz tar -C /mnt -xvpf base.txz
Don't reboot just yet tho, because we still need to do some…
We still have to do some things to make the system actually boot and be reachable via ssh.
First, to make it boot, we'll need to create
This file goes to
Actually, we're not 100% sure whether this is needed if you use UFS, feel free to report back if you try booting without it.
# load the zfs kernel module zfs_load="YES" # tell the bootloader to boot from the xboot/root dataset vfs.root.mountfrom="zfs:xboot/root"
Lastly, we'll have to set up the network configuration, make sure the SSH daemon starts up and is accessible for us.
For that we need an…
This file goes to
hostname="alkahest" # your choice # Disable kernel dumps so the 5G disk space won't get completely # filled at crash, which would break booting. dumpdev="NO" # Network configuration, the values that go here are prescribed by # your ISP. You can look them up with "ifconfig" and # "route show default". ifconfig_re0="inet 18.104.22.168 netmask 255.255.255.224" defaultrouter="22.214.171.124" # automatically start the SSH daemon sshd_enable="YES"
Now that we have the first FreeBSD set up, let's add a user so we can actually log in via SSH.
Switch into the newly installed system:
and add a user who's part of the
wheel group (which allows the user
to switch to root via
su) by running
$ adduser Username: fnord Full name: Uid (Leave empty for default): Login group [fnord]: Login group is fnord. Invite fnord into other groups? : wheel Login class [default]: Shell (sh csh tcsh git-shell fish bash rbash nologin) [sh]: Home directory [/home/fnord]: Home directory permissions (Leave empty for default): Use password-based authentication? [yes]: no Lock out the account after creation? [no]: Username : fnord Password : <disabled> Full Name : Uid : 1001 Class : Groups : fnord wheel Home : /home/fnord Home Mode : Shell : /bin/sh Locked : no OK? (yes/no): y
Next, add your SSH public key on the local machine you're doing
this from to the created users'
mkdir /home/fnord/.ssh # in vi: press i, middle-mouse paste your pubkey, press ESC, # then type :x and press enter to save and quit. vi /home/fnord/.ssh/authorized_keys # change ownership and permissions of the involved files chown -R fnord /home/fnord/.ssh chmod 700 /home/fnord/.ssh chmod 644 /home/fnord/.ssh/authorized_keys
Lastly, you might want to set a root password:
You can skip this step, but it will lead to any user in the
group being able to
su to root without any password check.
Congratulations, you just did a completely manual FreeBSD installation – Welcome to the club of leet kids.
reboot and hope like hell the system actually boots.
We recommend having a ping on the IP address of the remote machine running before you reboot, so you can see how the machine goes down and (hopefully) becomes reachable again after a minute or two.
If everything went well, the SSHD will be reachable a few seconds after the machine starts responding to ping again.
If you can't
ssh into the machine, reboot into your ISPs rescue
system and try to figure out where you went wrong – or bug me in
the comments, on the fediverse or on IRC. ;P
Encrypted system setup
After doing all this, we finally get to set up the encrypted system we actually want to run and use.
ssh into your created user and
su to root.
For this guide, we'll go with simplicity and just create one big
ZFS pool with all the space still free on
gpart add -t freebsd-ufs -l tank ada0
Yeah yeah, "tank" is ZFS terminology, deal with it.
gpart add -t freebsd-zfs -l tank ada0
Going on, let's create a keyfile so the cryptography will be working with more entropy than just the passphrase:
dd if=/dev/random bs=512 count=1 of=/boot/crypto.key
This gives you 512 byte (or 4096 bit) of entropy.
We honestly don't know if anything past 256 bits of that is actually used, feel free to weigh in if you know!
Next, we'll create the actual crypto device.
# load the kernel module for hardware-accelerated crypto. # only do this if the CPU in question supports it. # might be a different module for non-intel CPUs. kldload aesni geli load # same as kldload geom_eli # We're using AES-XTS because it does encryption *and* authentication # while being supported by AESNI. Otherwise you can forego either # authentication by using something like AES-CBC or forego full # hardware acceleration by supplying an authentication algorithm # with the -a parameter. geli init -e AES-XTS -l 256 -K /boot/tank.key gpt/tank geli attach -k /boot/tank.key gpt/tank
If you have some experience with geli, you might have noticed that we're foregoing setting the boot flag when initializing our geli device. This is critically important – if you set the boot flag, not only will the boot setup hang at the passphrase prompt when booting – any FreeBSD live image, like the ISP rescue image, will too! This means that in order to boot anything, you'll have to boot into a live Linux to destroy any partitions containing geli devices and redo your encryption setup.
If it worked, the device
/dev/gpt/tank.eli now exists and
we can create and mount our filesystem on the encrypted device.
newfs -U -j gpt/tank.eli mount gpt/tank.eli /mnt
zpool create -o altroot=/mnt tank gpt/tank.eli zfs create -o mountpoint=/ tank/root
Encrypted setup install
Now, we basically have to rinse and repeat what we did for the boot install:
pkg install ca_root_nss fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz tar -C /mnt -xvpf kernel.txz tar -C /mnt -xvpf base.txz cp -p /boot/loader.conf /mnt/boot/ cp -p /etc/rc.conf /mnt/etc/ chroot /mnt # switch into the encrypted install adduser # add user 'fnord' mkdir /home/fnord/.ssh vi /home/fnord/.ssh/authorized_keys # add your public key chown -R fnord /home/fnord/.ssh chmod 700 /home/fnord/.ssh chmod 644 /home/fnord/.ssh/authorized_keys passwd
The last thing on our plate is writing a script that attaches the encrypted device and boots through to it.
We have split this into multiple files/stages to make it easier to repair and debug the encrypted system from the boot system.
Just put them wherever it's convenient for you.
In our setup, they live in
This is a wrapper script, that just calls the other scripts in the right order to boot through into the encrypted system.
#!/bin/sh sh xboot-pre.sh # make sure relevant kernel modules are loaded sh xboot-crypto.sh # attach all geli devices sh xboot-finish.sh # reboot into the zpool
This script just makes sure we have the kernel modules we need loaded.
#!/bin/sh kldload aesni kldload geom_eli echo "loaded relevant kernel modules. you can (probably) safely ignore errors above this."
#!/bin/sh echo "Holy passphrase:" read passphrase echo $passphrase > /tmp/passphrase set -e # show errors set -x # show commands as they're being run geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank rm -f /tmp/passphrase set +x echo "Attached all geli devices."
If you have just one geli device, you technically don't need to write the passphrase into a file; In that case you can just do
geli attach -k /boot/tank.key gpt/tank. The reason for writing the passphrase into a file is a comfort feature for when you have multiple geli devices (and possibly multiple different keys) using the same passphrase.
This is where the magic happens.
kenv let's us set the
root file system of the next boot, while
reboot -r let's
us boot into it from the currently loaded kernel.
#!/bin/sh kenv vfs.root.mountfrom="ufs:gpt/tank.eli" reboot -r
#!/bin/sh kenv vfs.root.mountfrom="zfs:tank/root" reboot -r
Now that we have everything together, let's go over how the complete boot process plays out.
- The machine starts
- The unencrypted system boots, connects to the network and starts sshd
sshinto the unencrypted system
- You execute
xboot.shand input your passphrase to reboot into the encrypted system
Phew. That was a lot of work. Pat yourself on the back for being a good cookie. Here's a gif of red pandas being adorable:
But honestly, we should probably talk about…
There are a couple.
First and foremost, the system we use to boot is neither encrypted nor signed, meaning it can be freely manipulated by anyone with hardware access. This includes the kernel that ends up running the encrypted system.
As, to the best of our knowledge, FreeBSD doesn't currently support
Secure Boot, this sadly isn't something we can get around – in the
future it would be nice if we could adapt this pattern to have a
signed system so the machine at least refuses to boot a compromised
system. For now, you can use
freebsd-update IDS on the boot system
to check whether files have been tampered with.
Also, the pattern described in here uses the "legacy" boot mechanism instead of UEFI, so there'll be some changes needed for that when Secure Boot rolls around on FreeBSD.
If you use ZFS, only the dataset supplied in
is mounted, but not any children – even if you supply the whole
pool instead of a specific dataset (at least as of 12.1).
So if you have any other datasets that need to be mounted at boot
time, those have to go into
/etc/fstab. The lines to mount them
look like this:
tank/root/usr /usr zfs rw 0 0
Another thing is that you have to maintain two systems, even if
the first one is base-only. Make sure to check out the
manual and read up on its
-b flag if you want to update the
unencrypted system from within the encrypted one.
An advanced setup
As a bonus, we're throwing in the actual scripts and configs we used to set our system up for inspiration. This does some fancy/harebrained things like HDD+SSD hybrid mirrors and sets up the unencrypted and the encrypted system in one go.
It very probably isn't something you just want to throw at your machine - running install.sh very definitely will shred your current system. You have been warned. ¯\_(ツ)_/¯
ada0is an SSD,
ada2are normal HDDs.
#!/bin/sh echo "Do this in /tmp and have enough memory, or else!" set -x kldload zfs gmirror load geli load # cat with "set -e" enabled will cancel execution if the cat'ed files don't exist set -e echo "loader.conf?" cat loader.conf echo "---------------" echo "rc.conf?" cat rc.conf echo "---------------" ### pkg install ca_root_nss ### echo "generating keyfiles…" dd if=/dev/random bs=512 count=1 of=swap.key dd if=/dev/random bs=512 count=1 of=fastread.key dd if=/dev/random bs=512 count=1 of=l2arc.key dd if=/dev/random bs=512 count=1 of=log.key dd if=/dev/random bs=512 count=1 of=septic.key dd if=/dev/random bs=512 count=1 of=tank.key ### echo "Holy Passphrase:" read passphrase echo $passphrase >> passphrase ### set +e gpart destroy -F ada0 gpart destroy -F ada1 gpart destroy -F ada2 set -e gpart create -s gpt ada0 gpart create -s gpt ada1 gpart create -s gpt ada2 ### gpart add -t freebsd-boot -s 512K -l gptzfsboot0 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 gpart add -t freebsd-boot -s 512K -l gptzfsboot1 ada1 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 gpart add -t freebsd-boot -s 512K -l gptzfsboot2 ada2 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2 ### gpart add -t freebsd-swap -s 2G -l swap ada0 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K swap.key gpt/swap ### gpart add -t freebsd-zfs -s 5G -l xboot0 ada0 gpart add -t freebsd-zfs -s 5G -l xboot1 ada1 gpart add -t freebsd-zfs -s 5G -l xboot2 ada2 zpool create -fo altroot=/mnt xboot mirror gpt/xboot0 gpt/xboot1 gpt/xboot2 zfs create -o mountpoint=/ xboot/root zpool set bootfs=xboot/root xboot fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/kernel.txz fetch https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/base.txz tar -C /mnt -xvpf kernel.txz tar -C /mnt -xvpf base.txz cp -p loader.conf /mnt/boot/ cp -p rc.conf /mnt/etc/ cp -p *.key /mnt/boot/ sync echo "xboot system built, unmounting" zpool export xboot ### gpart add -t freebsd-ufs -s 80G -l fastread0 ada0 gpart add -t freebsd-ufs -s 80G -l fastread1 ada1 gpart add -t freebsd-ufs -s 80G -l fastread2 ada2 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread0 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread1 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K fastread.key gpt/fastread2 geli attach -j passphrase -k fastread.key gpt/fastread0 geli attach -j passphrase -k fastread.key gpt/fastread1 geli attach -j passphrase -k fastread.key gpt/fastread2 gmirror label -b prefer fastread gpt/fastread0.eli gpt/fastread1.eli gpt/fastread2.eli gmirror configure -p 255 fastread gpt/fastread0.eli ### gpart add -t freebsd-zfs -l l2arc ada0 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K l2arc.key gpt/l2arc ### gpart add -t freebsd-ufs -s 50G -l log1 ada1 gpart add -t freebsd-ufs -s 50G -l log2 ada2 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K log.key gpt/log1 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K log.key gpt/log2 geli attach -j passphrase -k log.key gpt/log1 geli attach -j passphrase -k log.key gpt/log2 gmirror label log gpt/log1.eli gpt/log2.eli newfs -U -j /dev/mirror/log ### gpart add -t freebsd-ufs -s 500G -l septic1 ada1 gpart add -t freebsd-ufs -s 500G -l septic2 ada2 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K septic.key gpt/septic1 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K septic.key gpt/septic2 geli attach -j passphrase -k septic.key gpt/septic1 geli attach -j passphrase -k septic.key gpt/septic2 gmirror label septic gpt/septic1.eli gpt/septic2.eli newfs -U -j /dev/mirror/septic ### gpart add -t freebsd-zfs -l tank1 ada1 gpart add -t freebsd-zfs -l tank2 ada2 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K tank.key gpt/tank1 geli init -e AES-XTS -l 256 -s 4096 -J passphrase -K tank.key gpt/tank2 geli attach -j passphrase -k tank.key gpt/tank1 geli attach -j passphrase -k tank.key gpt/tank2 zpool create -fo altroot=/mnt tank mirror gpt/tank1.eli gpt/tank2.eli zfs create -o mountpoint=/ tank/root zfs create tank/root/usr zfs create -o setuid=off tank/root/usr/ports zfs create -o exec=off -o setuid=off tank/root/usr/src zfs create tank/root/var zfs create -o exec=off -o setuid=off tank/root/var/crash zfs create -o setuid=off tank/root/var/db zfs create -o exec=off -o setuid=off tank/root/var/empty zfs create -o exec=off -o setuid=off tank/root/var/run ### tar -C /mnt -xvpf kernel.txz tar -C /mnt -xvpf base.txz sync zfs set readonly=on tank/root/var/empty echo "If this boots, we're done for today."
#!/bin/sh kldload aesni kldload geom_eli kldload geom_mirror echo "loaded relevant kernel modules. you can (probably) safely ignore errors above this."
#!/bin/sh echo "Holy passphrase:" read passphrase echo $passphrase > /tmp/passphrase set -e # show errors set -x # show commands as they're being run geli attach -j /tmp/passphrase -k /boot/swap.key gpt/swap geli attach -j /tmp/passphrase -k /boot/l2arc.key gpt/l2arc geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank1 geli attach -j /tmp/passphrase -k /boot/tank.key gpt/tank2 geli attach -j /tmp/passphrase -k /boot/log.key gpt/log1 geli attach -j /tmp/passphrase -k /boot/log.key gpt/log2 geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread0 geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread1 geli attach -j /tmp/passphrase -k /boot/fastread.key gpt/fastread2 geli attach -j /tmp/passphrase -k /boot/septic.key gpt/septic1 geli attach -j /tmp/passphrase -k /boot/septic.key gpt/septic2 rm -f /tmp/passphrase set +x echo "Attached all geli devices."
#!/bin/sh set -x set -e kenv vfs.root.mountfrom="zfs:tank/root" reboot -r
#!/bin/sh # this script is mostly for rescue/debug purposes, not called by xboot.sh! zpool import -o altroot=/mnt tank mount /dev/mirror/log /mnt/var/log mount /dev/mirror/fastread /mnt/mnt/fastread mount /dev/mirror/septic /mnt/mnt/septic