Network booting an aarch64 SBC with u-boot and iPXE

I recently started trying to figure out network booting for aarch64 single board computers (SBC), such as the Raspberry Pi, for a new CI I've been helping out with at my Igalia day job. For one reason or another, I never participated in the Raspberry Pi "fad" (maybe because they use Broadcom chips, which are [or were] notoriously unfriendly on Linux? I don't recall why... But I digress...)

But, I do have a quite capable aarch64 SBC just laying around, literally collecting dust... the Purism Librem 5 DevKit!

While not exactly a Raspberry Pi, I believe many of the concepts pre-Linux boot are similar and this should serve as a decent replacement until the Great Chip Shortage of 2020-???? is over and those things are available for purchase again.

The general idea is that u-boot will execute iPXE, which will be responsible establishing a network connection and booting whatever the DHCP server on the other end tells it to boot. The end goal is to have it load/boot the Linux kernel and an initfs based on boot2container.

This is the first in a series of posts to get there. The focus of this initial post is building/setting up iPXE, the DHCP server, and doing a test boot from u-boot.

Building iPXE and configuring the devkit

The first step is to build iPXE, since I want to embed a script for it to run automatically on boot. I did the compilation on the devkit, since iPXE is a relatively small program and it didn't take too long to compile on this CPU:

$ git clone git://git.ipxe.org/ipxe.git
$ cd ipxe/src

## needed so that ipxe doesn't lock up if you want to C-b to enter the cmdline
$ cat << EOF > config/local/nap.h
#undef NAP_EFIX86
#undef NAP_EFIARM
#define NAP_NULL
EOF

## and create a simple ipxe script that will be executed when ipxe runs:
$ cat << EOF > ipxescript
#!ipxe

:retry_dhcp
echo Acquiring an IP
dhcp || goto retry_dhcp

echo Got the IP: $${netX/ip} / $${netX/netmask}

:retry_boot
echo Booting from DHCP...
autoboot || goto retry_boot
EOF

## build/install:
$ make bin-arm64-efi/snp.efi -j4 EMBED=ipxescript
$ doas cp bin-arm64-efi/snp.efi /boot/ipxe.efi

Loading things in u-boot is quite tedious, since you have to specify memory addresses to load files into, and the correct *load command to read files into memory. I already have an existing install of postmarketOS on my devkit, so I used the /boot partition (formatted as ext2) as a home for the iPXE binary. I created the following U-boot helper script for loading iPXE, since typing all of these in becomes tiresome very quickly:

$ cat << EOF > /tmp/ipxe 
echo ===== Loading iPXE =====
ext2load mmc 0:1 $kernel_addr_r ipxe.efi
ext2load mmc 0:1 $fdt_addr_r imx8mq-librem5-devkit.dtb
fdt addr $fdt_addr_r
fdt resize
echo ===== Running iPXE =====
bootefi $kernel_addr_r $fdt_addr_r
EOF

I'm not entirely sure if we need to specify/load the dtb, but it doesn't seem to hurt! Also note that this u-boot script is using bootefi to load the iPXE app. That'll be important later on when we try to boot a kernel.

The u-boot script must be compiled before u-boot can execute it:

$ mkimage -A arm64 -C none -O linux -T script -d /tmp/ipxe /tmp/ipxe.scr
$ doas cp /tmp/ipxe.scr /boot

In that last step, I copy it to /boot since I'm performing these steps on the devkit, and /boot is the ext2 partition I'll run iPXE from when booted into u-boot.

Configuring dnsmasq for BOOTP/DHCP

Now that all the necessary pieces are setup/installed on the devkit, the last step is to run dnsmasq on a host to provide BOOTP service to the devkit. This should be good enough for our purposes:

$ export workdir=/path/to/some/dir

## set to the network interface that is on the same physical LAN as the devkit that dnsmasq will bind to
$ export iface=eth0

## needs to run as root since it binds to ports < 1000
$ doas dnsmasq \
    --port=0 \
    --dhcp-hostsfile="$workdir"/hosts.dhcp \
    --dhcp-optsfile="$workdir"/options.dhcp \
    --dhcp-leasefile="$workdir"/dnsmasq.leases \
    --dhcp-boot=grubnetaa64.efi \
    --dhcp-range=10.42.0.10,10.42.0.100 \
    --dhcp-script=/bin/echo \
    --enable-tftp="$iface" \
    --tftp-root="$workdir"/tftp \
    --log-queries=extra \
    --conf-file=/dev/null \
    --log-debug \
    --no-daemon \
    --interface="$iface"

Note that the boot option sent to the client is grubnetaa64.efi. This is a binary I pulled from some Debian build of grub2 for aarch64, since it was annoying to have to build grub myself just for a quick smoke test.

Grub isn't necessary for booting the Linux kernel, but it is a small application that serves as a good test to make sure that u-boot, iPXE, and dnsmasq are happy.

If you're like me and run firewalls everywhere, you'll need to punch some holes in it for bootp / tftp to work.

Once dnsmasq is started, the devkit is reset and the u-boot script to run iPXE is executed:

Hit any key to stop autoboot:  0
u-boot=> env set boot_scripts ipxe.scr
u-boot=> boot
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /ipxe.scr
294 bytes read in 1 ms (287.1 KiB/s)

In Part 2, I'll cover booting the Linux kernel... Stay tuned!

Using ASNs and nftables to block connections

Blocking Facebook, and similarly-toxic sites/services, is a common theme amongst those who value privacy. Facebook goes to great lengths to track everyone, regardless of whether or not they have an account or use anything they "generously" offer to the public. Previously I had a long, long list of domains that Facebook owned, and set up unbound (the DNS resolver I run) to deny lookups to those domains. This was a classic game of cat & mouse, as Facebook would frequently acquire new domains and it was basically impossible to keep up.

Enter autonomous system numbers (ASN), which are unique identifiers that the IANA assigns to owners of public IP blocks. Using as ASN, it's possible to look up every IP "owned" by the thing the ASN was given to. Once you have every IP, it's trivial to generate a firewall rule (using nftables, at least) to block connections to them. You can evidently even get ASNs for entire ISPs (and therefore, effectively, [some] entire countries!)

I have done this in the script below, ASNs can be set to include others as well, but I have left the two ASNs for Facebook as a convenience to the reader :D

#!/bin/sh

set -euf

# facebook ASNs
ASNs="AS32934 AS11917"

get_asn_ips() {
        asn="$1"
        whois -h whois.radb.net -- -i origin "$asn" |  awk '/route:/ {printf("\t\t%s,", $2)}'
}

asn_ips=

for a in $ASNs; do
        asn_ips=$(printf "%s%s" $asn_ips $(get_asn_ips $a))
done

cat  <<EOF > /etc/nftables.d/50-nft_asn_block.nft
#!/usr/sbin/nft -f
table inet filter {
    set asn_blocked_addresses {
        type ipv4_addr
        flags interval
        elements = {
            $asn_ips
        }
        auto-merge
    }
    chain output {
        meta nfproto ipv4 ip daddr @asn_blocked_addresses log prefix "BLOCKED BY NFT_ASN_BLOCK: " drop;
    }
}
EOF

I have this set to run as a cron job every week, which might be too often (don't forget to reload nftables), but it works fine ¯\_ (ツ)_/¯

There are various ways to find an ASN, some searches allow you to specify the company/organization name, but the most common seem to do lookups based on a given IP address. I won't link to any here, because it's easy to find them using your favorite search engine.