Network booting an aarch64 SBC with u-boot and iPXE

I recently started trying to figure out network booting for aarch64 single board computers (SBC), such as the Raspberry Pi, for a new CI I've been helping out with at my Igalia day job. For one reason or another, I never participated in the Raspberry Pi "fad" (maybe because they use Broadcom chips, which are [or were] notoriously unfriendly on Linux? I don't recall why... But I digress...)

But, I do have a quite capable aarch64 SBC just laying around, literally collecting dust... the Purism Librem 5 DevKit!

sweeping off the devkit for recommissioning with a tiny broom

While not exactly a Raspberry Pi, I believe many of the concepts pre-Linux boot are similar and this should serve as a decent replacement until the Great Chip Shortage of 2020-???? is over and those things are available for purchase again.

The general idea is that u-boot will execute iPXE, which will be responsible establishing a network connection and booting whatever the DHCP server on the other end tells it to boot. The end goal is to have it load/boot the Linux kernel and an initfs based on boot2container.

This is the first in a series of posts to get there. The focus of this initial post is building/setting up iPXE, the DHCP server, and doing a test boot from u-boot.

Building iPXE and configuring the devkit

The first step is to build iPXE, since I want to embed a script for it to run automatically on boot. I did the compilation on the devkit, since iPXE is a relatively small program and it didn't take too long to compile on this CPU:

$ git clone git://git.ipxe.org/ipxe.git
$ cd ipxe/src

## needed so that ipxe doesn't lock up if you want to C-b to enter the cmdline
$ cat << EOF > config/local/nap.h
#undef NAP_EFIX86
#undef NAP_EFIARM
#define NAP_NULL
EOF

## and create a simple ipxe script that will be executed when ipxe runs:
$ cat << EOF > ipxescript
#!ipxe

:retry_dhcp
echo Acquiring an IP
dhcp || goto retry_dhcp

echo Got the IP: $${netX/ip} / $${netX/netmask}

:retry_boot
echo Booting from DHCP...
autoboot || goto retry_boot
EOF

## build/install:
$ make bin-arm64-efi/snp.efi -j4 EMBED=ipxescript
$ doas cp bin-arm64-efi/snp.efi /boot/ipxe.efi

Loading things in u-boot is quite tedious, since you have to specify memory addresses to load files into, and the correct *load command to read files into memory. I already have an existing install of postmarketOS on my devkit, so I used the /boot partition (formatted as ext2) as a home for the iPXE binary. I created the following U-boot helper script for loading iPXE, since typing all of these in becomes tiresome very quickly:

$ cat << EOF > /tmp/ipxe 
echo ===== Loading iPXE =====
ext2load mmc 0:1 $kernel_addr_r ipxe.efi
ext2load mmc 0:1 $fdt_addr_r imx8mq-librem5-devkit.dtb
fdt addr $fdt_addr_r
fdt resize
echo ===== Running iPXE =====
bootefi $kernel_addr_r $fdt_addr_r
EOF

I'm not entirely sure if we need to specify/load the dtb, but it doesn't seem to hurt! Also note that this u-boot script is using bootefi to load the iPXE app. That'll be important later on when we try to boot a kernel.

The u-boot script must be compiled before u-boot can execute it:

$ mkimage -A arm64 -C none -O linux -T script -d /tmp/ipxe /tmp/ipxe.scr
$ doas cp /tmp/ipxe.scr /boot

In that last step, I copy it to /boot since I'm performing these steps on the devkit, and /boot is the ext2 partition I'll run iPXE from when booted into u-boot.

Configuring dnsmasq for BOOTP/DHCP

Now that all the necessary pieces are setup/installed on the devkit, the last step is to run dnsmasq on a host to provide BOOTP service to the devkit. This should be good enough for our purposes:

$ export workdir=/path/to/some/dir

## set to the network interface that is on the same physical LAN as the devkit that dnsmasq will bind to
$ export iface=eth0

## needs to run as root since it binds to ports < 1000
$ doas dnsmasq \
    --port=0 \
    --dhcp-hostsfile="$workdir"/hosts.dhcp \
    --dhcp-optsfile="$workdir"/options.dhcp \
    --dhcp-leasefile="$workdir"/dnsmasq.leases \
    --dhcp-boot=grubnetaa64.efi \
    --dhcp-range=10.42.0.10,10.42.0.100 \
    --dhcp-script=/bin/echo \
    --enable-tftp="$iface" \
    --tftp-root="$workdir"/tftp \
    --log-queries=extra \
    --conf-file=/dev/null \
    --log-debug \
    --no-daemon \
    --interface="$iface"

Note that the boot option sent to the client is grubnetaa64.efi. This is a binary I pulled from some Debian build of grub2 for aarch64, since it was annoying to have to build grub myself just for a quick smoke test.

Grub isn't necessary for booting the Linux kernel, but it is a small application that serves as a good test to make sure that u-boot, iPXE, and dnsmasq are happy.

If you're like me and run firewalls everywhere, you'll need to punch some holes in it for bootp / tftp to work.

Once dnsmasq is started, the devkit is reset and the u-boot script to run iPXE is executed:

Hit any key to stop autoboot:  0
u-boot=> env set boot_scripts ipxe.scr
u-boot=> boot
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /ipxe.scr
294 bytes read in 1 ms (287.1 KiB/s)

PXE booting to grub

In Part 2, I'll cover booting the Linux kernel... Stay tuned!