Ian Bebbington - Network Booting Many Raspberry Pis

Intro

This is just a short post - mostly for my own benefit - on how to network boot multiple Raspberry Pis from an x86 Linux Server. While this has been covered many times in other posts none of them worked for me "out of the box". Here's what does.

Infrastructure

I will be using the following components

Hyper-V Virtual Machine running Raspberry Pi Desktop (aka Debian Buster with Raspberry Pi Desktop) downloaded from here as the network boot server.
Multiple Raspberry Pi 3B+ (the non-plus Raspberry Pi 3B requires additional steps) as network boot clients

Requirements

The official Raspberry Pi Network Boot instructions assume you're using a Raspberry Pi as the network boot server and can therefore "copy" a Raspbian installation from an SD Card that has been installed on the network boot client Raspberry Pi. As I want to use an Linux server - running in a virtualised environment no less - I will be using additional steps from Hackaday's excellent article on Network Booting The Pi 4.

Additionally, while each of the network boot client Raspberry Pi's will be running Raspbian Buster Lite, they will be used for different purposes so must run a unique Raspbian installation.

Steps

Network Boot Server

Create a virtual machine and install Debian Buster with Raspberry Pi Desktop. I will not cover instructions for doing this here as there are many virtualisation engines and instructions for each would be different; suffice to say I used a Gen 1 Hyper-V instance on Windows Server 2016 with 4 virtual cores, 8Gb of RAM and 64Gb of disk-space. Furthermore, after installation, I enabled SSH and used SSH to execute the following.

Install required software using the following command:

sudo apt-get install unzip kpartx dnsmasq nfs-kernel-server

Make a directory to contain the first network boot client image:
```
sudo mkdir -p /nfs/raspi1
```

Download and unzip the latest Raspbian Buster Lite image:

wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2019-09-30/2019-09-26-raspbian-buster-lite.zip
unzip 2019-09-26-raspbian-buster-lite.zip

Mount the Raspbian Buster Lite image to known locations:

sudo kpartx -a -v 2019-09-26-raspbian-buster.img
mkdir rootmnt
mkdir bootmnt
sudo mount /dev/mapper/loop0p2 rootmnt/
sudo mount /dev/mapper/loop0p1 bootmnt/

Copy the Raspbian Buster Lite image to the network boot client image directory created above:
```
sudo cp -a rootmnt/* /nfs/raspi1/
sudo cp -a bootmnt/* /nfs/raspi1/boot/
```
Ensure the network boot client image doesn't attempt to look for filesystems on the SD Card:
```
sudo sed -i /UUID/d /nfs/raspi1/etc/fstab
```
Replace the boot command in the network boot client image to boot from a network share. Ensure you replace [IP Address] with the IP address of your network boot server (note the modprobe.blacklist is required to successfully boot the Raspberry Pi 3B+ as described here):
```
echo "console=serial0,115200 console=tty root=/dev/nfs nfsroot=[IP Address]:/nfs/raspi1,vers=3 rw ip=dhcp rootwait elevator=deadline modprobe.blacklist=bcm2835_v4l2" | sudo tee /nfs/raspi1/boot/cmdline.txt
```
Enable SSH in the network boot client image:
```
sudo touch /nfs/raspi1/boot/ssh
```

Create a network share containing the network boot client image:

echo "/nfs/raspi1 *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports

Create a TrivialFTP folder containing boot code for all network boot clients

sudo mkdir /tftpboot
sudo cp /nfs/raspi1/boot/bootcode.bin /tftpboot/bootcode.bin
sudo chmod 777 /tftpboot

Enable and restart rpcbind and nfs-kernel-server services:

sudo systemctl enable rpcbind
sudo systemctl enable nfs-kernel-server
sudo systemctl restart rpcbind
sudo systemctl restart nfs-kernel-server

Reconfigure dnsmasq to server TFTP files only to Raspberry Pi instances as described here:

We need to add our settings to the dnsmasq config file, which is where most of the magic happens. Let’s talk about that “proxy” setting. What we’re asking dnsmasq to do is watch for DHCP requests, and rather than respond to those requests directly, wait for the primary DHCP server to assign an IP address. If dnsmasq sees a request for PXE information, it will send additional information to inform the PXE-capable device of the PXE server information. The upside is that this approach lets us support PXE booting without modifying the primary DHCP server.

Be sure to replace [Broadcast Address] with the broadcast address for your network (use ip address | grep brd to find it):
```
echo 'dhcp-range=[Broadcast Address],proxy' | sudo tee -a /etc/dnsmasq.conf
echo 'log-dhcp' | sudo tee -a /etc/dnsmasq.conf
echo 'enable-tftp' | sudo tee -a /etc/dnsmasq.conf
echo 'tftp-root=/tftpboot' | sudo tee -a /etc/dnsmasq.conf
echo 'pxe-service=0,"Raspberry Pi Boot"' | sudo tee -a /etc/dnsmasq.conf
```

Enable and restart the dnsmasq service:

sudo systemctl enable dnsmasq
sudo systemctl restart dnsmasq

Find the serial number of the first network boot client:
1. Tail daemon.log to :
```
sudo tail -f /var/log/daemon.log
```
2. Plug in a network cable and power cable to the first network boot client. After 10-30 seconds you should see output like this in the daemon.log:
  
  dnsmasq-dhcp[9460]: 653460281 available DHCP subnet: 192.168.1.255/255.255.255.0
  dnsmasq-dhcp[9460]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
  dnsmasq-dhcp[9460]: 653460281 PXE(eth0) b8:27:eb:ec:46:57 proxy
  dnsmasq-dhcp[9460]: 653460281 tags: eth0
  dnsmasq-dhcp[9460]: 653460281 broadcast response
  dnsmasq-dhcp[9460]: 653460281 sent size: 1 option: 53 message-type 2
  dnsmasq-dhcp[9460]: 653460281 sent size: 4 option: 54 server-identifier 192.168.1.102
  dnsmasq-dhcp[9460]: 653460281 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
  dnsmasq-dhcp[9460]: 653460281 sent size: 17 option: 97 client-machine-id 00:44:44:44:44:44:44:44:44:44:44:44:44:44...
  dnsmasq-dhcp[9460]: 653460281 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
  dnsmasq-tftp[9460]: file /tftpboot/bootsig.bin not found
  dnsmasq-tftp[9460]: sent /tftpboot/bootcode.bin to 192.168.1.112
  dnsmasq-dhcp[9460]: 653460281 available DHCP subnet: 192.168.1.255/255.255.255.0
  dnsmasq-dhcp[9460]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
  dnsmasq-dhcp[9460]: 653460281 PXE(eth0) b8:27:eb:ec:46:57 proxy
  dnsmasq-dhcp[9460]: 653460281 tags: eth0
  dnsmasq-dhcp[9460]: 653460281 broadcast response
  dnsmasq-dhcp[9460]: 653460281 sent size: 1 option: 53 message-type 2
  dnsmasq-dhcp[9460]: 653460281 sent size: 4 option: 54 server-identifier 192.168.1.102
  dnsmasq-dhcp[9460]: 653460281 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
  dnsmasq-dhcp[9460]: 653460281 sent size: 17 option: 97 client-machine-id 00:57:46:ec:fe:57:46:ec:fe:57:46:ec:fe:57...
  dnsmasq-dhcp[9460]: 653460281 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
  dnsmasq-tftp[9460]: file /tftpboot/feec4657/start.elf not found
  dnsmasq-tftp[9460]: file /tftpboot/autoboot.txt not found
  dnsmasq-tftp[9460]: file /tftpboot/config.txt not found
  dnsmasq-tftp[9460]: file /tftpboot/recovery.elf not found
  dnsmasq-tftp[9460]: file /tftpboot/start.elf not found
  dnsmasq-tftp[9460]: file /tftpboot/fixup.dat not found
  
  This shows that the first network boot client has successfully made requests to the TFTP service on the network boot service.
3. Notice the dnsmasq-tftp[9460]: file /tftpboot/feec4657/start.elf not found line. The 'feec4657' value is the serial number of the network boot client (it will obviously be different for you) and allows you to use different images for different devices.
Create a directory for the first network boot client in the /tftpboot directory (remembering to replace [SerialNumber] with the value you found above):
```
sudo mkdir /tftpboot/[SerialNumber]
```
Copy the boot directory from the /nfs/raspi1 directory to the new directory in /tftpboot:
```
sudo cp -a /nfs/raspi1/boot/* /tftpboot/[SerialNumber]/
```
Reconnect the power to the network boot client and it should now boot successfully. If you use sudo tail -f /var/log/daemon.log again you should see something like the following:

dnsmasq-dhcp[9460]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-dhcp[9460]: 653460281 PXE(eth0) b8:27:eb:ec:46:57 proxy
dnsmasq-dhcp[9460]: 653460281 tags: eth0
dnsmasq-dhcp[9460]: 653460281 broadcast response
dnsmasq-dhcp[9460]: 653460281 sent size: 1 option: 53 message-type 2
dnsmasq-dhcp[9460]: 653460281 sent size: 4 option: 54 server-identifier 192.168.1.102
dnsmasq-dhcp[9460]: 653460281 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[9460]: 653460281 sent size: 17 option: 97 client-machine-id 00:44:44:44:44:44:44:44:44:44:44:44:44:44...
dnsmasq-dhcp[9460]: 653460281 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
dnsmasq-tftp[9460]: file /tftpboot/bootsig.bin not found
dnsmasq-tftp[9460]: sent /tftpboot/bootcode.bin to 192.168.1.112
dnsmasq-dhcp[9460]: 653460281 available DHCP subnet: 192.168.1.255/255.255.255.0
dnsmasq-dhcp[9460]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-dhcp[9460]: 653460281 PXE(eth0) b8:27:eb:ec:46:57 proxy
dnsmasq-dhcp[9460]: 653460281 tags: eth0
dnsmasq-dhcp[9460]: 653460281 broadcast response
dnsmasq-dhcp[9460]: 653460281 sent size: 1 option: 53 message-type 2
dnsmasq-dhcp[9460]: 653460281 sent size: 4 option: 54 server-identifier 192.168.1.102
dnsmasq-dhcp[9460]: 653460281 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[9460]: 653460281 sent size: 17 option: 97 client-machine-id 00:57:46:ec:fe:57:46:ec:fe:57:46:ec:fe:57...
dnsmasq-dhcp[9460]: 653460281 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
dnsmasq-tftp[9460]: file /tftpboot/feec4657/autoboot.txt not found
dnsmasq-tftp[9460]: error 0 Early terminate received from 192.168.1.112
dnsmasq-tftp[9460]: failed sending /tftpboot/feec4657/start.elf to 192.168.1.112
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/config.txt to 192.168.1.112
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery.elf not found
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/start.elf to 192.168.1.112
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/fixup.dat to 192.168.1.112
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery.elf not found
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/config.txt to 192.168.1.112
dnsmasq-tftp[9460]: file /tftpboot/feec4657/dt-blob.bin not found
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery.elf not found
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/config.txt to 192.168.1.112
dnsmasq-tftp[9460]: file /tftpboot/feec4657/bootcfg.txt not found
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/cmdline.txt to 192.168.1.112
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/bcm2710-rpi-3-b.dtb to 192.168.1.112
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/config.txt to 192.168.1.112
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery8.img not found
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery8-32.img not found
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery7.img not found
dnsmasq-tftp[9460]: file /tftpboot/feec4657/recovery.img not found
dnsmasq-tftp[9460]: file /tftpboot/feec4657/kernel8-32.img not found
dnsmasq-tftp[9460]: error 0 Early terminate received from 192.168.1.112
dnsmasq-tftp[9460]: failed sending /tftpboot/feec4657/kernel8.img to 192.168.1.112
dnsmasq-tftp[9460]: error 0 Early terminate received from 192.168.1.112
dnsmasq-tftp[9460]: failed sending /tftpboot/feec4657/kernel7.img to 192.168.1.112
dnsmasq-tftp[9460]: file /tftpboot/feec4657/armstub8-32.bin not found
dnsmasq-tftp[9460]: sent /tftpboot/feec4657/kernel7.img to 192.168.1.112
dnsmasq-dhcp[9460]: 1754635714 available DHCP subnet: 192.168.1.255/255.255.255.0
dnsmasq-dhcp[9460]: 1754635714 available DHCP subnet: 192.168.1.255/255.255.255.0
rpc.mountd[26471]: authenticated mount request from 192.168.1.112:843 for /nfs/raspi1 (/nfs/raspi1)

Here we can see the following:
- sent /tftpboot/bootcode.bin to 192.168.1.112 -> We successfully sent the bootcode.bin to the network boot client
- sent /tftpboot/feec4657/[FILENAME] to 192.168.1.112 -> We successfully sent boot files from the device specific /tftpboot directory to the network boot client
- authenticated mount request from 192.168.1.112:843 for /nfs/raspi1 (/nfs/raspi1) -> the network boot client mounted to the system drive from the nfs share.
You should now be able to ssh into the network boot client using the following (replacing the [IP Address]) with the one you see):
```
ssh pi@[IP Address]
```
Using the default password of 'raspberry'.

Additional Network Boot Clients

To add additional network boot clients, simply repeat steps 3, 6-10, 15-18 replacing all instances of raspi1 with a new name.

Network Booting Many Raspberry Pis

A guide that works for me... mileage may vary