Provisioning Raspberry under MAAS (Metal as a service)

Clients setup

Prepare RPI UEFI firmware files

1. Download the UEFI firmware from https://github.com/pftf/RPi4/releases/download/v1.30/RPi4_UEFI_Firmware_v1.30.zip
2. Extract it into the fat32 formatted SD Card or boot partition of RPI SD card.
3. Let RPI boot from it, and change setting to disable 3GB ram limitation by:
   1. Going to: Device Manager → Raspberry Pi Configuration → Advanced Settings
   2. Disabled 3GB ram limitation and save.
4. Change boot order setting:
   1. Move IPV4 PXE Boot to the top and save. (So that RPI will always boot from MAAS first even after deployed, so we don't need to format storage when release machines)
5. The firmware files are ready to be copied into the netboot server. To easily copy it into the server, we can zip all the files (make sure to include RPI_EFI.fd, it saves the configuration changed that we have made) before transfer into the server.

Prepare Raspbian root and boot files (For Raspbian netboot)

We will copy root and boot files from RPI into our NFS server.

1. Create tar files from root and boot directory of RPI.
   1. cd /
   2. tar -cvpzf root.tar.gz --exclude=/root.tar.gz --one-file-system /
   3. tar -cvpzf boot.tar.gz /boot
2. Copy the tar files into the NFS server

Server setup

Install MAAS

1. Install MAAS with snap

   sudo snap install --channel=3.0/stable maas

2. Install maas-test-db (it is a preconfigured Pogresql for MAAS)

   sudo snap install maas-test-db

3. Initialize MAAS as region+rack controller

   sudo maas init region+rack --database-uri maas-test-db:///

4. Create an admin account with

   sudo maas createadmin

Configure netboot image

1. Login to MAAS with a previously created account.
2. Go to Image and select Ubuntu 20.04LTS and ARM64 Architecture.
3. Click saves and waits for images to finish sync into the controller.
4. Then toggle automatically image sync to disable it (because MAAS will change tftp root directory after images synced).

Set minimum kernel for commission

1. Go to Settings -> Configuration -> Commissioning, and select:

   • focal(hwe-20.04-edge) as default minimum kernel verion.

---

Notes: If there is no option to select kernel, it means the images are not yet synced. To use MAAS CLI to set default commission kernel:

# save api key
sudo maas apikey --username=admin > api_key
# login with MAAS CLI
maas login admin http://192.168.1.2:5240/MAAS - < api_key
# set kernel with MAAS CLI
maas admin maas set-config name=default_min_hwe_kernel value=hwe-20.04-edge

Configure DHCP server to provide TFTP IP to client

1. Go to settings -> DHCP Snippets, and add a new snippet with this option:

   option tftp-server-name "192.168.1.2";

Serve UEFI firmware with TFTP server

Because MAAS already provides a TFTP server, we just need to copy the UEFI boot files onto the TFTP root directory.

1. Create rpi-uefi directory in /var/snap/maas/common/maas/boot-resources

   mkdir /var/snap/maas/common/maas/boot-resources/rpi-uefi

2. Unzip our UEFI boot files into the directory.

   unzip uefi-1.3.zip -d /var/snap/maas/common/maas/boot-resources/rpi-uefi

3. Because we want to serve different RPI clients with different boot files, we will create a dir symlink with RPI serial number to the boot directory.

   ln -s /var/snap/maas/common/maas/boot-resources/rpi-uefi /var/snap/maas/common/maas/boot-resources/current/${SERIAL}

---

Notes: The default TFTP_PREFIX bootloader configuration option in RPI is to use the serial number as a prefix.

Netboot the RPI

Netboot the RPI, it will be shown as a new machine on MAAS. We can then select the device and commission it.

For Raspbian netboot

Install NFS Server

apt install nfs-kernel-server

Setup NSF root and boot directory

1. mkdir -p /srv/nfs/${SERIAL}
2. tar xvzf root.tar.gz --directory /srv/nfs/${SERIAL}
3. tar xvzf boot.tar.gz --directory /srv/nfs/${SERIAL}

---

Note: we will have the following directory:

1. /srv/nfs/${SERIAL}: root directory for RPI with ${SERIAL} number
2. /srv/nfs/${SERIAL}/boot: boot directory for RPI with ${SERIAL} number

And we get root.tar.gz and boot.tar.gz from the RPI SD card

Change cmdline.txt so that RPI will mount its root directory from NFS server after netboot

1. echo "console=serial0,115200 console=tty root=/dev/nfs nfsroot=${NFS_SERVERIP}:/srv/nfs/${SERIAL},vers=4.1,proto=tcp rw ip=dhcp rootwait elevator=deadline cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory" > /srv/nfs/${SERIAL}/boot/cmdline.txt

Change fstab file so that RPI won't mount the SD card

1. echo "proc /proc proc defaults 0 0" > /srv/nfs/${SERIAL}/etc/fstab

Update NFS rules

We edit the /etc/exports configuration file to add NFS directory and run "exportfs -ra" to reload the configuration

1. echo "/srv/nfs/${SERIAL} *(rw,sync,no_root_squash,no_subtree_check)" >> /etc/exports
2. exportfs -ra

Mount client boot directory onto TFTP root directory

1. mkdir -p /var/snap/maas/common/maas/boot-resources/current/${SERIAL}
2. echo "/srv/nfs/${SERIAL}/boot /var/snap/maas/common/maas/boot-resources/current/${SERIAL} none defaults,bind 0 0" >> /etc/fstab
3. mount -a

---

Now the server is ready for netboot.

Setup scripts

Raspbian netboot for 4 RPI devices

#!/bin/sh

var1="82824dee 4b3be06c 3b045773 fd09fc29" #serial
var2="red yellow black green" #hostname
SERVERIP=192.168.1.2

set -- $var2
for SERIAL in $var1
do
HOSTNAME=$1

mkdir -p /srv/nfs/${SERIAL}
tar xvzf root.tar.gz --directory /srv/nfs/${SERIAL}
tar xvzf boot.tar.gz --directory /srv/nfs/${SERIAL}

echo "/srv/nfs/${SERIAL} *(rw,sync,no_root_squash,no_subtree_check)" \
>> /etc/exports
exportfs -ra

mkdir -p /var/snap/maas/common/maas/boot-resources/current/${SERIAL}
echo "/srv/nfs/${SERIAL}/boot /var/snap/maas/common/maas/boot-resources/current/${SERIAL} none defaults,bind 0 0" >> /etc/fstab
mount -a

sudo tee /srv/nfs/${SERIAL}/etc/fstab > /dev/null <<EOT
proc            /proc           proc    defaults        0       0
#/dev/mmcblk0p1       /var/lib/rancher     ext4    defaults    0 0 #for k3s
EOT

echo "console=serial0,115200 console=tty root=/dev/nfs \
nfsroot=${SERVERIP}:/srv/nfs/${SERIAL},vers=4.1,proto=tcp rw ip=dhcp rootwait \
elevator=deadline cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory\
" | sudo tee /srv/nfs/${SERIAL}/boot/cmdline.txt > /dev/null

OLDHOSTNAME="$(cat /srv/nfs/${SERIAL}/etc/hostname)"
sed -i "s/${OLDHOSTNAME}/${HOSTNAME}/g" /srv/nfs/${SERIAL}/etc/hosts
echo ${HOSTNAME} > /srv/nfs/${SERIAL}/etc/hostname

shift
done

For Ubuntu OS

Different points between Ubuntu OS and Rasbian OS:

1. Boot firmware for Ubuntu OS are stored in /boot/firmware instead of /boot
2. We need to specify the mount points for root and boot in the fstab file of the Ubuntu OS.
3. For Ubuntu 20.04 we need to update boot/config.txt with the following:

[pi4]
kernel=vmlinuz
max_framebuffers=2
initramfs initrd.img followkernel

The script to setup cluster

#!/bin/sh

var1="82824dee 4b3be06c 3b045773 fd09fc29" #serial
var2="red yellow black green" #hostname
SERVERIP=192.168.1.2

set -- $var2
for SERIAL in $var1
do
HOSTNAME=$1

mkdir -p /srv/nfs/${SERIAL}
tar xvzf root.tar.gz --directory /srv/nfs/${SERIAL}
tar xvzf boot.tar.gz --directory /srv/nfs/${SERIAL}

echo "/srv/nfs/${SERIAL} *(rw,sync,no_root_squash,no_subtree_check)" \
>> /etc/exports
exportfs -ra

SYMLINK_TFTP_ROOT="/var/snap/maas/common/maas/boot-resources/current"
TFTP_ROOT=$(cd -P "$SYMLINK_TFTP_ROOT" && pwd)

mkdir -p ${TFTP_ROOT}/${SERIAL}
echo "/srv/nfs/${SERIAL}/boot ${TFTP_ROOT}/${SERIAL} none defaults,bind 0 0" >> /etc/fstab
mount ${TFTP_ROOT}/${SERIAL}

ln -s ${TFTP_ROOT}/${SERIAL}/firmware/bcm2711-rpi-4-b.dtb ${TFTP_ROOT}/${SERIAL}/bcm2711-rpi-4-b.dtb
ln -s ${TFTP_ROOT}/${SERIAL}/firmware/start4.elf ${TFTP_ROOT}/${SERIAL}/start4.elf
ln -s ${TFTP_ROOT}/${SERIAL}/firmware/fixup4.dat ${TFTP_ROOT}/${SERIAL}/fixup4.dat
ln -s ${TFTP_ROOT}/${SERIAL}/firmware/config.txt ${TFTP_ROOT}/${SERIAL}/config.txt
chmod 777 ${TFTP_ROOT}/${SERIAL}/vmlinuz

sudo tee /srv/nfs/${SERIAL}/etc/fstab > /dev/null <<EOT
proc            /proc           proc    defaults        0       0
${SERVERIP}:/srv/nfs/${SERIAL} /       nfs4     defaults,rw,nolock             0       0 # data to be shared to server
${SERVERIP}:/srv/nfs/${SERIAL}/boot/firmware /boot/firmware       nfs4     defaults,rw,nolock             0       1 # data to be shared to server
#/dev/mmcblk0p1       /var/lib/rancher     ext4    defaults    0 0 #for k3s
EOT

echo "net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 \
nfsrootdebug rootfstype=nfs4 root=/dev/nfs nfsroot=${SERVERIP}:/srv/nfs/${SERIAL} \
rw elevator=deadline rootwait fixrtc cgroup_enable=cpuset cgroup_memory=1 \
cgroup_enable=memory" | sudo tee /srv/nfs/${SERIAL}/boot/cmdline.txt > /dev/null

OLDHOSTNAME="$(cat /srv/nfs/${SERIAL}/etc/hostname)"
sed -i "s/${OLDHOSTNAME}/${HOSTNAME}/g" /srv/nfs/${SERIAL}/etc/hosts
echo ${HOSTNAME} > /srv/nfs/${SERIAL}/etc/hostname

shift
done

Script to recreate symlink/mount into TFTP root (If MAAS has synced images and removed them)

Symlink UEFI firmware

SERIALS="379a846d 65d02964 349b6628 77274846"
for i in $SERIALS
do
  ln -s /var/snap/maas/common/maas/boot-resources/rpi-uefi /var/snap/maas/common/maas/boot-resources/current/${i}
done

Mount client boot directory

var1="82824dee 4b3be06c 3b045773 fd09fc29"
for SERIAL in $var1
do
mkdir -p /var/snap/maas/common/maas/boot-resources/current/${SERIAL}
done
mount -a

Use Ansible to manage clients

Use Ansible to manager cluster