Hi everybody,
like many other people that use RPIs I’ve become weary of having to replace my SD cards frequently.
I recently spent some days migrating a Rhasspy satellite to network boot and thought why not share my findings, especially for Rhasspy because this requires an additional tweak (because of Docker).
Should you choose to accept this mission - make a backup of your SD card in case something doesn’t work. Also read this tutorial till the very end before starting.
Requirements:
- some Linux experience, this tutorial isn’t extremely detailed
- a cabled network connection - wifi won’t work
- an NFS server, if you have a NAS it will probably support this
- an iSCSI server (“target”), if you have a NAS it MIGHT support this, check it
Having a potent NAS at my disposal already I started migrating my RPIs to network boot. There is a way to get rid of the SD card entirely, but that’s not what I did. In my case the PI is still getting the boot image from the SD card, but that’s only a read operation. Afterwards it’ll mount an NFS share as rootfs which then is badgered with write operations rather than the SD card.
1. The NFS part
Exporting an NFS share from my NAS to the PI was quite easy. I’m not going to cover this here.
I took the NFS steps from here: https://keyboardinterrupt.org/save-your-sd-cards-raspberry-pi-on-a-network-file-system/
Backup your original /boot/cmdline.txt in case you want to use the SD card again later:
cp /boot/cmdline.txt /home/pi/
Mount the previously exported NFS share to some temporary path, e.g. /mnt/nfs
Now sync your root fs to it:
rsync -Phax --numeric-ids / /mnt/nfs/
Now edit your /boot/cmdline.txt to look similar to this:
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=<YOUR_NFS_SERVER_IP>:<NFS_SHARE_NAME> ip=<yourRPIsIPaddress>::<NETWORK_GATEWAY>:255.255.255.0:<YOUR_PI_HOSTNAME>:eth0:off elevator=deadline
Reboot. Your PI should come up soon. If you have a look with the mount command you should see that your rootfs is actually mounted from the NFS server, not from the SD card.
Cool, everything works now, doesn’t it? …Oh wait, docker won’t come up. Why’s that?
2. The iSCSI part
While the Docker service is running you’ll find some “overlay” mount moint. Unfortunately this won’t work when running on top of an NFS filesystem. That’s why we now need to use iSCSI in parallel (to NFS) to have something the OS considers a real disk even though it it may come the same physical way as the NFS share does.
Before I repeat things that can be found in other manuals already - just have a look at this post:
Alternatively if you want an extremely extremely short version have a look here: https://community.linuxmint.com/tutorial/view/1446
The required steps can be boiled down to
- attaching the iSCSI disk
- option A: create a partition with fdisk, however I preferred
- option B: create a volume group and a logical volume to be more flexible in the future
- use
mkfsto create a file system on the partition or the LV, depending on which option you chose. - do NOT modify fstab, just yet
You can find the initiator ID of your RPI in
/etc/iscsi/initiatorname.iscsi
You need this id in order to authorize it on your iSCSI server.
Now mount that partition or LV to some temporary path like e.g. /mnt/iscsi
Sync the docker directory to it
rsync -Phax --numeric-ids /var/lib/docker/ /mnt/iscsi/
Now back to the fstab:
Before modying the fstab I would firmly recommend to simply reboot your PI first. In my case the disk isn’t added fast enough which causes the entire boot process to block. In case this happens to you: Because your rootfs is already on the NFS server you can simply use any other NFS capable system or the NAS itself to go there and modify the fstab that RPI is using.
After the reboot use fdisk -l to check if the iSCSI disk is really automatically attached to your system after the reboot.
If it isn’t, you’ll need to modify files in /etc/iscsi
I had to modify the general iSCSI config file as well as the file related to the disk underneath the nodes directory. The are entries similar to “startup type” which you need to change from “manual” to “automatic”.
If the disk is then finally automatically attached to the RPI after a reboot you CAN try to add an entry to fstab like
/dev/mapper/datavg-lvdocker /var/lib/docker ext4 defaults 0 0
However again in my personal case this caused the boot process to block. Then you can instead have it mounted by autofs (instead of fstab):
sudo apt install -y autofs
Edit the file /etc/auto.master.d/docker.autofs
Content:
/- /etc/auto_docker.direct
File /etc/auto_docker.direct then needs to have this content (adjust the disk name of course):
/var/lib/docker -fstype=auto :/dev/mapper/datavg-lvdocker
Now stop docker and rename its storage directory:
sudo systemctl stop docker.service
sudo mv /var/lib/docker /var/lib/docker-old
Restart autofs to mount the new storage:
sudo systemctl restart autofs
Give the existing sync a refresh:
rsync -Phax --numeric-ids /var/lib/docker-old/ /var/lib/docker
And start docker afterwards:
sudo systemctl start docker.service
You should be good to go now.
One might alternatively store the entire RPI OS on an iSCSI disk without going the extra mile of using NFS. That’s a valid approach, but I still prefer the one with NFS.
When everything is confirmed to be working you can cleanup the old docker directory:
rm -R /var/lib/docker-old
