Snapper, rollback, and btrfs

1 partitioning for btrfs

A swap file on a btrfs volume is difficult and complicated to setup. Easier to use a standard swap partition.

So, your partition scheme should be 512 megabyt efi, a few gigabytes swap, and the rest one btrfs partition. Almost all distros support btrfs in manual partitioning. Since you will be using snapper, you need at least thirty gigabytes.

A btrfs file system can go over many partitions on many devices, but this is a bad idea for your root file system, and is apt to get scary complicated and require much ricing.

Since partitions with btrfs files on them are resizable, another solution is to have one partition for the OS, and another for home and opt. This allows you to reinstall linux, while keeping your possibly large data.

2 rollbacks

snapper rollback is confusing, because it does not rollback anything, merely creates the rollback subvolume. You are still in your original environment, not the the environment you wanted to roll back to, and when you reboot, still in the original environment, and grub-btrfs, which allows you to actually boot into the rollback subvolume, is mysterious, because when you boot into rollback subvolume using the grub menu created by bit grub-btrfs, the rollback is not permanent until the grub-btrfs daemon makes it permanent.

After the daemon has worked its magic, by rewriting the two grub.cfg files, then you will always reboot into a rolled back system, until you do yet another rollback.

Debian is confusing and mysterious, because it has two grub.cfg files, one on the efi disk which points to the real grub.cfg, which you just rolled back.

Arch makes things considerably simpler by putting the entire /boot directory on the efi partition, so you only have one grub.cfg, and rolling back has no effect on it. This makes the behavior of rollback more intelligible and unsurprising, for your efi partition is your unmoved mover. Arch also, however, recommends a one gigabyte efi partition, while on debian you can get away with a much smaller efi partition.

When using manual configuration with archinstall, you have to manually ensure the efi partition is mounted as /boot, formatted as fat 32, and marked bootable and esp, and if you get any of those things wrong it will complain there is a problem, but not tell you what the problem is.

Rollback is not backup. It is a restore point, against your operating system getting into bad state, usually as a result of you tinkering with your system, sometimes as a result of bleeding edge software, or mutually incompatible software, from the arch repositories, which contain everything under the sun, including stuff that does not work with slightly different arch configurations. For your hardware physically going down, as for example your hard disk gets corrupted or dies, you also need backup.

2.1 manual snapshotting and manual rollback

First, in order to setup an environment where snapper and grub-btrfs will do what you want them to do and expect them to do, we are going to do a manual snapshot and a manual rollback, and only then should we install them.

Your fstab will contain a subvol= entry for the root mount, but this is always ignored, unless you are doing something clever like booting directly through efi, rather than using the multistage grub boot, though the other options eventually take effect. This merely causes confusion.

Your btrfs /etc/fstab will be setup with the default options, which are not very good.

Edit defaults,subvol=@rootfs to defaults,noatime,discard,autodefrag,compress=zstd,commit=360

Your root subvolume will have lots of stuff in a single subvolume, which is a problem for snapshots, for you only want to snapshot the real OS, not everything, for this is backup against ricing your system to death, not backup against hardware failure.

It will likely also have /home on that single subvolume, which is a problem because you probably want to snapshot your home separately and manually, and restore them separately using cp -a. The home directory is apt to get very large, and have lots of stuff on it that you really do not care about restoring, except in the event of hardware failure, against which snapshots do not protect. They protect against you breaking your software configuration.

Most utilities for managing subvolumes and snapshots expect your file system to follow the subvolume layout prescribed by arch wiki for snapper setup, so this layout is based on that layout, except it excludes a few more directories from snapshotting than it does.

/tmp is almost always on tmpfs, not your root subvolume, /run is similarly always on a special file system. /var/run is usually a symlink to run or contains only a symlink to run and is normally mounted on tmpfs

You probably need to make /home /opt, /var/tmp, /var/cache, /var/spools, and /var/lib/docker into subvolumes

Create a new mount point, /btrfsroot that will always point to the top level ID5 btrfs subvolume. subvolid=5

Create your subvolumes in it with names like @snapshots, @home, @opt, @var_tmp, @var_cache, @var_lib_docker, and @var_spools at the same level as @rootfs with btrfs subvolume create. Note that Arch recommends this flat setup, most software is setup to use it, and snapper-rollback requires extra configuration if /btrfsroot does not exist under this name. If everything is flat then switching does not lead to complications. Which complications may explain why I had so much grief with snapper and grub-btrfs. Arch set the standard, Arch was first to implement btrfs backup and recovery software, therefore everyone should follow the Arch standard.

2.1.1 setting up the subvolumes and fstab

assuming you have already mounted subvolid=5 in /btrfsroot

Subvolid=5 is the real top of the subvolume tree in your system, and we are going to create a flat layout, in which all the subvolumes we create at the top of the tree.

This is arch standard, and all the software for manipulating subvolumes, such as snapper, was developed in arch and for arch, so life gets complicated if you deviate.

ls -hal /btrfsroot should show one subvolume, the one containing your root file system, typically called @, or @rootfs, or something like that.

Create a snapshot of / on /btrfsroot with a name like @rootfs_current with btrfs subvolume snapshot / /btrfsroot/@rootfs_current

Copy the contents of /home /opt, /var/tmp, /var/cache, /var/lib/docker,/var/lib/nerdctl, and /var/spools from @rootfs2 into @home, @opt etcetera on /btrfsroot using cp -a

fn=home
btrfs subvolume create /btrfsroot/@$fn
mkdir -p /btrfsroot/@rootfs_current/$fn
cp -a /btrfsroot/@rootfs_current/$fn/* /btrfsroot/@$fn
rm -r /btrfsroot/@rootfs_current/$fn/*

and similarly for opt. Which will likely generate errors that do not matter because there is likely nothing in opt.

mkdir -p  /btrfsroot/@rootfs_current/.snapshots
btrfs subvolume create /btrfsroot/@snapshots

And then the subdirectories of var

fn=tmp
btrfs subvolume create /btrfsroot/@var_$fn
mkdir -p /btrfsroot/@rootfs_current/var/$fn
cp -a /btrfsroot/@rootfs_current/var/$fn/* /btrfsroot/@var_$fn
rm -r /btrfsroot/@rootfs_current/var/$fn/*

And similarly for cache, spools, and log

Then

mkdir -p /btrfsroot/@rootfs_current/var/lib/docker
rm -r /btrfsroot/@rootfs_current/var/lib/docker/*
btrfs subvolume create /btrfsroot/@var_lib_docker

After doing this for every directory that needs it (if the directory is currently empty or nonexistent, will still work despite a stream of error messages) btrfs subvolume list -t /btrfsroot should show something like this

# btrfs subvolume list -t /btrfsroot
ID      gen     top level       path
--      ---     ---------       ----
256     105     5               @rootfs
257     85      5               @snapshots
258     102     5               @rootfs_current
259     96      5               @home
260     97      5               @opt
261     100     5               @var_tmp
262     101     5               @var_cache
263     101     5               @var_spools
264     101     5               @var_log
265     102     5               @var_lib_docker

Edit /btrfsroot/@rootfs_current/etc/fstab to mount the corresponding subvolumes.

# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system>                                 <mount point>      <type>          <options>                                                                                          <dump>  <pass>
# / was on /dev/sda3 during installation
# grub causes the / subvol to ignored.
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /                  btrfs           defaults,noatime,discard,autodefrag,compress=zstd,commit=360                                        0      0
# /boot/efi was on /dev/sda1 during installation
UUID=A544-7AD5                                  /boot/efi          vfat            umask=0077                                                                                          0      1
# swap was on /dev/sda2 during installation
UUID=cf659755-043d-4bc0-aa20-8c966ee6793f       none               swap            sw                                                                                                  0      0
/dev/sr0                                        /media/cdrom0      udf,iso9660     ro,users,auto,nofail                                                                                0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /btrfsroot         btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvolid=5               0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /.snapshots        btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@snapshots        0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /home              btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@home             0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /opt               btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@opt              0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /var/tmp           btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@var_tmp          0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /var/cache         btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@var_cache        0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /var/spools        btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@var_spools       0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /var/log           btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@var_log          0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /var/lib/docker    btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@var_lib_docker   0      0
UUID=9a0c25cc-9fd1-445a-bd9f-ecfc7fdf16f2       /var/lib/nerdctl   btrfs           rw,relatime,discard,ssd,autodefrag,compress=zstd,commit=300,space_cache=v2,subvol=@var_lib_nerdctl  0      0

If you are subject to sudden power outages (you do have a ups, don’t you) you might want to set commit=30. When you recover from a sudden power down, the file system will likely reflect the last commit. notatime considerably reduces wear and tear on solid state drive, and commit=300 reduces wear and tear on a solid state drive. discard, for cooperating hardware or a virtual machine file system, allows that hardware or virtual machine to manage its own free list, so that it does not have to allocate space for garbage.

Interfere in the grub boot process with 'e' so as to boot into this new @rootfs_current

btrfs subvolume show / # to check that you are in the expected new subvolume.

btrfs subvolume show /
btrfs subvolume show /btrfsroot
btrfs subvolume show /home
btrfs subvolume show /opt
btrfs subvolume show /var/tmp
btrfs subvolume show /var/cache
btrfs subvolume show /var/spools
btrfs subvolume show /var/log
btrfs subvolume show /var/lib/docker

If using systemd, edit the systemd journal limit to SystemMaxUse=300M, as the default is enormous, and anything much bigger than 200M-300M is unusable.

nano /etc/systemd/journald.conf

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
SystemMaxUse=200M
#SystemKeepFree=
#SystemMaxFileSize=

But rebooting into this is still not setup

cat /boot/efi/EFI/«distro»/grub.cfg
cat /boot/grub/grub.cfg |grep subvol

Will show everything is still setup to boot into the old subvolume, but you are now in the new subvolume @rootfs_current and want to still be in it after reboot.

grub-install #normally rewrites all the needed grub.cfg files.
cat /boot/efi/EFI/«distro»/grub.cfg
cat /boot/grub/grub.cfg |grep subvol

Should now show everything setup to boot into the new subvolume @rootfs_current

But grub-install probably failed to recreate /boot/grub/grub.cfg, in which case you should recreate it a way appropriate to your distro, which for debian is grub-mkconfig -o /boot/grub/grub.cfg

Check them again, until they are correct.

Now reboot. Check that you are in the desired root file subvolume with

btrfs subvolume show /
cat /boot/efi/EFI/«distro»/grub.cfg
cat /boot/grub/grub.cfg |grep subvol

(This is for debian – arch puts the the entire boot directory on the efi disk, which means that if you break it, you cannot roll it back What you have to do instead if boot is broken is boot the livecd, chroot or arch-chroot into an earlier snapshot, and rebuild the efi partition.)

On systemd journalctl |grep Command will show you the history of what subvolumes you have been booting into.

2.1.2 manual rollback on arch

In arch, your entire /boot directory is on the efi partition, and you cannot roll it back. If it is broken, you boot a livecd and rebuild it.

And on arch, because arch is bleeding edge, and arch is built around empowering you to tinker, you often break the /boot partition.

mount /dev/sda3 -o subvol=@foo /mnt
# in an environment where /mnt is read only, create
# a mount point in /tmp, which is a memory only file system
mount /dev/sda1 /mnt/boot
arch-chroot /mnt

pacman -S linux
mkinitcpio -v -k <kernel_version> -g /boot/<kernel26.img> # Rebuild the initramfs
pacman -S grub
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg

It is probably a good idea to copy the EFI shell from the CDROM, and, within linux, add it to the grub menu, and within the shell, add it to the UEFI boot menu (with bcfg boot add 5 shellx64.efi) as the built in UEFI shell is rather harsh.

2.1.3 what you just accomplished

If you are still in @rootfs_current, you have not only configured your system for snapper and grub-btrfs, but you have manually done everything that they are going to do, manually created a snapshot and switched into the snapshot, and therefore are in a position to understand what is happening and be unsurprised by what they do.

And also, you do not actually need them, because you now know how to just do it all yourself.

By creating all these subvolumes, you ensured that future snapshots of / only restore your operating system and its configuration, enabling you to return to an earlier configuration, as you will undoubted need to do if you spend any time tinkering with your operating system, especially if you are tinkering with a bleeding edge system such as arch or an arch derived distro like endeavourOS.

Hurray. All done

2.1.4 explanation of what you just accomplished.

And possible ways you could have done similar things.

On UEFI, it starts with the UEFI bios executing a file in the efi partition.

On debian bookworm with ext4 installed, that eventually executes /EFI/debian/grubx64.efi, which is /boot/efi/EFI/debian/grubx64.efi in what is about to become the debian root.

/EFI/debian/grubx64.efi then interprets /EFI/debian/grub.cfg Which is not necessarily the real grub.cfg. It usually merely tells it to use a bios command to access actual root partition, using the real file system, containing the real grub.cfg at /boot/grub/grub.cfg

The location of that file system is search.fs_uuid «actual_uuid» root hd0,gpt2

However on debian trixie with btrfs the UEFI bios executes /EFI/BOOT/grubx64.efi which then interprets/EFI/debian/grub.cfg Which is not the real grub.cfg. It merely tells it to use a bios command to access actual root partition, using the real file system, containing the real grub.cfg at @rootfs/boot/grub/grub.cfg

Implying that at this point in the process, the root file system is subvolid=5 (Is it relying on the default being set correctly. Seems like it is not because it is not using mount syntax.)

And the real grub.cfg, still operating in subvolid=5, launches linux with a command to set the root to the actual subvol, which was originally something like @rootfs, but which you subsequently changed to @rootfs_current.

So to boot in a different subvol, to choose the subvol=@foo, /EFI/debian/grub.cfg has to point to a grub.cfg that specifies that subvol. Which the grub.cfg in that subvol should point to, but does not necessarily point to.If everything is setup consistently, it points @foo/boot/grub/grub.cfg, and the grub.cfg in /boot/efi/EFI//debian/grub.cfghas to point to that grub.cfg @foo/boot/grub/grub.cfg

The trouble is these files are set at different times, by different software, running as a result of different commands. If they are not consistent, boot will still work, but you will be confused because the grub menu is not what you expected, and does not get you to the root subvolume you expected. When you take a snapshot, the grub.cfg in that snapshot does not point to itself. And if the snapshot is read only, which it usually is, is never going to be rewritten to point to itself. So when switching root subvolumes, you are relying on grub.cfg that lives in a different suvolume. And are probably booting into a subvolume whose grub.cfg does not point to itself, and which /EFI/debian/grub.cfg does not point to.

If you are manually switching subvolumes, rather than relying grub-btrfs to do it for you, you recreate the grub files in the new subvolume,

Or you can do it grub-btrfs style, and manually edit /boot/grub/grub.cfg in your current subvolume to have additional boot menu entries pointing to other subvolumes, then boot into that other subvolume, at which point neither grub.cfg points to it, and the next time you reboot you will get the old submenu of the old subvolume, until you rewrite the grub.cfg in your new subvolume and grub.cfg on the efi disk.

Of course, some distros, quite a lot of distros, have the real grub.cfg on the efi disk. And, of course, you and grub-efi can both get confused about what is the real grub.cfg. The elaborate indirection is to accommodate old software that expects to find grub.cfg at /boot/grub/grub.cfg, but on an efi system, it is much saner to have the real grub at /EFI/debian/grub.cfg

This is all a convolutedly complex backwards compatibility mess, and some people are moving some of the time towards a less convoluted, but less backwards compatible, boot process. Systemd boot is cheerfully throws away all this stuff for something much more straightforward, but that is monolithic big coorporation sofware that you cannot control or influence. You are not in control of your linux if you are not booting by grub (or booting directly from the efi, which quite a lot of people are now doing.)

Or, if you want, like grub-btrfs, to be insanely clever, you could put a bunch of root file systems in @snapshots, and build a @foo/boot/grub/grub.cfg that points to many different subvolumes.

In fact you can place none of your root linux file systems in the top level of subvolid=5, put them all in @snapshots/«dir»/«snapshot»

We have, however, the issue that the grub-btrfs daemon rebuilds boot/grub/grub.cfg in the current linux root subvolume, and presumably points tells /boot/efi/EFI/debian/grub.cfg to point to it. So when you boot into some other subvolume, /boot/efi/EFI/debian/grub.cfg will still be pointing to the grub.cfg in the old subvolume. Though the grub-btrfs daemon is going to rewrite it, assuming that you have the grub-btrfs daemon running in the the linux file system you just booted, which you may not.

If you are doing all this manually, rather than relying on the grub-btrfs daemon, you have to be mindful of all this indirection. On the other hand, when snapper and grub-btrfs are running, they will take care of all this indirection for you.

The command to rebuild grub.cfg merely spits out an output stream (which you can grep for @rootfs), and you have to direct it to the right location for your system. Which tends to vary a lot.

If you have tinkered all this manually, and it works then: Hurrah. You are now setup for efficient snapshotting, and have manually created one recovery subvolume you can always boot into manually, from which you can list what recovery subvolumes you have, and then boot into one of them.

Thereafter you can always create a readwrite backup manually, by snapshotting yet another top level copy of your root file system to @rootfsn and can always boot into it manually if things go wrong.

3 install yay

As an unprivileged user:

sudo pacman -Syu
sudo pacman -S --needed base-devel git
mkdir -p ~/.cache/yay
cd ~/.cache/yay
git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si

After all, the whole point of arch is total fredom and total control – including the freedom to install completely unvetted software. If you want to play it safe, use pacman, rather than yay, but if you wanted to play it safe, why aren’t you using debian?

3.1 configuring snapper

At this point, assuming you have manually changed the linux root subvolume, you don’t actually need snapper. But it automatically creates a whole lot of snapshots, so you do not have to pay attention.

Before installing snapper, make a snapshot manually in case you regret snapper.

yay -S snapper

This installs the snapper service, three disabled services, and a three timer services to run them at intervals – which should be setup automatically, but check that they are

Created symlink '/etc/systemd/system/timers.target.wants/snapper-boot.timer' → '/usr/lib/systemd/system/snapper-boot.timer'.
Created symlink '/etc/systemd/system/timers.target.wants/snapper-cleanup.timer' → '/usr/lib/systemd/system/snapper-cleanup.timer'.
Created symlink '/etc/systemd/system/timers.target.wants/snapper-timeline.timer' → '/usr/lib/systemd/system/snapper-timeline.timer'.
Created symlink '/etc/systemd/system/sysinit.target.wants/snapperd.service' → '/usr/lib/systemd/system/snapperd.service'.

systemctl status snapper-boot
systemctl status snapper-cleanup
systemctl status snapperd

snapper likes to assume that config root exists, that being the default config, and any other config should be specified with snapper -c «config» but install seldom automatically creates it.

Futher, the command to create a config is broken. When you snapper -c root create-config /, it will insist on creating a new snapper subvolume. Which lives inside your root subvolume. And when you take a snapshot, it is not included, and so, when you rollback your system to a previous writeable snapshot it is not there any more. It is just an ordinary directory. And suddenly snapper does not work, and in consequence btrfs-grub, which you used to roll yourself back, no longer works either. Suddenly you are in deep doodoo.

So, before you create the root config, or any other, unmount your /.snapshots volume

I assume you have already setup top level subvolumes and /etc/fstab

Following the instructions for setting up snapper in the arch wiki for snapper setup.

umount /.snapshots
rm -r /.snapshots
snapper -c root create-config /
btrfs subvolume show /.snapshots  # should show the wrong snapshots volume
ls -hal /.snapshots  # should be empty
btrfs subvolume delete /.snapshots
mkdir /.snapshots
mount -a
btrfs subvolume show /.snapshots  # should show the right snapshots volume
# snapper will not care that you switched subvolumes underneath it.

snapper -c home create-config /home
# If you ever expect to switch to a snapshot of
# home, set its /home/.snapper subvolume up as for root
# in /btfrsroot/home/@snapper but you probably
# never intend to switch, you intend to restore selected parts of
# /home when needed from /home/.snapper with cp -a
snapper list-configs
snapper create -d"first snapshot of root, without most software, among them without grub-btrfs"
snapper -c home create -d"first snapshot of empty home"
snapper list
snapper -c home list

I am assuming you have already modified /etc/fstab to explicitly mount .snapshots and all your subvolumes as decribed in setup top level subvolumes and /etc/fstab, and as described in the arch wiki for snapper setup. so that they will survive in a different snapshot. If not, do it now.

You very rarely want to rollback /home, and if you do want to roll it back, you roll it back by modifying fstab andor an explicit mount command. If you ever rollback /home, you must first make sure all its subvolumes including /home/.snapper somewhere else and are explicitly mounted by /etc/fstab, or when you rollback, you lose them.

You care about restore points for your operating system, because you often make changes and find you want to reverse them.

You do not care much about backup for your operating system, because you can always re-install.

You don’t really care that much about restore points for home. What you care very much about is backups for home.

And btrfs insists on sending only read only snapshots, such as those created by snapper

There are multitude of bash scripts available for synchronising snapshots over ssh to a backup device, typically a btrfs formatted usb drive, snap-sync is, like most things, in the pacman repository. Though the use case that writers of btrfs had in mind was backing up over the home network over ssh to one of your other computers.

It might be a good idea, if backing up to thumbdrives, to create a special subvolume with all the stuff that you care about, and only the stuff that you care about, to create a snapper config for this special subvolume, and to create encrypted btrfs thumbdrives with a long, hard to guess passphrase. This use case for snap-sync is documented by jwillickers

When creating an encrypted thumbdrive, use Veracrypt. Luks has no hardening against being plugged into a hostile computer.

A Veracrypt thumbdrive is indistinguishable from a wiped thumbdrive. On the other hand, a bootable thumbdrive containing a veracrypt partition is very obviously a veracrypt thumbdrive. So you should keep an uninteresting bootable backup in the clear, that boots an os that can read another thumbdrive, one that is end to end a single veracrypt partition, and can do an ssh btrfs send of it.

Veracrypt also supports hidden volumes, unlike Luks.

There is a bootable memory only linux, which you can boot from a thumbdrive, then pull out that thumbdrive.

The primary use case for Luks is booting from a hard drive that is mostly encrypted.

3.2 setup rollback

3.2.1 rollback not setup

You can boot into a read only snapshot by interfering with grub menu by hitting 'e' and boot into a read only snapshot, or a read write snapshot created by snapper rollback. Which requires you to know the path to suitable read only snapshot, which you probably do not know.

In a read only snapshot, the only useful thing you can do is create a snapshot of it, which will by default be read write, and then manually boot into that.

Recall that snapshots are not backups against your hard disk dying. They are restore points for tinkering with your operating system. To make an actual backup, you use

And you can then ensure that reboots will work by recreating the two grub.cfg files, as you did in snapshotting and manual rollback

3.2.2 or snapper rollback script

This script is intended to be run after you have booted from a ro snapshot, which you can do by installing grub-btrfs, or by manually interfering with the grub menu by hitting 'e'

You have to edit the config file, snapper-rollback.conf before manually installing it, and create the mount point it wants (which you did in the setup above)

Thus you do not use the snapper rollback command, which seems hard to get working. Particularly if you are inside a readonly snapshot.

https://github.com/jrabinow/snapper-rollback

Have not tried this, so not documenting it here yet.

You have to edit the config file, /opt/snapper-rollback/snapper-rollback.conf before manually installing it, and create the mount point it wants (/opt being the right place to build or keep software that is not associated with any particular user except root. Or maybe /root is the right place.)

3.2.3 Or install grub-btrfs

Which on debian, you have to do from source. In fact you are doing it everywhere from source, because on an Arch derived linux you are installing it from yay, though I hear a rumor that the Manjaro installer can set all this up for you automatically. Their livecd requires a four gigabytes of ram to run, which is is disturbingly large for linux.

grub-btrfs installation guide

Which guide is remarkably useless. The main point is that after install you have to activate the service and regenerate grub.cfg. and you should edit /etc/default/grub-btrfs/config to reflect your distro and the structure of btrfs subvolumes.

yay grub-btrfs
systemctl status grub-btrfsd
systemctl enable --now grub-btrfsd
snapper create -d"after installing and configuring grub-btrfs"
systemctl status grub-btrfsd
# The grub-btrfsd daemon should respond to snapper create by
# automatically rebuilding grub

check that boot is now working, you have a submenu of snapshots.

boot into a recent readonly snapshot

snapper list
snapper rollback «snapshot number»
# you will likely get the error that snapper does not know the current ambit,
# tell it to make its best guess
snapper --ambit classic rollback «snapshot number»
shutdown -r now

Well, that did not work, and now I have a grub btrfs debugging issue.