NixOS configurations

NOTE: Code moved to https://git.firecat53.me/firecat53/nixos. Issues and PRs still accepted here for now. Github repo maintained as a read-only mirror.

Machines

• Laptop laptop
• Homeserver homeserver
• Backup server backup
• VPS cloud server vps
• Office secondary/spare desktop office
• Examples:
  • Flake install w/ home-manager and sops.
    • Encrypted or unencrypted base-btrfs or base-zfs
  • Bare minimum flake install for testing. base-minimal
• Installer/rescue ISO hosts/installer/iso - custom auto-build monthly into /mnt/downloads/iso for use with the trusty-boot-key project.
• Stage-1 install target minimal (hosts/installer/minimal) - bootable, no sops, no home-manager. Gets a bare-metal rebuild far enough along to run the real config. Its hardware-configuration.nix is a placeholder, regenerated during the install. See Installing locally.

Emergency restore kit

The trusty-boot-key USB carries the ISO plus the nixos/, dotfiles/ and shared/ trees, refreshed by flash_drive_update.sh.

Any secrets on the drive are all encrypted in some way. Age secret key stored in the password vault(s).

Building the system with sops-nix does not need forgejo — use --override-input my-secrets path:<path/to/nixos-secrets against the copy on the drive.

Update flow

forgejo is the transport and main is the deploy branch: every host pulls it by URL, and only the desktops (laptop, office) keep a checkout, for editing and on-demand rebuilds. flake.lock is committed, so main records exactly what the fleet runs.

• Lock bumps: the flake-lock-update timer on homeserver (hosts/homeserver/services/flake-lock-update.nix) runs nix flake update --commit-lock-file at 0400 daily in its own clone under /var/lib/flake-lock-update, builds every host's toplevel, and only then pushes main. A broken input fails the unit instead of reaching all five hosts at once: main stays on the last good lock, the 0440 upgrades are no-ops, and prometheus alerts on the failed unit. Each build is held by a result-<host> symlink, which roots the current toplevel for the binary cache below and lets the weekly gc reclaim only superseded ones.
• All five hosts auto-upgrade at 0440 from git+https://git.firecat53.me/firecat53/nixos.git?ref=main. That repo is public, so the config side needs no credentials.
• Secrets: the my-secrets flake input points at the private nixos-secrets repo over ssh. Each host authenticates with its own /etc/ssh/ssh_host_ed25519_key, registered on that repo as a read-only deploy key — the same key sops-nix already uses as the host's age identity, so it grants nothing the host didn't already have. The ssh alias lives in /etc/ssh/ssh_config (hosts/modules/common/sshd.nix) rather than ~/.ssh/config, because nixos-upgrade evaluates the flake as root. A newly built host needs its deploy key added before its first build.
• Deploy: merge to main and push. Hosts pick it up at with the auto-upgrade or use the immediate path below.
• Rollback: git revert + push, or nixos-rebuild --rollback on the affected host when you need it now. git log -p flake.lock on main is the audit trail of what every host ran on a given day.

Note the whole path depends on git.firecat53.me, which resolves to the VPS and proxies back to homeserver over wireguard — so a VPS outage stops unattended upgrades fleet-wide, including on the host that actually serves forgejo.

Testing, and deploying without waiting

nixos-rebuild test/switch --flake .#<host> still builds the working tree on demand, so day-to-day testing is unchanged. The catch: a machine left on a test build silently reverts to main at the next morning auto-upgrade, so re-apply the working tree after waking a machine you were mid-test on.

To push an urgent change to another host immediately:

nixos-rebuild switch --flake .#<host> --target-host <host> --build-host <host> --sudo

--build-host runs nix flake archive --to ssh://<host> under the hood, which ships the flake and its resolved inputs — so the target needs no checkout, and no deploy-key access of its own, for this path. --sudo authenticates through pam_rssh against the ssh agent, so it doesn't prompt. main stays the unattended path: anything deployed this way is overwritten with the morning auto-upgrade unless it is also committed and pushed.

Changing secrets

nixos-secrets must be pushed, not just committed, before nix flake update my-secrets will see the change.

Binary cache

homeserver serves its own /nix/store to the other hosts, so a nightly upgrade mostly comes off the LAN instead of cache.nixos.org.

It is served over ssh, not http (nix.sshServe in hosts/homeserver/services/nix-cache.nix). nix.sshServe creates an unprivileged nix-ssh account whose only capability is ForceCommand nix-store --serve, read-only, with forwarding and TTY off. Authorization is the per-host key already in ssh-keys.nix — the clients are derived from nixosConfigurations ∩ hostKeys, so a new host is authorized once it exists in flake.nix and its host key is recorded, with no list to maintain (install media has no host key).

Three things are easy to get wrong:

• Clients reject unsigned paths, whatever the transport. Locally-built paths carry no signature, so homeserver signs at build time via nix.settings.secret-key-files (sops nix-cache-key); clients trust the public half through extra-trusted-public-keys. secret-key-files only signs paths built after it is enabled — sign what is already there once:

  sudo nix store sign --key-file /run/secrets/nix-cache-key --all
  

priority=10 puts it ahead of cache.nixos.org (40), so anything homeserver has is preferred. the garbage-collection policy is the retention policy. If a path is deleted from the store it leaves the cache, so the cache holds the most recent builds but not much history.

Local packages

pkgs/ contains derivations for small one-off apps maintained alongside this repo. Each lives in pkgs/<name>/ with its own default.nix, and is wired up through pkgs/default.nix ({ pkgs }: { ... = pkgs.callPackage ./<name> {}; }). Service modules consume them with localPkgs = import ../../../pkgs { inherit pkgs; }.

Local Package list

• today — minimal Flask webapp for quick diary, workout, and book entries into the wiki. Deployed on homeserver via hosts/homeserver/services/today.nix at today.lan.firecat53.net.
• dashboard - static home service bookmark page served at home.lan.firecat53.net

Adding / removing services and hosts

Domain naming conventions

name	served by	reachable from	what it's for
*.lan.firecat53.net	homeserver Traefik	LAN + wireguard only	the "home" name every homeserver web app gets (real LE certs, but the name resolves to a LAN address)
*.firecat53.me	VPS Traefik	internet	public front door: proxies over wireguard to the homeserver .lan name, or to a VPS-local port
*.firecat53.com	VPS Traefik	internet	VPS-native public apps (grafana, VPS nextcloud, syncthing, Authelia portal, apex website)
firecat53.net + subs (matrix., s., nc., …)	homeserver Traefik directly (443 port-forward)	internet	federated/public apps that must be reached directly (Matrix, Akkoma, Nextcloud) — hand-written, not in the registry

Rules of thumb:

• At home or on wireguard → use the .lan name. Direct to the homeserver, no VPS hop; basicAuth prompt if the service has one.
• Out in the world → use the .me name. The VPS terminates TLS, Authelia 2FA's you (if auth = true), then proxies over wireguard to the same homeserver backend.
• .lan and .me are two doors to one app — the entry in hosts/modules/service-registry.nix is what ties them together.
• Service on the VPS itself → .me if it's registry-managed (behind Authelia), .com if it's hand-written and fully public.
• Not web / not HTTP isn't covered by any of this: ssh, syncthing sync, and wireguard are direct port openings on their hosts; Forgejo ssh (git.firecat53.me:2222) is a TCP passthrough on the VPS Traefik (proxy-me.nix) to the homeserver.

Web services (*.lan.firecat53.net + *.firecat53.me)

Standard web services on both hosts are driven by a single shared registry, hosts/modules/service-registry.nix. Derived from it:

consumer	generates
homeserver services/lan-proxy.nix	the .lan Traefik routers/services (+ -me/-noauth companions)
vps services/proxy-me.nix	the *.firecat53.me reverse-proxy routers/services
vps services/authelia.nix	2FA-protected domains (auth = true) + access_control rules
vps services/gatus.nix	homeserver .lan backend resolution for monitoring
homeserver services/dashboard.nix	build-time assertion that dashboard tiles point at real services

Each entry's fields — the attr name is the *.firecat53.me subdomain, and every flag controls exactly one generated thing (nothing is derived from another flag):

field	where	meaning
lan	homeserver sets	the .lan router host, and the backend the VPS proxies to over wireguard
port	all	backend port on localhost
url	all	backend URL; overrides port for non-localhost backends (e.g. hass VM)
auth	homeserver/vps	true gates <sub>.firecat53.me behind Authelia 2FA
passHost	homeserver	VPS forwards the real *.firecat53.me Host header (see below)
meRouter	homeserver	homeserver companion router on <sub>.firecat53.me (see below)
basicAuth	homeserver	homeserver router gets the auth basicAuth middleware
vpsBypass	homeserver	homeserver -noauth companion router skipping basicAuth for VPS traffic
rules	homeserver/vps	optional; per-service Authelia access_control rules (see below)

Service running on homeserver:

1. Create hosts/homeserver/services/<name>.nix defining just the app (bound to a localhost port), and add it to hosts/homeserver/services/default.nix. No Traefik config.
2. Add one entry to the homeserver set in the registry (or lanOnly for services that should not be exposed at *.firecat53.me):

   <sub> = { lan = "<sub>.lan.firecat53.net"; port = <port>; auth = <true|false>; };
   

3. Rebuild homeserver, then vps. The .lan router, the <sub>.firecat53.me router, the Authelia rule (if auth = true), and Gatus resolution appear automatically.

To inspect the fully-rendered result of the generated config:

nix eval --json .#nixosConfigurations.<host>.config.services.traefik.dynamicConfigOptions | jq

Services with their own HTTP basic auth (basicAuth = true — e.g. gollum, today, syncthing, transmission): the auth model is basicAuth on the LAN, Authelia on the internet — not both. The VPS proxies *.firecat53.me traffic in from 10.200.200.5, so set vpsBypass = true to generate a companion router that matches that source IP and omits the auth middleware, letting Authelia-2FA'd requests (and Gatus probes) through without a second prompt. LAN/wireguard clients keep hitting the plain Host() router and still get basicAuth. This relies on the homeserver Traefik not trusting forwarded headers (so ClientIP is the real TCP source). Services without basicAuth (app-level login, or auth = false) need no companion router.

Apps that build absolute redirects/URLs from the Host header (e.g. gollum, sonarr, OAuth redirect flows): the ClientIP trick won't work, because the backend would see the .lan host and redirect clients there (broken off-LAN). Instead set passHost = true so the VPS forwards the real <sub>.firecat53.me host, plus meRouter = true to generate the homeserver companion router keyed on that host (no ClientIP needed — only the VPS ever sends that host, already 2FA'd, so it carries no basicAuth).

Service running locally on the VPS:

1. Create hosts/vps/services/<name>.nix (+ add it to services/default.nix), binding the app to a localhost port.
2. Add one entry to the vps set in the registry:

   <sub> = { port = <port>; auth = <true|false>; };
   

3. Rebuild vps.

Removing a service: delete its registry entry (and the service file + its default.nix import). The .lan router, proxy router, auth rule, and monitor resolution all disappear with it — and the dashboard assertion fails at build time if a tile still points at it. Remove OIDC from authelia.nix if necessary.

Oddballs stay hand-written in their own service files (path-prefix rules, extra middlewares, non-.lan hosts): matrix-synapse, akkoma, nextcloud, and the nginx lan.firecat53.net apex. If a service outgrows the registry schema, move it to a hand-written stanza rather than adding fields.

Authelia: access control and OIDC

Access Control rules: auth = true entries get a blanket two_factor rule. To override that for specific paths (e.g. keep some public), add a rules list to the entry — Authelia access_control rules evaluated before the blanket rule, first match wins. Omit each rule's domain; authelia.nix derives it from the attr name. For example, Microbin uses this to keep paste viewing public but require authentication for submitting/delting pastes.

OIDC (apps that log in through Authelia rather than forward-auth, e.g. immich, audiobookshelf): the client is defined in identity_providers.oidc.clients in authelia.nix (hashed client_secret + redirect_uris), and the app stores the plaintext secret in its own sops secret. This pair is cross-host (Authelia on the VPS, app on the homeserver), so it is not registry-derived. See the secret-generation commands at the top of authelia.nix.

Hosts

To add a host:

1. Add it to flake.nix (mkSystem).
2. Set its wireguard address in the host's configuration.nix (networks."wg0".address).
3. Add its wireguard/LAN IPs to networking.hosts in hosts/modules/desktops/networking.nix.
4. Add sops keys (see the install/post-install sections below).
5. Install per General Install Procedures.

Monitoring

In general, VPS services are monitored from the Gatus instance on homeserver and homeserver LAN services are monitored from Gatus on VPS via a wireguard tunnel. Prometheus scrapes metrics from all the servers and feeds Alertmanager and Grafana.

Podman healthchecks

A HostSystemdServiceCrashed alert for a unit like e2d6c5…c3d86-3c8e64…0a02.service is not a real service. Podman runs each container healthcheck in a transient unit named after the container ID, and a failed probe leaves it failed until the next tick.

Because the prometheus systemd collector is enabled, this generates a lot of recurring noise in prometheus due to these checks. Service monitoring (Gatus) is used instead of the container health checks for issue notifications. The container health checks are filtered via a regex in prometheus-exporters.nix. Note that flag overrides node-exporter's default exclusions rather than extending them, hence the defaults repeated there.To make podman health checks actionable, set --health-on-failure=restart, or export .State.Health.Status as a textfile metric (pattern: airvpn-port-check.nix). The container's own podman-<name>.service is unaffected and still alerts if a container dies.

What the qBittorrent VPN checks actually prove

VPS gatus checks the QBT GUI and socks-proxy, but both are port-mapped to the host (-p 127.0.0.1:8081:8081, -p 2222:22 on the pod), so they answer even if the tunnel is down — they prove the containers are alive, not that traffic is egressing. Tunnel egress is covered by vps services/airvpn-port-check.nix, which probes the AirVPN forwarded port from outside. It can take 30-45 minutes for the system to notice that the AirVPN tunnel is actually down.

General Install Procedures

Tips

1. Generate hostId (for ZFS systems): head -c4 /dev/urandom | od -A none -t x4
2. Hetzner VMs apparently require grub instead of systemd-boot (as of 2025-08)
3. Available options (defined in hosts/modules/common/options.nix): a. isRemote (bool) - set for hosts directly exposed to the internet (not behind the LAN firewall). Controls the wireguard endpoint and keeps LAN-only ports (e.g. Eternal Terminal 2022) closed publicly. Default false. b. isVirtual (bool) - set for virtual hardware (VPS or VM). Disables hardware-specific services (fwupd, smartd). Default false. c. latestZFSKernel (bool) - set to use latest available ZFS compatible kernel. Default false. d. tmuxStatusColor (str) - tmux status bar background color, for per-host visual distinction. Default "#cba6f7" (catppuccin mocha mauve).

Installing using nixos-anywhere

https://github.com/nix-community/nixos-anywhere/blob/main/docs/quickstart.md

1. Create new (Ubuntu is fine) cloud server. Add one of the public keys. Adjust DNS 'A' records if needed.
2. SSH into the new box and update the disk device name(s) and partition layout (if needed) in disko-config.nix.
3. nix run github:nix-community/nixos-anywhere -- --generate-hardware-config nixos-generate-config ./hosts/<host>/hardware-configuration.nix --flake .#<host> --target-host root@<ip or domain>
4. If problems arise, add --no-reboot to the above command so you can troubleshoot the new install.
5. Post install — sshd generates the host key on first boot, so the rekey there is required before the config will build cleanly.

Installing locally on a new machine using the ISO installer

Two stages because a full host closure will not reliably build from a booted ISO — it has run the installer out of memory more than once, even at 8GB+.

Stage 1 installs #minimal (hosts/installer/minimal): small enough to build without running out of memory, with no sops or home-manager. Install also generates a new SSH host key.

Stage 2 re-keys the secrets to use the new host key and then builds the actual machine config.

# 1. Boot the ISO. Device keys from ssh-keys.nix are already authorized for
#    the `nixos` user, so just ssh in.
sudo mkdir -p /run/data && sudo mount /dev/disk/by-label/DATA /run/data
cp -a /run/data/{nixos,shared,dotfiles} ~/
NIXOS=~/nixos/nixos

# 2. Partition (update device ids first).
sudo disko --mode disko $NIXOS/hosts/<host>/disko-config.nix

# 3. Hardware scan into the minimal target. Drop --no-filesystems if not disko.
sudo nixos-generate-config --no-filesystems --show-hardware-config --root /mnt \
  > $NIXOS/hosts/installer/minimal/hardware-configuration.nix

# 4. Stage 1. --no-root-password: users.nix already sets initialHashedPassword.
#    If the build runs out of space: sudo mount -o remount,size=8G /run/user/0
sudo nixos-install --flake $NIXOS#minimal --no-root-password
sudo cp -a ~/{nixos,shared,dotfiles} /mnt/home/firecat53/
sudo chown -R 1000:1000 /mnt/home/firecat53/{nixos,shared,dotfiles}
sudo umount -R /mnt && sudo zpool export -a && sudo systemctl reboot

Then post install.

Post install

1. Change the firecat53 and root passwords.
2. Sync ~/nixos/ to the new machine (configs and secrets) if needed.
3. Generate SSH keys per SSH key generation. Needed to push to forgejo in step 4.
4. Rekey. Also update hostKeys.<host> in ssh-keys.nix and re-add the pubkey as a read-only deploy key on the forgejo nixos-secrets repo — it is the my-secrets deploy key too. Being read-only, it cannot push: that needs the device key from step 3, or do the push from a working host.

# Add AGE-SECRET-KEY-1... from the pw vault for any machine using home-manager
mkdir -p ~/.config/sops/age
(umask 077; $EDITOR ~/.config/sops/age/keys.txt)
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt

# deriving the recipient - on the new host itself:
ssh-to-age -i /etc/ssh/ssh_host_ed25519_key.pub
# ... or remotely, for a host with no age key of its own:
ssh-keyscan <hostname> | ssh-to-age

# then update the sops config with the new age recipient
cd ~/nixos/nixos-secrets
$EDITOR .sops.yaml                # &<host> age1...
sops updatekeys <host>/secrets.yaml common/secrets.yaml
git commit -am 'Rekey <host>' && git push

5. Build the real config. --override-input covers the window before the rekey reaches forgejo; afterwards refresh the lock and commit it.

cd ~/nixos/nixos
nixos-rebuild switch --flake .#<host> \
  --override-input my-secrets path:$HOME/nixos/nixos-secrets --sudo
nix flake update my-secrets && git commit -am 'Rekey <host>' && git push

Until that lock change is on main, the daily auto-upgrade rebuilds from a .sops.yaml with the old host key and activation fails.

If the rekey was done remotely there is no local checkout to override against: push it first, then bump the lock and deploy from the working host. 6. sudo nmcli connection import type wireguard file /etc/wireguard/wg0.conf for networkmanager. 7. Update syncthing device ID's if necessary. Re-add servers on phones and wife's laptop if needed.

SSH key generation (new or rebuilt host)

The device key (devices.* in hosts/modules/common/ssh-keys.nix) — not the host key, which sshd generates at stage-1 boot and the install procedure rekeys into hostKeys.* of that same file.

Each desktop/laptop host gets its own SSH keypair — private halves never leave the box, only pubkeys land in ssh-keys.nix. Sequence matters because the host can't reach itself via key auth until its pubkey is authorized elsewhere.

1. On the new/rebuilt host, generate the device key as firecat53:

ssh-keygen -t ed25519 -C "firecat53@<hostname>" -f ~/.ssh/id_ed25519
wl-copy < ~/.ssh/id_ed25519.pub

2. On a working host with repo access, paste the pubkey into hosts/modules/common/ssh-keys.nix under the matching devices.<host> attribute. Commit and push.
3. Add the device pubkey to: a. GitHub / forgejo account (web UI) — needed for git operations b. HomeAssistant ~/.ssh/authorized_keys for the root user c. Any other external service the host needs to reach
4. Rebuild every host that should authorize this device:
5. (Desktops/laptops using the autossh tunnel only) The passphraseless autossh private key is shared across all tunnel clients and lives in sops as autossh-key. Add it to the host's sops file (the matching pubkey is already in ssh-keys.nix as autossh, authorized on homeserver). To rotate, generate one new keypair, update autossh in ssh-keys.nix, and re-encrypt autossh-key into every desktop sops file.

Specific host instructions

Minimal and Base Installs

1. Copy/rename desired exmaple directory to hosts/xxxxx.
2. Update CHANGEME items (disk device id, disk encryption, etc).
3. Update configuration as desired. a. If using base-btrfs with encryption, rename disko-config-luks.nix to disko-config.nix
4. Add new host to flake.nix.
5. Sops-nix (if needed): a. Add any sops-nix keys to nixos-secrets/xxxx/secrets.yml b. Add new host to nixos-secrets/.sops.yml c. sops updatekeys happens after install d. Update flake inputs
6. Install using nixos-anywhere

BACKUP server

1. Install
2. ssh-keygen -f /etc/ssh/backup && chown backup: /etc/ssh/backup. Change backupPull to the public key in ssh-keys.nix and rebuild all servers.
3. sudo -i -u backup ssh -i /etc/ssh/backup <backup source hostname(s)> and accept fingerprint

LAPTOP/OFFICE desktops

1. Install
2. Login to Vaultwarden
3. Login to Firefox Sync a. Extensions - ClearURLs, floccus, Gnome Shell integration, Proxy SwitchyOmega 3, Stylus, uBlock Origin, User-Agent switcher and Manager, Vimium
4. Open Syncthing on this machine and other machines. Ensure syncing is setup.
5. Stow (dotfiles)

cd home/firecat53/docs/family/scott/src/dotfiles
stow -t /home/firecat53/ --dotfiles stow/
stow gomuks music passwords python ssh-scotty

Homeserver

Disko (WARNING: instructions not completely verified working yet)

This directory contains disko configuration for homeserver's two-NVMe-drive mirrored ZFS setup with systemd-boot.

Current Layout

Both NVMe drives have identical partition layouts:

Part	Size	Purpose
p1	1G	EF00 ESP (vfat) - /boot on nvme0
p2	4G	Unused (legacy bpool placeholder)
p3	~1.8T	rpool (ZFS mirror)
p4	8G	Encrypted swap

• rpool: Mirrored across both NVMe drives
• ESP: Only the first drive's ESP is mounted at /boot (systemd-boot)
• datapool: Separate SATA drives (not managed by disko)

Safety Information

WARNING: Do NOT run disko --mode disko on an existing system. It would reformat the drives. The disko config is used only as a NixOS module for fileSystems generation, and as a reference for future fresh installs.

Scenario A: Fresh Install (Empty Drives)

1. Boot into NixOS installer

2. Clone your configuration

git clone <your-repo-url> /tmp/nixos-config
cd /tmp/nixos-config

3. Review and adjust disko-config.nix

Check these settings in hosts/homeserver/disko-config.nix:

• Disk devices: Update device paths to match your drives
• Partition sizes: Adjust if needed (swap=8G, rpool uses remaining space)
• Pool/dataset options: Modify compression, reservation, etc. as desired

4. Run disko to partition and format

sudo nix run github:nix-community/disko -- --mode disko /tmp/nixos-config/hosts/homeserver/disko-config.nix

This will:

• Partition both NVMe drives
• Create the mirrored rpool ZFS pool and all datasets
• Format the ESP partition
• Set up encrypted swap on both drives

Note: This does NOT touch the SATA drives (datapool). Import datapool separately after install.

5. Install NixOS

Disko automatically mounts everything to /mnt.

sudo nixos-install --flake /tmp/nixos-config#homeserver

6. Reboot

reboot

7. Post-install

sudo zpool import -f datapool

Scenario B: Fresh Install + Migrate Data via zfs send/recv

1-4. Follow Scenario A steps 1-4

Run disko to partition, create pools, and mount everything. This creates empty datasets.

5. Receive ZFS data into the new pools

Before installing NixOS, populate the datasets with your data:

# Import the old/backup pool with an alternate name
sudo zpool import -R /tmp/oldpool oldrpool

# Recursive send of all data datasets
sudo zfs snapshot -r oldrpool/data@migrate
sudo zfs destroy -r rpool/data
sudo zfs send -R oldrpool/data@migrate | sudo zfs recv rpool/data

# Fix mountpoints to use legacy (disko expects legacy mounts)
sudo zfs set mountpoint=legacy rpool/data/home
sudo zfs set mountpoint=legacy rpool/data/podman_volumes
# ... etc for each dataset

Note: You generally don't need to migrate system datasets (rpool/nixos/*) since NixOS will rebuild those during install. Focus on the rpool/data/* datasets. datapool lives on separate SATA drives - just import it directly.

After migrating, re-mount everything:

sudo umount -R /mnt
sudo nix run github:nix-community/disko -- --mode mount /tmp/nixos-config/hosts/homeserver/disko-config.nix

6. Clean up and install

sudo zpool export oldrpool
sudo nixos-install --flake /tmp/nixos-config#homeserver
reboot

Recovery from failed drive (e.g. nvme0 failed)

1. Boot: enter the firmware boot menu and select the second NVMe drive. It boots via EFI/BOOT/BOOTX64.EFI (the removable-media fallback).
2. Replace the dead drive in the ZFS mirror:
   • sudo zpool replace rpool <old-nvme0-part3> /dev/disk/by-id/<new>-part3
3. Recreate the ESP on the replacement drive and let the next rebuild resync:
   • sudo mkfs.vfat -F32 -n ESP /dev/disk/by-id/<new>-part1
   • sudo nixos-rebuild boot --flake .#homeserver # repopulates /boot

VPS (cloud server)

1. Install using nixos-anywhere

Mail

homeserver runs mbsync with mail stored in ~/mail. Syncthing replicates it to/from the desktops with Neomutt working from that local maildir. Only sending goes out directly via SMTP.

Notmuch

The xapian database lives at ~/mail/.notmuch and is excluded from syncthing by ~/mail/.stignore, so each desktop keeps its own tags. Only the maildir itself is shared between laptop and office.

One folder tag per maildir folder is rebuilt from the maildir on every run by the post-new hook, so mail moved in one client retags itself on the next index. unread and flagged come from the maildir flags directly. The folder list lives in home-manager/common/mail-folders.nix and also drives neomutt's sidebar and the indexing path unit, so the three can't drift apart.

Mail arrives by syncthing rather than a local delivery agent, so nothing triggers indexing on its own. notmuch-new.path watches every folder's cur and new for inotify events, and since inotify only reports live changes, the notmuch-new timer sweeps hourly to catch mail that landed while the session was down. On a new desktop run notmuch new once by hand first.