sys-fips Setup for Qubes OS

Goal: run a single FIPS node inside one ProxyVM (sys-fips) so any AppVM routed through it can access the FIPS mesh.

Architecture

AppVMs → sys-fips (ProxyVM) → sys-firewall → sys-net → Internet
                 └─ fips0 (TUN) → FIPS mesh (encrypted)

This setup is intentionally single-node per VM. If you want another independent FIPS identity/path, clone the VM (for example, sys-fips2).

---

Files in this directory

File	Purpose
README.md	This guide
configs/fips.yaml	Single-node config template
scripts/00-build-fips.sh	Run in build qube: build fips, fipsctl, fipstop into local bin/
scripts/01-dom0-create-proxyvm.sh	Run in dom0: creates sys-fips ProxyVM
scripts/02-install-fips.sh	Run in sys-fips: installs deps, binaries, and systemd units
scripts/03-configure-identity.sh	Run in sys-fips: configures /etc/fips/fips.yaml, identity mode
scripts/03-generate-node-configs.sh	Compatibility wrapper to 03-configure-identity.sh
scripts/04-start-fips.sh	Run in sys-fips: starts single FIPS node
scripts/04-start-fips-nodes.sh	Compatibility wrapper to 04-start-fips.sh
scripts/05-stop-fips.sh	Run in sys-fips: stops node + cleanup
scripts/05-stop-fips-nodes.sh	Compatibility wrapper to 05-stop-fips.sh
scripts/06-configure-dns.sh	Run in sys-fips: dnsmasq forwards .fips to localhost:5354 and applies Qubes /32 DNS compatibility patch
scripts/07-route-appvms.sh	Run in sys-fips: IPv6/ip6tables forwarding for routed AppVMs
scripts/08-add-peer.sh	Run in sys-fips: add static peer to /etc/fips/fips.yaml
scripts/09-add-known-peers.sh	Run in sys-fips: add two known bootstrap peers and restart FIPS
scripts/test-connectivity.sh	Unified test script
scripts/test-local.sh	Compatibility wrapper to test-connectivity.sh --local
scripts/test-from-appvm.sh	Compatibility wrapper to test-connectivity.sh --appvm
scripts/test-full-suite.sh	Compatibility wrapper to test-connectivity.sh --full

---

Quick start

1) Create sys-fips (run in dom0)

sudo bash 01-dom0-create-proxyvm.sh [template] [upstream_netvm]

Defaults:

• template: debian-12
• upstream netvm: sys-firewall

For VPN chaining, pass sys-vpn as upstream:

sudo bash 01-dom0-create-proxyvm.sh debian-12 sys-vpn

2) Build in this repo, then copy this entire folder

From this fips_setup repo in your build qube:

bash ./scripts/00-build-fips.sh
qvm-copy-to-vm sys-fips /home/user/anvil/fips_setup/

This keeps your update workflow simple: pull latest code, rebuild once in this repo, copy the folder, run install in sys-fips.

3) Install/configure/start (inside sys-fips)

cd ~/QubesIncoming/*/fips_setup/scripts
sudo bash 02-install-fips.sh
bash 03-configure-identity.sh
sudo bash 04-start-fips.sh
sudo bash 06-configure-dns.sh
sudo bash 07-route-appvms.sh

4) Add at least one external peer

sudo bash 08-add-peer.sh
sudo systemctl restart fips

Or add both known bootstrap peers automatically:

sudo bash 09-add-known-peers.sh

5) Test

sudo bash test-connectivity.sh --full

6) Point AppVMs at sys-fips (in dom0)

qvm-prefs my-appvm netvm sys-fips

---

Clone workflow

# dom0
qvm-clone sys-fips sys-fips2
qvm-start sys-fips2

Inside sys-fips2:

sudo bash 05-stop-fips.sh
sudo rm -f /etc/fips/fips.key /etc/fips/fips.pub
bash 03-configure-identity.sh
sudo bash 04-start-fips.sh

Now sys-fips and sys-fips2 are independent FIPS nodes.

---

Notes

• One FIPS node per ProxyVM is the production target.
• The previous two-node local loopback approach was useful for local experiments but is not required for normal Qubes deployment.
• FIPS mesh encryption is end-to-end. A VPN upstream is optional and can be chained.
• 06-configure-dns.sh removes dnsmasq --local-service from Debian's helper to avoid dropping AppVM DNS on Qubes /32 links.

---

Lessons learned from real sys-fips troubleshooting

These are the key fixes required to make AppVM → sys-fips → FIPS mesh work reliably in Qubes.

1) Intercept AppVM DNS in sys-fips with nftables

Qubes AppVM DNS normally targets the Qubes gateway addresses (for example 10.139.1.1 / 10.139.1.2). To make .fips resolution work from routed AppVMs, sys-fips must DNAT DNS to local dnsmasq.

Required nft hooks in sys-fips:

• table ip qubes, chain dnat-dns
• table ip qubes, chain custom-input

dnat-dns redirects UDP/TCP 53 traffic to local DNS (10.139.1.1:53) and custom-input allows DNS from AppVM interfaces (for example vif+).

2) IPv6 forwarding + route to FIPS space

sys-fips must forward IPv6 and route FIPS mesh space (fd00::/8) to fips0:

• enable IPv6 forwarding (sysctl net.ipv6.conf.all.forwarding=1)
• ensure route exists: ip -6 route add fd00::/8 dev fips0

Without this, AppVM traffic may resolve .fips names but fail to reach the resolved FIPS IPv6 destination.

3) AppVMs need a valid IPv6 source + route toward sys-fips

In testing, routed AppVM access required:

• a synthetic IPv6 on eth0 (example: fdaa::15/128)
• route of FIPS space via Qubes link-local gateway:
  • ip -6 route add fd00::/8 via fe80::fcff:ffff:feff:ffff dev eth0

This ensures traffic to resolved .fips AAAA records leaves the AppVM through sys-fips.

4) Persist network fixes in /rw/config/rc.local

For reboot safety:

• persist nftables DNS interception + IPv6 forwarding/route in sys-fips /rw/config/rc.local
• persist AppVM synthetic IPv6 + route in each AppVM /rw/config/rc.local that needs .fips

Do not rely on one-off manual commands for production.

5) Chromium-family browser caveat

Observed behavior:

• Firefox followed system resolver path and worked once networking was fixed.
• Chromium-family browsers (agent-browser, Brave) could still show ERR_NAME_NOT_RESOLVED even when dig, getent, ping, and curl worked.

Current operational workaround is to pin the .fips hostname in /etc/hosts for Chromium-family use-cases. See plans/chromium-fips-workaround.md for tested commands and policy notes.