Two nodes is the classic homelab trap. You buy a second machine, join it to the cluster, and assume you now have redundancy — until the day one node dies and the survivor freezes every VM it still owns, because two nodes can never form a majority of two. This post documents a real 2-node Proxmox VE cluster made boringly resilient: an external quorum device, shared iSCSI storage from the NAS, live migration measured at 11 seconds, and a power-fail shutdown that has actually been tested. Boring means nothing dramatic happens when something breaks.
Step 1: Understand why two nodes is not redundancy#
Corosync uses majority voting. Each node holds one vote, so a 2-node cluster has 2 expected votes and needs 2 to be quorate. Lose one node and the survivor holds 1 vote out of 2 — that is not a majority. The cluster goes read-only: you cannot start VMs, migrate, edit /etc/pve, or fail anything over. The surviving node is up and useless.
Check the current state on either node:
pvecm status
pvecm nodesRead the Quorum information block: Expected votes, Total votes, Quorate. With two nodes online you will see 2 expected and 2 total. Kill one and total drops to 1 while expected stays 2 — no majority, no quorum.
The lazy fix people reach for is forcing the expected vote count down:
# Emergency only. Manual, temporary, and split-brain bait.
pvecm expected 1That is a break-glass command for a node you are actively recovering, not a design. It survives one reboot and does nothing to protect you the next time. It also happily lets both nodes each believe they own the cluster if the link — not the node — is what failed. The real fix is a third vote.
Step 2: Add an external quorum device#
A QDevice is a third arbiter that holds a vote without running any VMs. The cluster becomes 3 expected votes; one node plus the arbiter is 2 of 3, a real majority. This is the single biggest resilience win in a 2-node cluster, and it costs a container.
In this lab the QDevice runs as a small Docker container on the NAS, image bcleonard/proxmox-qdevice, attached to the same subnet as the cluster’s virtualization network. The NAS is already on a UPS and already the shared-storage dependency, so it is the natural arbiter — the point is only that it is a third failure domain, not a third hypervisor.
Bring it up on the NAS:
docker run -d \
--name proxmox-qdevice \
--restart unless-stopped \
-p 192.0.2.50:5403:5403 \
bcleonard/proxmox-qdeviceBind the published port to the cluster’s virtualization subnet, not 0.0.0.0 — the QDevice only needs to be reachable by the Proxmox nodes. pvecm qdevice setup needs SSH to the QDevice host once, for the key exchange — publish it bound to the same subnet temporarily (-p 192.0.2.50:22:22), run the setup, then remove that mapping and recreate the container. At runtime the QDevice only listens on 5403.
Install the client on both Proxmox nodes:
apt update
apt install -y corosync-qdeviceThen, on one node only, register the device. Replace the address with your QDevice host:
pvecm qdevice setup 192.0.2.50The setup exchanges SSH keys with the QDevice host and pushes the corosync config to both nodes. Verify:
pvecm status
corosync-qdevice-tool -sYou should now see a Qdevice line in the membership information, expected votes of 3, and the qdevice reported as Qdevice (votes 1) with an A,V,NMW quality flag — Alive, Vote, Not Master Wins. If you ever need to back it out:
pvecm qdevice removeTest it for real before you trust it. Power off one node and confirm on the survivor that pvecm status still reports Quorate: Yes and that you can start a VM. If it does not, you have configured a decoration, not an arbiter.
Step 3: Give both nodes the same storage#
Quorum lets the cluster keep making decisions. Shared storage is what lets a VM actually run on either node. Without it, migration means copying disks, and a dead node means a dead VM until you restore it from backup.
Here the NAS exports one iSCSI LUN of roughly 900 GB with one-way CHAP authentication, and the nodes put an LVM volume group on top of it. Local NVMe stays as the pve VG on each node for the hypervisor system and a few node-pinned VMs; the shared LUN carries the VM bulk.
Install the initiator and set the initiator name on both nodes:
apt install -y open-iscsi
systemctl enable --now iscsid
cat /etc/iscsi/initiatorname.iscsiAuthorise that initiator name on the NAS target and set the CHAP credentials there first. Then configure CHAP on the node in /etc/iscsi/iscsid.conf:
node.session.auth.authmethod = CHAP
node.session.auth.username = <chap-user>
node.session.auth.password = <chap-secret>
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = <chap-user>
discovery.sendtargets.auth.password = <chap-secret>Discover and log in. This lab runs the target on port 3260 — adjust to yours:
iscsiadm -m discovery -t sendtargets -p 192.0.2.60:3260
iscsiadm -m node --targetname <iqn-of-your-target> -p 192.0.2.60:3260 --login
lsblkAdd the target to Proxmox once, from the GUI or the CLI, so both nodes get it from the cluster config. Set content none — you do not want Proxmox writing directly to the raw LUN, only LVM on top of it:
pvesm add iscsi iscsi-nas \
--portal 192.0.2.60:3260 \
--target <iqn-of-your-target> \
--content noneCreate the LVM VG on the LUN and expose it as shared storage. Do the pvcreate/vgcreate once, on one node only — the second node discovers the VG through LVM, not by repeating the create:
pvcreate /dev/disk/by-path/<your-lun-path>
vgcreate vg-vms /dev/disk/by-path/<your-lun-path>
pvesm add lvm shared-vms \
--vgname vg-vms \
--content images,rootdir \
--shared 1The --shared 1 flag is the one that matters. It tells Proxmox that both nodes see the same bytes, which is the precondition for migrating a VM without copying its disk.
Confirm both nodes agree:
pvesm status
vgsGrowing the LUN later. The LUN in this lab started around 600 GB and was expanded live to roughly 900 GB, with VMs running. Grow the target on the NAS first, then on each node rescan the session, grow the physical volume, and restart the Proxmox daemons so the new size is picked up:
iscsiadm -m node --targetname <iqn-of-your-target> -R
pvresize /dev/disk/by-path/<your-lun-path>
systemctl restart pvestatd
systemctl restart pvedaemon
vgspvestatd caches storage sizes; without the restart the GUI keeps showing the old capacity long after LVM knows better.
Step 4: Measure live migration, do not assume it#
With shared storage in place, migration moves memory and CPU state only. Nothing is copied to disk, so the wall-clock time is a function of RAM dirty rate and link speed, not VM size.
qm migrate <vmid> proxmox-node-2 --onlineWatch it and time it:
time qm migrate <vmid> proxmox-node-2 --online
qm listMeasured in this lab: 11 seconds for a running VM, end to end, on a ~0.5 ms inter-node link. The VM keeps its network sessions; a continuous ping to the guest during the migration shows a blackout too short to notice.
To confirm the migration is genuinely shared-storage and not sneaking a disk copy, migrate with --with-local-disks omitted. If a disk is not on shared storage, Proxmox will refuse or fall back to copying — and the time will be minutes, not seconds. For comparison, moving a few tens of gigabytes between storages in this lab took roughly 2 minutes:
qm move_disk <vmid> scsi0 local-lvmThat contrast is the whole point of Step 3. 11 seconds versus 2 minutes is the difference shared storage makes.
Move a whole node’s workload before maintenance:
ha-manager status
pvenode migrateall proxmox-node-2 --maxworkers 2Optional but recommended: put the important VMs under the HA manager so the surviving node restarts them automatically after a node failure. Quorum from Step 2 is what makes this safe — without the QDevice, HA on two nodes is a fencing accident waiting to happen.
ha-manager add vm:<vmid> --state started --max_restart 3
ha-manager statusStep 5: Back up to an isolated network#
Shared storage is a single logical copy. If the LUN is corrupted, both nodes see the corruption instantly. Backups are the layer that survives that, and they belong somewhere the production network cannot casually reach.
Put the backup store on its own network segment, isolated from the virtualization and management networks, with firewall rules that permit only the backup traffic. If ransomware or a bad rm reaches the cluster, it must not reach the backups.
Register the NFS store:
pvesm add nfs nfs-backup \
--server 203.0.113.20 \
--export /volume/backup \
--content backup \
--options vers=4.2Snapshot mode keeps VMs running during the backup. Cap the bandwidth so a full-cluster backup does not starve the storage network:
vzdump --all --mode snapshot --bwlimit 20480 --maxfiles 7Single VM, explicit storage and retention:
vzdump <vmid> --storage nfs-backup --maxfiles 5 --remove oldVerify what actually landed:
pvesm list nfs-backupRestore is the only test that counts. Do it at least once, on a throwaway VMID, before you need it:
qmrestore /mnt/pve/nfs-backup/dump/vzdump-qemu-<vmid>-<timestamp>.vma.zst 999 \
--storage shared-vmsAn untested backup is a hypothesis. Restore one per quarter and it becomes a fact.
Step 6: Make power failures boring#
Everything above assumes the machines stay on. A homelab loses power, and abrupt shutdown of a running hypervisor with open iSCSI sessions is exactly how you corrupt VM filesystems.
The UPS here is an EATON Ellipse PRO 650. NUT runs in server mode on the NAS — which is also the QDevice and iSCSI host, so it must be the last thing standing — and both Proxmox nodes run as NUT clients.
Install the client on both nodes:
apt install -y nut-clientPoint them at the NUT server in /etc/nut/upsmon.conf:
MONITOR ups@203.0.113.30 1 <monuser> <monpass> slave
SHUTDOWNCMD "/sbin/shutdown -h +0"
NOTIFYFLAG ONBATT SYSLOG+EXEC
NOTIFYFLAG LOWBATT SYSLOG+EXECSet the mode and start it:
echo 'MODE=netclient' > /etc/nut/nut.conf
systemctl enable --now nut-monitor
upsc ups@203.0.113.30upsc should report the UPS status, battery charge, and runtime. If it does not, fix that before writing a single shutdown rule.
The timing policy is where most setups get it wrong. Do not shut down on the first flicker, and do not wait until the battery is empty. This lab uses upssched with two timers: 30 seconds on battery to ride out brownouts, and 600 seconds before an early, controlled shutdown — well ahead of the low-battery cutoff.
/etc/nut/upssched.conf:
CMDSCRIPT /etc/nut/upssched-cmd
PIPEFN /run/nut/upssched.pipe
LOCKFN /run/nut/upssched.lock
AT ONBATT * START-TIMER onbatt-warn 30
AT ONBATT * START-TIMER early-shutdown 600
AT ONLINE * CANCEL-TIMER onbatt-warn
AT ONLINE * CANCEL-TIMER early-shutdown
AT LOWBATT * EXECUTE forced-shutdown/etc/nut/upssched-cmd — make it executable and keep it short:
#!/bin/sh
case $1 in
onbatt-warn)
logger -t upssched-cmd "UPS on battery for 30s"
;;
early-shutdown|forced-shutdown)
logger -t upssched-cmd "Initiating cluster shutdown: $1"
/sbin/upsmon -c fsd
;;
esacupsmon -c fsd is the command that matters: it sets the forced-shutdown flag, and every NUT client monitoring that UPS shuts down. Wire it in and make it executable:
chmod +x /etc/nut/upssched-cmd
systemctl restart nut-monitorTested, not assumed. A real power-fail test was run in this lab: mains cut, UPS on battery, timers expired, and both Proxmox nodes shut down cleanly and simultaneously on critical battery. Do this test on a weekend with nothing important running. Pull the mains, watch the syslog, and confirm both nodes go down in order:
journalctl -u nut-monitor -fAlso set the BIOS power-on behaviour to “last state” or “always on” on both nodes, otherwise your clean shutdown becomes a manual trip to the rack when power returns.
What boring looks like#
Boring is the goal. Here is what boring measures out to in this lab:
- Quorum: 3 expected votes (2 nodes + 1 QDevice). One node can die and the survivor stays quorate and writable.
- Live migration: 11 seconds, online, no disk copy, no dropped sessions.
- Disk move for comparison: ~2 minutes for a few tens of GB — the cost you avoid by using shared storage.
- Shared storage: one iSCSI LUN of roughly 900 GB, grown live from around 600 GB with VMs running, CHAP-authenticated, LVM on top.
- Backups: vzdump snapshot mode to NFS on an isolated network, retention 5–7 generations, restore rehearsed on a throwaway VMID at least quarterly.
- Power loss: 30 s brownout tolerance, 600 s controlled shutdown, both nodes verified down cleanly in a real mains-cut test.
None of this is exotic. It is one container for a third vote, one LUN both nodes can see, and one UPS daemon that actually gets tested. The 2-node trap is not that two nodes are too few — it is that two nodes without an arbiter, without shared storage, and without a tested power policy only look like a cluster. Add the third vote first; everything else builds on it.
