Replacing a failed disk in the ZFS root

This guide explains how to replace a failed disk in a ZFS mirror pool on Linux. The procedure covers taking the disk offline, copying the partition table to the replacement disk, re-adding the disk to the pool, and installing the bootloader after resilvering is complete.

Prerequisites

Root privileges are required. The examples below assume a server with two disks: /dev/sda (working) and /dev/sdb (failed). Adjust device names to match your setup.

Step 1: Check the pool status

Before making any changes, check the current state of the pool:

zpool status -v

The output will show which disk is faulted and its identifier in the pool.

Step 2: Take the failed disk offline

If the disk is ONLINE but failing, take it offline before the physical replacement:

zpool offline rpool <failed-device>

ZFS will stop attempting to use the failing disk before the physical replacement. Replace <failed-device> with the identifier shown in zpool status -v. If the disk is already FAULTED or REMOVED, skip this step and proceed directly to Step 3.

Step 3: Replace the disk physically

If your server supports hot swap, replace the disk without shutting down.

Otherwise, power off the system:

poweroff

After installing the replacement disk, boot the server and confirm the new disk is visible:

lsblk -o NAME,SIZE,MODEL
ls -l /dev/disk/by-id/

Use the by-id identifier of the new disk in the steps that follow.

Step 4: Determine the partition table type

Install gdisk if it is not already present:

apt install gdisk -y

Check the partition table type on the working disk:

gdisk -l /dev/sda

In the output, look for the Partition table scan section. For MBR the result will show MBR: MBR only and GPT: not present. For GPT it will show MBR: protective and GPT: present.

Step 5: Prepare and copy the partition table to the new disk

Most ZFS root setups use GPT partitioning. The MBR method is included for legacy systems.

GPT

Before proceeding, identify the /dev/sdX name that corresponds to the new disk, as partitioning tools require it:

lsblk -o NAME,SIZE,MODEL,SERIAL

Clear any old signatures from the replacement disk before copying the partition table:

wipefs -a /dev/disk/by-id/<new-disk>

The following command copies the partition table from /dev/sda (source) to /dev/sdb (destination):

sgdisk -R /dev/sdb /dev/sda

If the replacement disk is larger than the original, expand the GPT to use the full disk:

sgdisk -e /dev/sdb

Then assign a new unique UUID to the replacement disk:

sgdisk -G /dev/sdb

Reload the partition table so the kernel picks up the new layout:

udevadm trigger
udevadm settle

If the replacement disk was previously used in another ZFS pool, clear any old ZFS metadata to avoid conflicts:

zpool labelclear /dev/disk/by-id/<new-disk-partX> 2>/dev/null || true

Replace <new-disk-partX> with the actual ZFS partition identifier.

MBR

The following command copies the partition table from /dev/sda (source) to /dev/sdb (destination):

sfdisk -d /dev/sda | sfdisk /dev/sdb

If the new partitions are not visible in the system after copying, reload the partition table:

udevadm trigger
udevadm settle

Step 6: Add the disk to the pool

Replace the faulted disk entry in the pool with the new disk. First, check the exact identifier of the failed disk as shown in the pool:

zpool status rpool

In the output, find the entry for the faulted disk under the pool's device list. It will appear as a full path, for example /dev/disk/by-id/ata-WDC_WD10_12345-part3, with a status of FAULTED, UNAVAIL, or OFFLINE. Use that path as <old-device> in the replace command:

zpool replace rpool <old-device> /dev/disk/by-id/<new-disk-part>

Replace <old-device> with the identifier of the faulted disk from the previous step. Replace <new-disk-part> with the by-id identifier of the replacement disk partition, using the same partition number as the working disk (for example, -part3 or -part4). To find the correct identifier, run ls -l /dev/disk/by-id/.

If the old disk is fully unavailable (UNAVAIL) and zpool replace returns an error, add the -f flag to force the replacement:

zpool replace -f rpool <old-device> /dev/disk/by-id/<new-disk-part>

After running zpool replace, confirm that resilvering has started and the new disk appears as ONLINE:

zpool status

Resilvering continues automatically in the background.

Step 7: Monitor resilvering

Track resilvering progress with:

zpool status -v

The output shows progress, speed, and estimated time remaining. Wait until the status shows state: ONLINE and resilvering is complete before proceeding. Depending on disk size, this may take hours.

Step 8: Verify the boot dataset

Before installing the bootloader, verify that the pool has the correct boot dataset configured:

zpool get bootfs rpool

The output should show a dataset path such as rpool/ROOT/debian. If the value is none or points to the wrong dataset, set it manually:

zpool set bootfs=rpool/ROOT/debian rpool

Replace rpool/ROOT/debian with the actual root dataset name on your system. To list available datasets, run zfs list.

Step 9: Install the bootloader

Install the bootloader on the replacement disk after resilvering completes. The system continues to run from the original disk during resilvering, so this step is safe to perform once the pool is healthy.

BIOS (Legacy)

grub-install /dev/disk/by-id/<new-whole-disk>

On some systems, grub-install may not accept a by-id path. In that case, replace /dev/disk/by-id/<new-whole-disk> with the corresponding device name, for example /dev/sdb.

UEFI

Mount the EFI partition from the replacement disk to a temporary directory before running grub-install. This avoids shadowing the currently active EFI partition:

mkdir -p /mnt/new_efi
mount /dev/disk/by-id/<new-disk-part1> /mnt/new_efi

Replace <new-disk-part1> with the actual EFI partition identifier (typically -part1 or -part2; partition type EF00). Check with ls -l /dev/disk/by-id/ if unsure.

If the EFI partition is empty, copy the contents from the existing EFI partition before running grub-install:

cp -r /boot/efi/* /mnt/new_efi/

Install the bootloader:

apt install grub-efi-amd64 efibootmgr -y
grub-install --target=x86_64-efi --efi-directory=/mnt/new_efi 
--bootloader-id=debian --recheck /dev/disk/by-id/<new-whole-disk>
update-grub
update-initramfs -u -k all

Replace --bootloader-id=debian with the name matching your distribution (for example, ubuntu or debian). Replace /dev/disk/by-id/<new-whole-disk> with the by-id identifier of the entire replacement disk, not a partition.

Running update-initramfs ensures that the initial ramdisk includes the current ZFS modules and recognizes the new pool layout. Without this step, the system may fail to boot after the replacement.

Unmount the EFI partition after installation:

umount /mnt/new_efi

Verify the EFI boot entries:

efibootmgr -v

proxmox-boot-tool format /dev/sdb2
proxmox-boot-tool init /dev/sdb2

Conclusion

After installing the bootloader, reboot the server and verify that it boots normally. Run a scrub to check for any read/write errors that may have occurred during resilvering:

zpool scrub rpool

Then verify the pool status:

zpool status -v

The pool should be online with no errors. If any issues appear during the process, check the system logs with journalctl -xe for additional details.