Internal Boot: mkbootable reports success though every operation fails, leaving unformatted partitions and an unbootable system

Summary

Configuring Internal Boot onto two disks that carried pre-existing LVM thin pools appears to succeed in the WebGUI. It partitions both disks and then fails to create any filesystem on them. The system will not boot from the internal devices, because the ESP and the boot pool were never created.

mkbootable erases the on-disk LVM2 signature but does not tear down the active device-mapper mappings, and does not re-read the partition table before using the new partitions. Both leave the target partitions EBUSY. Every subsequent step fails, and mkbootable continues through all of them without checking exit status or surfacing an error.

Reproduction

1. Take two disks that previously held an LVM thin pool, with the VG still active, so device-mapper nodes are present and dmsetup ls is non-empty.

2. Run the Internal Boot setup targeting those disks as a mirrored boot pool.

3. Observe: partitions are created, filesystems are not. The system will not boot from them.

Evidence

From the flash logs/syslog. An identical sequence follows for sdc one second later.

Jul 31 13:53:48 emhttpd: shcmd (33): /usr/local/ungrub/mkbootable add sdb 32768
Jul 31 13:53:49 mkbootable: writing GPT on device (sdb), with boot area size 32768 MiB
Jul 31 13:53:49 root: /dev/sdb: 8 bytes were erased at offset 0x00000218 (LVM2_member)signature only
Jul 31 13:53:50 root: Creating new GPT entries in memory.
Jul 31 13:53:50 root: Warning: The kernel is still using the old partition table.cause 2
Jul 31 13:53:50 root: The new table will be used at the next reboot or after you
Jul 31 13:53:50 root: run partprobe(8) or kpartx(8)
Jul 31 13:53:50 root: The operation has completed successfully.false success
Jul 31 13:53:50 mkbootable: creating zpool flash in partition /dev/sdb3
Jul 31 13:53:50 root: cannot open '/dev/sdb3': Device or resource busy
Jul 31 13:53:50 root: cannot create 'flash': one or more vdevs refer to the same device, or one of
Jul 31 13:53:50 root: the devices is part of an active md or lvm devicecause 1
Jul 31 13:53:50 mkbootable: creating dataset flash/boot and mounting on /boot-transfer
Jul 31 13:53:50 root: cannot create 'flash/boot': no such pool 'flash'
Jul 31 13:53:50 root: cannot open 'flash/boot': dataset does not exist
Jul 31 13:53:50 root: mkfs.fat: unable to open /dev/sdb2: Device or resource busy
Jul 31 13:53:50 root: mount: /boot-transfer/efi: /dev/sdb2 already mounted or mount point busy.
Jul 31 13:53:50 kernel: /dev/sdb2: Can't open blockdev
Jul 31 13:53:50 root: Installing for x86_64-efi platform.
Jul 31 13:53:50 root: grub-install: error: failed to get canonical path of `rootfs'.
Jul 31 13:53:50 root: Installing for i386-pc platform.
Jul 31 13:53:50 root: grub-install: error: failed to get canonical path of `rootfs'.
Jul 31 13:53:50 root: Cannot get properties of flash: no such pool available.

failurethe two root causes, both unchecked

Analysis

Two independent causes of EBUSY, neither checked for:

1. Erasing the LVM2 signature does not remove active device-mapper nodes. The thin-pool DM targets still held the disks open. ZFS says so in as many words: “the devices is part of an active md or lvm device.”

2. The kernel had not re-read the partition table. sgdisk warned to run partprobe(8) or kpartx(8). mkbootable proceeded to mkfs.fat and zpool create on /dev/sdb2 and /dev/sdb3 before those nodes existed.

mkbootable then logged “The operation has completed successfully” and continued through five consecutive failures without aborting or reporting anything to the UI.

Suggested fixes

• Before partitioning, deactivate any VGs and remove DM holders on the target with vgchange -an and dmsetup remove, or refuse to proceed and tell the user why.

• After writing the GPT, run partprobe and udevadm settle before using the new partitions.

• Check the exit status of zpool create, mkfs.fat and grub-install. Abort and surface the first failure rather than continuing.

• Do not log “completed successfully” before the operations that can fail have run.

Workaround

Clear the holders and signatures before running Internal Boot setup:

vgchange -an <vgname>
dmsetup ls                    # confirm empty for those disks
wipefs -a /dev/sdX
partprobe /dev/sdX

A reboot also clears the device-mapper state, after which the setup should succeed.

Related

Same symptom, no diagnosis, two weeks stale: Internal Boot: Incorrect or no file system. Worth cross-linking, since that reporter is likely stuck on this.

Edited July 31Jul 31 by TheRealPeteJM