MHDFreefall

Perfect, thanks again. Procedure saved to my "unraidhowto.txt" for future reference.

Do I need to do anything to turn ZFS data verification back on, or will it revert after the next reboot?

Great, disk mounted! Thanks a lot for the help. Even if there is an error somewhere most of the files should be ok. Is there any reason to not trust the disks? I will run a long SMART test for a while to see if any errors pop up. Do you know what can have caused the corrupt data if not disk failure? I guess there is no advantage to having a three disk mirror when the file system fails. I will make the disks into a two disk mirrored pool, and set up some kind of periodic backup to the last disk in order to have a ready to go but slightly out of date mirror if the cache pool fails again.

root@Shrek:~# zpool import -fF -o readonly=on cachepool cannot import 'cachepool': I/O error Destroy and re-create the pool from a backup source. dbgmsg.txt

root@Shrek:~# zdb -l /dev/sdi1 ------------------------------------ LABEL 0 ------------------------------------ version: 5000 name: 'cachepool' state: 0 txg: 613982 pool_guid: 624089324922130285 errata: 0 hostname: 'Shrek' top_guid: 3293130853435037943 guid: 11309557061343029751 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 3293130853435037943 whole_disk: 0 metaslab_array: 132 metaslab_shift: 34 ashift: 12 asize: 8001558413312 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 11309557061343029751 path: '/dev/sdi1' whole_disk: 0 DTL: 96 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 15152360535842760272 path: '/dev/sdh1' whole_disk: 0 DTL: 88 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 17771728439540138945 path: '/dev/sdl1' whole_disk: 0 DTL: 97 create_txg: 4 features_for_read: com.delphix:hole_birth com.delphix:embedded_data com.klarasystems:vdev_zaps_v2 labels = 0 1 2 3 root@Shrek:~# zdb -l /dev/sdh1 ------------------------------------ LABEL 0 ------------------------------------ version: 5000 name: 'cachepool' state: 0 txg: 613982 pool_guid: 624089324922130285 errata: 0 hostname: 'Shrek' top_guid: 3293130853435037943 guid: 15152360535842760272 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 3293130853435037943 whole_disk: 0 metaslab_array: 132 metaslab_shift: 34 ashift: 12 asize: 8001558413312 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 11309557061343029751 path: '/dev/sdi1' whole_disk: 0 DTL: 96 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 15152360535842760272 path: '/dev/sdh1' whole_disk: 0 DTL: 88 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 17771728439540138945 path: '/dev/sdl1' whole_disk: 0 DTL: 97 create_txg: 4 features_for_read: com.delphix:hole_birth com.delphix:embedded_data com.klarasystems:vdev_zaps_v2 labels = 0 1 2 3 root@Shrek:~# zdb -l /dev/sdl1 ------------------------------------ LABEL 0 ------------------------------------ version: 5000 name: 'cachepool' state: 0 txg: 613982 pool_guid: 624089324922130285 errata: 0 hostname: 'Shrek' top_guid: 3293130853435037943 guid: 17771728439540138945 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 3293130853435037943 whole_disk: 0 metaslab_array: 132 metaslab_shift: 34 ashift: 12 asize: 8001558413312 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 11309557061343029751 path: '/dev/sdi1' whole_disk: 0 DTL: 96 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 15152360535842760272 path: '/dev/sdh1' whole_disk: 0 DTL: 88 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 17771728439540138945 path: '/dev/sdl1' whole_disk: 0 DTL: 97 create_txg: 4 features_for_read: com.delphix:hole_birth com.delphix:embedded_data com.klarasystems:vdev_zaps_v2 labels = 0 1 2 3

root@Shrek:~# zpool import cachepool -o readonly=on cannot import 'cachepool': I/O error Destroy and re-create the pool from a backup source. root@Shrek:~# zdb -l /dev/sdi failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3 root@Shrek:~# zdb -l /dev/sdh failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3 root@Shrek:~# zdb -l /dev/sdl failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3 root@Shrek:~#

root@Shrek:~# zpool import pool: cachepool id: 624089324922130285 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: cachepool ONLINE mirror-0 ONLINE sdi1 ONLINE sdh1 ONLINE sdl1 ONLINE shrek-diagnostics-20250705-1203.zip

I got tired of cache disks failing (about once a year) and restoring from appdata and VM backups, some of which always failed to work, so I made a ZFS cache pool of three mirrored disks. I just got finished setting everything back up, when some dockers stopped working today. I was unable to stop the failed dockers, so I killed the docker service and shut down the array, but due to active processes on the cache pool which did not respond to kill -9 i had to hard reset the server. Starting up again the cache pool of three disks is unmountable. All three disks complete SMART short tests without error. Is there anything I can do to save the pool?

Final update and conclusion. I replaced the USB drive again, and after a few weeks I still have not had any problems. So it seems that the problem was the first two USB drives, even though I have not had any problems with them afterwards for other use.

Every time I restart the Unraid server, it fails to restart with either a bz image checksum error or kernel panic. I first replaced the USB drive, but the first one I replaced has been working perfectly for everything else afterwards, so I do not think it is a failing usb drive. After the first few tries I found I just need to replace the bz-files with new ones from a backup, and everything works until the next reboot. I have let memtest run over a weekend without finding any errors. Everything else seems to be running fine. What are the possible causes of this corruption?

I just purchased an HP Z800 (2x X5690, 128 MB RAM) (https://support.hp.com/in-en/document/c01709726) I got cheap to upgrade my server, but it will not recognize the Unraid usb flash device as bootable. - The Z800 is able to boot off the two other bootable usb flash devices I have, one with with Win10 and one with Kaspersky rescue disk (Gentoo linux based). I installed Win10 on the Z800 from the usb stick just to make sure it worked fine. - The Unraid usb flash device was taken from the old server where it worked fine, and went back in the old server after the attempts and still works fine there. The old server has a Gigabyte G1.Sniper B5-CF mainboard with an i5-4460, 24 MB RAM. - I used the USB creator to make the flash device with the Kaspersky rescue disk into an Unraid usb flash device. After being turned into an Unraid boot device, the device would no longer boot on the Z800. I tried both with "allow UEFI boot" and without. - I have read Unraid forum posts with other Z800 issues, but none with any problems with the boot device. This indicates that others have managed to get this to work... - The Z800 BIOS is version 3.61A, which is the latest at the time of writing. Are there any changes I can make to the Unraid USB to enable it to boot? Are there any specific settings in the BIOS I should look for? Any other advice? Edit: Based on another thread I found, I reinstalled the "Kaspersky Rescue Disk" onto one USB, removed all the files, replaced with the manual install files, and ran the "Make bootable" as Admin. The Z800 boots fine now.