October 26Oct 26 Hello,I'm hoping someone here can help.So I have a couple of zfs pools. One in particular (named Mediafun) is made up of 13 devices. I had some errors show up on one of them, Device 12 (sdn). I kept using it until the system removed the disk.I ran a quick SMART test on that disk that came back with no errors. I tried to reintroduce it, and the resilver process started but was moving extremely slow (0.3%, 70 days to go) and didn't run for too long before the drive was shown as removed again by the system and seemed to stop the resilver process.In the meantime I had a second drive, Device 5 (sdt), that started stating it had errors. I decided to shut down the server and check to see if the same breakout cable was connected to both drives as I have had loose/bad cables in the past. Device 12 (that was previously removed by the system) is connected to a breakout cable and Device 5 has a sata cable connected directly to a Sata port on the motherboard. I re-seated only those two sata cables and powered up the server.Now all drives are showing as offline and when I try to start the array I'm receiving "Wrong Pool State mediafun - invalid config". Device 12 (sde) still shows as unassigned.I've tried rebooting as well as assigning the drive that was removed (Device 12) back into that same slot. I'm wondering if stopping it during the resilver process was problematic as I'm seeing "sdn1/old" as OFFLINE and "sdn1" as UNAVAIL when I run zpool import.Any suggestions would be welcome.*******************************I ran lsblk -o name,size,model,serial-Result was all the drives are seen.When I run "zpool import" I see the below:pool: mediafunid: 2954754274543482954state: DEGRADEDstatus: One or more devices were being resilvered.action: The pool can be imported despite missing or damaged devices. Thefault tolerance of the pool may be compromised if imported.config:mediafun DEGRADEDraidz2-0 DEGRADEDsde1 ONLINEsdf1 ONLINEsda1 ONLINEsds1 ONLINEsdt1 ONLINEsdb1 ONLINEsdd1 ONLINEsdj1 ONLINEsdh1 ONLINEsdk1 ONLINEsdo1 ONLINEreplacing-11 UNAVAIL insufficient replicassdn1/old OFFLINEsdn1 UNAVAILsdm1 FAULTED corrupted dataWhen I run zpool status -v mediafun I receive the following: pool: mediafun state: DEGRADEDstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Mon Oct 26 07:52:13 2026 0B / 108T scanned, 0B / 108T issued 0B resilvered, 0.00% done, no estimated completion timeconfig: NAME STATE READ WRITE CKSUM mediafun DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 sde1 ONLINE 0 0 4 sdf1 ONLINE 0 0 4 sda1 ONLINE 0 0 4 sds1 ONLINE 0 0 4 sdt1 ONLINE 0 0 4 sdb1 ONLINE 0 0 4 sdd1 ONLINE 0 0 4 sdj1 ONLINE 0 0 4 sdh1 ONLINE 0 0 4 sdk1 ONLINE 0 0 4 sdo1 ONLINE 0 0 4 replacing-11 UNAVAIL 0 0 0 insufficient replicas sdn1/old OFFLINE 0 0 0 sdn1 REMOVED 0 0 0 7355020672882961856 FAULTED 0 0 0 was /dev/sdm1I've been trying to work through Claude but things are getting pretty in the weeds for me. Part of what its come back with is it thinks that Device 13 (sdm) somehow had it's label changed so it's GUID is different than the GUID ZFS is expecting. I'm still unsure why anything for that drive may have been changed.In it's words:Two different records are involved.The pool’s config (zdb -e -C mediafun) says position 12 is a disk with guid 7355020672882961856 at /dev/sdm1. This is what ZFS expects, and it’s the value we want. The config is read from the labels of the healthy disks, so it is intact. It also shows that the attempted replace on sdm1 never made it into the pool’s config.The label on the disk itself (zdb -l /dev/sdm1) says guid: 8450324955016200314. That is the stub label that overwrote the original.When ZFS opens sdm1, it compares the guid in the disk’s own label to the guid the config expects. 8450324955016200314 isn’t 7355020672882961856, so ZFS decides this isn’t the right disk and marks it FAULTED with “corrupted data”. The disk is fine. Its label is simply wrong.This is the mismatch the reconstruction would fix. The goal is to make sdm1’s label say 7355020672882961856, so it matches what the config already says.I realize this is a dump of info, let me know if I can clarify anything.Diagnostics are attached.mediaunraid-diagnostics-20261026-1226.zip Edited October 27Oct 27 by khior More information and clarification
October 27Oct 27 The pool should still be recoverable. 11 of the 13 devices are fine and raidz2 can survive two missing devices, but to reimport the the pool using the GUI is best to clear the repoacling line.With the pool imported using the CLI, post the ouptu from zpool status -g mediafun
October 27Oct 27 Author Running zpool status -g mediafun returns the following: pool: mediafun state: DEGRADEDstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Mon Oct 26 07:52:13 2026 0B / 108T scanned, 0B / 108T issued 0B resilvered, 0.00% done, no estimated completion timeconfig: NAME STATE READ WRITE CKSUM mediafun DEGRADED 0 0 0 7917958360641739530 DEGRADED 0 0 0 12441589783024071164 ONLINE 0 0 4 6719155247144394418 ONLINE 0 0 4 7709659236461100366 ONLINE 0 0 4 11888992234242972831 ONLINE 0 0 4 456132895116674593 ONLINE 0 0 4 1778687606147317910 ONLINE 0 0 4 3065735978438110781 ONLINE 0 0 4 8405909615551463313 ONLINE 0 0 4 15925004999378470692 ONLINE 0 0 4 14978431327348962353 ONLINE 0 0 4 5776364110155512343 ONLINE 0 0 4 7997955580701256598 UNAVAIL 0 0 0 insufficient replicas 4734395572079804081 OFFLINE 0 0 0 16037769206436712332 REMOVED 0 0 0 7355020672882961856 FAULTED 0 0 0 was /dev/sdm1
October 27Oct 27 Author I had previously tried the following command based on some earlier searches:zpool import -o readonly=on -R /mnt/rescue mediafunMy understanding is it imported it in a read-only state to an alternate mount point. I'm guessing I would need to reverse that as well?Also, when I run "zpool list" I only see mediafun although I have a couple other pools. To see those other pools I'd need to import those also?:NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTmediafun 213T 0 213T - - 0% 0% 1.00x DEGRADED /mnt/rescue
October 27Oct 27 Thanks. In the replacing vdev, 16037769206436712332 is the new device that was being resilvered, and it has no data yet. Detaching it cancels the replacement. Device 12 will then show as one missing member instead of the "replacing" entry, which should let the GUI better import the pool.The pool is currently imported read-only, and a detach needs write access. So first export it, then import it again without the read-only option. The -N option means the datasets are not mounted:zpool export mediafunzpool import -N mediafunzpool detach mediafun 16037769206436712332If any of these commands gives an error, stop and post the full output. Otherwise post the output of:zpool status mediafunUse the normal command this time, not -g. Leave the pool as it is after that, and don't run any other commands on it. I'll post the next steps for the GUI import.About your other questions:- Yes, the read-only import to /mnt/rescue needs to be undone. The zpool export above does that.- zpool list only shows pools that are imported. Your other pools are normally imported by Unraid when the array starts, and the array hasn't started because of this pool. Don't import them manually. They will be imported when the array starts.- Don't try to rebuild or edit the label on Device 13. Writing labels by hand can make things much worse. The data on that disk will be rebuilt by a resilver later, the same as Device 12.- Until both devices are resilvered, the pool has no redundancy left. If you have important data on it that isn't backed up anywhere, copy it off as soon as the pool is mounted again.
October 27Oct 27 Author Thank you very much. All commands ran successfully.The output of zpool status mediafun is as follows: pool: mediafun state: DEGRADEDstatus: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.action: Replace the device using 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J scan: scrub in progress since Tue Oct 27 10:54:23 2026 2.41T / 108T scanned at 154G/s, 0B / 108T issued 0B repaired, 0.00% done, no estimated completion timeconfig: NAME STATE READ WRITE CKSUM mediafun DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 sde1 ONLINE 0 0 0 sdf1 ONLINE 0 0 0 sda1 ONLINE 0 0 0 sds1 ONLINE 0 0 0 sdt1 ONLINE 0 0 0 sdb1 ONLINE 0 0 0 sdd1 ONLINE 0 0 0 sdj1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdo1 ONLINE 0 0 0 sdn1 OFFLINE 0 0 0 7355020672882961856 FAULTED 0 0 0 was /dev/sdm1
October 27Oct 27 Good, the replacing entry is gone. The pool now shows 11 online members plus the two missing ones, which is what we wanted. First, export the pool again: zpool export mediafun Then import it using the GUI with only the 11 online devices: - on Main click on the first device for that pool and then "Remove pool"- back on Main, create a new pool with the same name (mediafun) and 11 slots- assign the 11 online devices, and leave the filesystem set to auto- start the array to import the poolThe pool will import degraded, and that is expected. If you see a warning that data on a device will be overwritten, or any other error, don't start the array. Post a screenshot instead.Once the array is started, post new diagnostics so we can confirm all looks good before the next step, same if the poo doesn't import. Until the two missing devices are resilvered the pool has no redundancy. If there is anything important on it that isn't backed up, it's a good idea to copy it somewhere else now.If the diagnostics confirm everything is OK, stop the array, assign both missing devices at the same time, and start the array to begin the resilver. Before that, replace or reseat the cable for Device 12. Its SMART report shows CRC errors, and those usually point to a connection problem.
October 27Oct 27 Author Ok, I guess the only questions I have are:The original pool had 13 slots with a zfs filesystem, Raid Z-2. I thought the amount of disks chosen for a ZFS RAID pool are set and I couldn't add slots afterward once the pool is created. If I create a new pool with the same name and only 11 slots, will I be able to add both devices?I had an issue last year where I had to recreate a pool due to an invalid configuration. In that scenario I believe it was important to keep the disks in exactly the same slots they were in previously for each pool as I was using the "New Config" option. Is that important here?
October 27Oct 27 26 minutes ago, khior said:The original pool had 13 slots with a zfs filesystem, Raid Z-2. I thought the amount of disks chosen for a ZFS RAID pool are set and I couldn't add slots afterward once the pool is created. If I create a new pool with the same name and only 11 slots, will I be able to add both devices?Yes, the pool should be imported degraded the two missing devices will show with a red x, so still 13 in total, also, and FYI, raidz pools can now be expanded, but that's a different issue.27 minutes ago, khior said:I had an issue last year where I had to recreate a pool due to an invalid configuration. In that scenario I believe it was important to keep the disks in exactly the same slots they were in previously for each pool as I was using the "New Config" option. Is that important here?Not necessary with pools and the current releases; Unraid auto sorts the devices during import.
October 27Oct 27 Author Ok, it seems to have started, got a message while it was starting that "Mediafun 13 is in an error state". It now looks like this:Diags attached mediaunraid-diagnostics-20261027-1224.zip Edited October 27Oct 27 by khior
October 27Oct 27 Post new diags please, but I'm about to go offline, so I can probably only check them tomorrow.
October 27Oct 27 Author Thanks for all the help - edited last post to include diagsErrors are running up, not sure why: Edited October 27Oct 27 by khior
October 28Oct 28 Author Just to update this thread:So the good news is that I was able to backup the data from the problematic pool. Rsync reported some files with I/O errors while reading and others that didn’t pass verification, but those were in the minority. I'm happy (and relieved).It was really weird as I watched the error count of all drives in this pool climb up to a high of 152,000 per drive by the time all of the files were copied over (~3 hours). One drive in particular also got very hot (560 C).I’m not sure what your thoughts are but my guess is maybe a cable issue based on the exact same synchronized error count on all these drives. I’m guessing the HBA might not be the problem as all of these problems are on my “mediafun” pool and I also have a separate pool of 4 drives that have no problems and are being served by this same HBA (a Broadcom LSI SAS 9305-16i). I think a couple of drives are connected directly to the motherboard’s Sata port but otherwise all of the other drives are being served by this HBA and are via mini SAS to Sata breakout cables.After all the data was copied, I ended up stopping the array and shutting down my server to try to stop the error count and let the drives cool. I ordered all new breakout cables and, depending on what you recommend based on the diags, I would like to replace these cables before powering up and bringing the array online again.
October 28Oct 28 Good to hear you were able to copy most of the data off.The errors are not caused by the cables. When every disk in a raidz pool shows the same checksum error count, ZFS found blocks it could not rebuild. With two devices missing, a raidz2 pool has no redundancy left, so when a block fails its checksum ZFS cannot tell which disk returned the bad data, and it adds the error to all of them. A cable or disk problem would show READ or WRITE errors only on the affected disks, and I/O errors for those disks in the syslog. The diagnostics have none, for the whole boot. The files rsync could not read or verify are those same damaged blocks. The count went up each time they were read, by rsync and by the scrub that was also running.Those blocks are permanently damaged. Resilvering the two missing devices cannot fix them, because the resilver rebuilds the missing disks from that same data.Also, when the array started, one dataset failed to mount:cannot mount 'mediafun/photos_immich': Input/output errorSo nothing in that dataset was copied, because it was not mounted. Your photos_immich share is set to the backupandiso pool, so that dataset on mediafun may only be an old copy. Please check that nothing in it is missing from your backup.Since you now have a copy of the data, I recommend that you recreate the pool and restore the data, not resilver. Then you start with a clean pool, and not one with damaged files and a dataset that does not mount.Before that:- Replace the cables as planned. Device 12 has CRC errors and command timeouts in its SMART report, so its connection needs attention.- Check the cooling for the disk that reached 56 C. That is hot for a hard drive.- After you replace the cables, boot the server, start the array, and post the output of:zpool status -v mediafunAlso post new diagnostics. The output lists the damaged files, so you can check them against what rsync reported. The diagnostics show if any disk still logs connection errors with the new cables.
October 28Oct 28 Author Thanks again. At the moment I'm waiting for the cables to be delivered. Then I'll install them, check zpool status as well as grab new diags. I'm poking around the last set of diags now...noob question, where can I find a list of the damaged files in the diags? Also, I'm erring on the side of super caution at the moment, would it be safe to proceed to delete and recreate that pool immediately after grabbing and posting the latest diagnostics or should I wait for word back in this forum before deleting and recreating that pool in case there is something that jumps out on your end, something I'm missing?
October 28Oct 28 20 minutes ago, khior said:where can I find a list of the damaged files in the diags?Full list is not in the diags, pool needs to be imported, and you can then check with zpool status -v mediafunIt should be OK to recreate and reformat the pool immediately, but you can post diags after that to check for any ATA or other errors.
October 29Oct 29 Author Ok, cables are replaced. I started the array. This is the output of "zpool status -v mediafun" (Not including the file names listed with errors, they were just a few and replaceable). I'm also attaching a set of Diagnostics.I was reading a few different ways to correctly remove this pool to repurpose these drives. I stopped the array and erased the pool, however, I still see mediafun listed. Before starting the array again to reform the pool should I just choose "Remove Pool" or do they need to be unassigned before restarting the array?root@MediaUnraid:~# zpool status -v mediafun pool: mediafun state: DEGRADEDstatus: One or more devices has experienced an error resulting in data corruption. Applications may be affected.action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub in progress since Tue Oct 27 10:54:23 2026 24.4T / 108T scanned at 30.2G/s, 20.1T / 108T issued at 8.68M/s 8K repaired, 18.62% done, no estimated completion timeconfig: NAME STATE READ WRITE CKSUM mediafun DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 sda1 ONLINE 0 0 4 sdd1 ONLINE 0 0 4 sde1 ONLINE 0 0 4 sdr1 ONLINE 0 0 4 sdn1 ONLINE 0 0 4 sdl1 ONLINE 0 0 4 sdh1 ONLINE 0 0 4 sdj1 ONLINE 0 0 4 sdk1 ONLINE 0 0 4 sdg1 ONLINE 0 0 4 sdb1 ONLINE 0 0 4 sdn1 OFFLINE 0 0 0 7355020672882961856 FAULTED 0 0 0 was /dev/sdm1 errors: Permanent errors have been detected in the following files:mediaunraid-diagnostics-20261028-1856.zip Edited October 29Oct 29 by khior More information
October 29Oct 29 The new diagnostics look good. There are no disk or controller errors in the log for the whole boot, and the CRC count on Device 12 has not gone up since you replaced the cables. The pool status showed 8147 data errors before the erase, so recreating the pool was the right decision.You don't need to unassign the devices first. "Remove pool" is enough:- With the array stopped, click the first device of the mediafun pool and click "Remove pool".- On Main, add a new pool with the same name and 13 slots.- Assign all 13 disks. This includes the 24TB disk and the disk that was Device 13. The order is not important.- Click the first pool device, set the file system to ZFS, confirm allocation profile to raidz2, and 1 group of 13 devices. Click Apply.- Start the array. The pool will show as unmountable, which is normal for new disks. At the bottom of the Main page, check that the Format list shows only the 13 mediafun disks, then confirm and click Format.When the format is complete, restore your data and post new diagnostics, so we can confirm that the pool is healthy and that there are still no disk errors.
October 29Oct 29 Author I went through the above but after the formatting was complete the pool shows the message "Unmountable: wrong or no file system". I shut down the array and tried once more to recreate the pool with the same result. Edited October 29Oct 29 by khior More detail
October 29Oct 29 Oct 29 07:59:21 MediaUnraid root: mountpoint '/mnt/mediafun' exists and is not emptyDelete that folder or reboot and try again
October 29Oct 29 Author Didn't notice that was still there, thanks. That did it. I'm copying over my data now. new daigs attached just in case needed. mediaunraid-diagnostics-20261029-0908.zip
October 29Oct 29 The new diagnostics look good. The mediafun pool is online with all 13 devices, there are no errors, and the log for the whole boot shows no disk or controller errors. The CRC count on the 24TB disk has not changed since you replaced the cables.While the restore is running, keep an eye on the disk temperatures, especially the one that got hot before. After the restore is complete, I suggest you run a scrub to confirm that all is good.
October 30Oct 30 Author Thanks again for all the assistance. I have another unraid box that I have been backing up to for some time and the restore was going well for about the last 24 hours using rsync until just about now, about 17TB copied. I had a disk in that pool that was displaying SMART errors but it seemed ok until now.I noticed the speed of the restore just reduced sharply and the error count on that one drive has skyrocketed. Reads and writes are fluctuating between 0kb and a couple hundred KBs while the error count climbs and I'm getting notices that it's failing. Not sure if I should stop the sync, kick out that drive or let it run. I'm leaning toward stopping it as much as I don't want to. I do have a spare 16TB drive for that system as well I could swap out and let it resilver if needed.(Edit): After letting it run for a bit more I think it's fairly obvious that I need to replace that drive...I'll:Stop the backup array (K-Unraid)Unassign that diskStart the backup arrayShut down the backup arrayreplace the diskstart the backup array and let it rebuild.I don't see this progressing otherwise. I'm a bit gun shy as what I thought was a simple disk replace before is what kicked off this whole mess, and now my data is only in one spot.Diags from that other system are attached if you have the time...k-unraid-diagnostics-20261030-0824.zip Edited October 30Oct 30 by khior
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.