I hope I haven't made a major mistake with my qualified ignorance...

Situation is as follows: Unraid array of five data disks and two parity disks has a problem with Disk4. The disks on the array are formatted with ZFS. Zpool status shows that there is an error on the disk and recommends restoring a data backup. I cannot identify a specific file that is corrupt. The error is:

"One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
config:

        NAME        STATE     READ WRITE CKSUM
        disk4       ONLINE       0     0     0
          md4p1     ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        disk4:<0x5af494>"

A specific file cannot be identified that contains the error.

Two other issues seem to be related to the same error:

Unraid says when booting up on the console: "VERIFY3(size <= rt->rt_space) failed (281442899902464 <= 14951145472) PANIC at range_tree.c:436:range_tree_remove_impl()"

And the last problem, the array can no longer be stopped. Unraid always hangs when shutting down and then has to be hard reset.

My attempted solutions:

First, I moved all important data from Disk4 to another of the disks. That worked without any problems. My next action was to run a parity check in the hope that this would correct errors. Unfortunately without success. The parity check went through about 95 percent of the time and then hung for many hours with the message saying that it needed another two hours to complete. But it didn't do anything anymore.

My next attempt at a solution is based on experiences from the time before I reformatted the disks to ZFS. The idea now was to simply pull out the problematic disk and then start a rebuild with a replacement disk ready. So I restarted Unraid with disk 4 missing. I checked the note that I wanted to replace disk 4 as soon as possible and started the array. At this point I would have expected a running array with emulated Disk4. Now the GUI has been showing "Starting..." for more than an hour and all disks are shown at the top with a green dot, only Disk4 says "not installed". The status line in the window says "Array is starting. Disks are being mounted".

At this point I don't know what to do next. Can I cancel the whole thing? I probably have to, because it's obviously not possible to mount it. What can I do without causing greater damage (if it hasn't already happened)?

Edited December 3, 20232 yr by paulinus
little addition

26 minutes ago, paulinus said:

disk4:<0x5af494>"

This usually means metadata corruption, if you already copied the data suggest reformatting the disk and restoring it, also a good idea to run memtest, bad RAM can cause this type of issues.

In any case, I will now have to hard reset the computer again. Then I will only be able to start it in maintenance mode, since it stops when the disks are mounted. Can I just reformat Disk4 in maintenance mode? What then happens to the parity data?

Before I took disk 4 out of the array, Unraid started and was able to mount the array. Other than the reported error messages, everything seemed to work. It could also be written to and read from Disk 4.

Edited December 3, 20232 yr by paulinus
Addition

I have now replaced the memory modules and inserted the previous disk 4 back into the array. After starting, Unraid offered me to restore disk 4. I have confirmed this. I've been waiting for the restore to start for ten minutes now. 

The status line in the GUI window says "Array is starting, disks are being mounted". It looks like the server is hanging again.

It can't be the memory modules, I just replaced them. What can I do next?

Syslog shows the following lines:

Dec  3 20:37:29 Gandalf kernel: VERIFY3(size <= rt->rt_space) failed (281442899902464 <= 14951145472)
Dec  3 20:37:29 Gandalf kernel: PANIC at range_tree.c:436:range_tree_remove_impl()
Dec  3 20:37:29 Gandalf kernel: Showing stack for process 22643
Dec  3 20:37:29 Gandalf kernel: CPU: 0 PID: 22643 Comm: zpool Tainted: P           O       6.1.64-Unraid #1
Dec  3 20:37:29 Gandalf kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014
Dec  3 20:37:29 Gandalf kernel: Call Trace:

 

Unraid still says "Mounting disks...", since more then two hours now. I can access the discs using the console. However, the shares are not available. Does anybody have an idea, what I can do? 

 

Since the filesystem is panicking on import it can hang Unraid, if all the data is already copied, set the filesystem for that disk to a different one to start the array normally, you can then format the disk, again, if all data is already copied.

So that I can give the hard drive a different format, I assigned it to a different slot. So I could set btrfs and when starting the array I could set the hard disk to be formatted.
 

Unfortunately, Unraid still panics when i am staring the array. It even panics when I remove the hard drive with the problem from the system completely. I guess I didn't understand the process properly. What am I doing wrong?

Edited December 4, 20232 yr by paulinus

9 hours ago, paulinus said:

So that I can give the hard drive a different format, I assigned it to a different slot.

You don't need to change the slot, if enabled disable array auto start, reboot, click on that disk on main and change the filesystem to a different one, start the array, format the disk.

Unfortunately I can't format the disk. Whatever I do, as soon as I start the array the computer hangs. Even if I completely removed the affected disk, the computer hangs.

I have currently completely removed Disk4 from the system. I started the array in the hope that it would boot up and maybe emulate Disk4 or even forget it. Both would be ok, but unfortunately Unraid has been saying "Mounting disks..." since yesterday evening. As soon as I start the array, "VERIFY3(size <= rt->rt_space) failed (281442899902464 <= 14951145472)" and "PANIC at range_tree.c:436:range_tree_remove_impl()" appear on the console.

The existing discs themselves seem ok. I can access it in the console and also through the GUI I can access the directories of the existing disks. They are just not available as shares.

Edited December 4, 20232 yr by paulinus

Is it possible that Unraid would just have to recreate the configuration of the array? If it had to recreate the parity in the process, that would be ok with me. Finally I had moved all important data from Disk4 to another disk, there is only data on Disk4 that can be deleted.

27 minutes ago, paulinus said:

Unfortunately I can't format the disk. Whatever I do, as soon as I start the array the computer hangs. Even if I completely removed the affected disk, the computer hangs.

It can still be emulating that disk, post new diags before array start.