UNRAID 6.12.4 CRASHING WITH KERNAL PANIC- NEED ASSISTANCE

[Mirrormir]

Hi everyone,

Been an unRAIDer for about 9 month now and have not had any issues until I converted my file system to ZFS. Now the server will only be up for one to two day before I find it had crashed with a kernel panic.  Having already spent some time looking at similar issues on the forum here I think I have corruption in the zpools. Now the array will not start it gets stuck "Mounting Drives".

 

As alway I have attached diagnostic report and a syslog.

Any assistance would be greatly appreciated. 

locutus-diagnostics-20231028-0943.zip locutus-syslog-20231028-0849.zip

[JorgeB]

Enable the mirror to flash drive on the syslog server and post that after an array start.

[Mirrormir]

Thanks for the quick reply, as requested I have attached the new syslog.

locutus-syslog-20231028-1053.zip

[JorgeB]

Disk1 filesystem is crashing on mount, see if it can be mounted read-only:

 

zpool import -o readonly=on disk1

 

[Mirrormir]

Hi JorgeB, 

 

Ran the command in unraid terminal but did not and reply just a flashing cursor.

[JorgeB]

Make sure you reboot first if you had already tried to start the array, if it doesn't mount in read-only mode after a fresh boot best bet is to re-format the disk and restore the data from a backup, also a good idea to run memtest.

[Mirrormir]

Okay, just rebooted but that results in the server getting stuck "Starting Diagnostics collection......."

From here it will not reboot or shutdown also collects no data.

[JorgeB]

You need to disable array auto-start, or it will try to mount the disk and fail.

[Mirrormir]

Just rebooted with the array stopped and ran the command as before, this time I got the cursor back but no change of state to the drive.

 

zpool status gives this output:

 

root@Locutus:~# zpool status
  pool: disk1
 state: ONLINE
  scan: scrub repaired 0B in 03:14:41 with 0 errors on Sun Oct 22 05:14:42 2023
config:

        NAME        STATE     READ WRITE CKSUM
        disk1       ONLINE       0     0     0
          sdd1      ONLINE       0     0     0

errors: No known data errors
root@Locutus:~# 

Edited October 28, 2023 by Mirrormir

[JorgeB]

The pool was imported read-only, you can now start the array, assuming there aren't issues with other filesystems, disk1 will still show unmountable in the GUI but the data will be accessible under /mnt/disk1, you can copy everything to another disk/pool and re-format disk1.

[Mirrormir]

Thanks Jorge,

 

I have been doing some further investigation using the zpool and zfs commands and found that there is indeed a corrupted file on disk3 that I would like to remove and replace.

Some of my backups are a little out of date as this issue has been going on since the end of September and ever time I tried to run a backup the system would crash with a kernel panic.

[Mirrormir]

I've posted and attached the status of all my pools.

zpool status 29-10-2023.rtf

[Mirrormir]

Hi jorge,

I have started the array, witch took a long time to start but now it tells me "Unmountable or Unsupported or no file system:" and that's the whole array.

[JorgeB]

36 minutes ago, Mirrormir said:

"Unmountable or Unsupported or no file system:"

Like mentioned this is normal for disk1

1 hour ago, JorgeB said:

disk1 will still show unmountable

 

 

 

36 minutes ago, Mirrormir said:

and that's the whole array.

What do you mean? If other disks are unmountable post new diags.

[Mirrormir]

To save my data I shutdown the server after restarting disk 1 is now showing as missing with a red cross.

[Mirrormir]

In an attempt to bring disk1 back on-line I made a new config but now it's not showing up, so seems to dead.

New diagnostics uploaded.

locutus-diagnostics-20231029-1116.zip

[JorgeB]

1 hour ago, Mirrormir said:

To save my data I shutdown the server after restarting disk 1 is now showing as missing with a red cross.

That's unrelated to the previous issue, check power/connections.

 

45 minutes ago, Mirrormir said:

In an attempt to bring disk1 back on-line I made a new config but now it's not showing up, so seems to dead.

That was a bad idea, now you cannot emulated the missing disk.

[Mirrormir]

Okay my fault so what the best way forward from here?

[JorgeB]

Does the disk come back online if you swap cables with known good ones? Did you star the array after the new config? If not it's still possible to re-enable parity, you need a working disk to replace the missing one, same size or larger.

[Mirrormir]

All the disk cable used in the Array are connected with SFF 8087 to SATA from a SAS HBA. Disk1 however has turned up under Unassigned Devices for some reason.

[Mirrormir]

And no I didn't start the Array.

[JorgeB]

1 hour ago, Mirrormir said:

Disk1 however has turned up under Unassigned Devices for some reason.

You can then re-assign disk1 and check parity is already valid before starting the array, start in maintenance mode to avoid the crash due to disk1 filesystem, then repeat the procedure above mo mount disk1 read only and start the array normally, post new diags after that.

[Mirrormir]

Thanks Jorge,

Just to make sure I get this right, I will need to remove disk1 from the Unassigned Devices and then assign it back in the Array as disk 1.

Is that correct?...

[JorgeB]

Yes, if you didn't start the array after the new config all devices will have a blue icon and you can still add disk1, don't forget to check "parity is already valid " and " maintenance mode " before array start, any doubts ask. 

[Mirrormir]

Thanks Jorge,

Sound good parity is showing valid.