UNRAID 6.12.4 STILL CRASHING WITH KERNAL PANIC- NEED FUTHER ASSISTANCE

[Mirrormir]

@JorgeB RE: UNRAID 6.12.4 CRASHING WITH KERNAL PANIC- NEED ASSISTANCE,

 

My previous post.

As of last Tuesday all was working fine but on Wednesday around the Kernal panic issue retuned and the system halted.  At that point I had to switch the server off and it's been off for the last couple of day, I have just powered it back on into maintenance mode as I though it would need to do a Corrective Parity Check but it has not as the Parity is showing as VALID which is strange as I has a "unclean shutdown detected"

 

I would like to find out what is the root cause of the crashes I am thinking it might be an add-on or a docker issue.

locutus-diagnostics-20231104-0715.zip

Edited November 4, 2023 by Mirrormir

[JorgeB]

2 hours ago, Mirrormir said:

Parity is showing as VALID which is strange as I has a "unclean shutdown detected"

This is normal, valid in this case just means no disk is disabled.

 

For the crashing you can enable the syslog server and post that after a crash to see if there's something there.

 

 

[Mirrormir]

I have just had my share that was active disappear so I have posted a syslog that does have an error, looks like there maybe an other currupted file.

locutus-syslog-20231104-1245.zip

[Mirrormir]

Hi Guys,

 

System crashed again so as requested I have attached the syslog file from the flash drive.

 

I've restarted unRAID and put it into maintenance mode but it seem happy that the Parity is okay so I will put it back into normal mode and see what happens.

 

Lets hope you find a clue in the logs,

Also just noticed that it was running a scheduled task - see attached picture.

syslog

Edited November 5, 2023 by Mirrormir

[Mirrormir]

H Guys 

 

anyone arround to look at my issue?

[JorgeB]

5 minutes ago, Mirrormir said:

anyone arround to look at my issue?

Please have some patience, people don't spend 24 hours in the forum, also need to account for different time zones.

 

Nov  4 23:35:40 Locutus kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Nov  4 23:35:40 Locutus kernel: ? _raw_spin_unlock+0x14/0x29
Nov  4 23:35:40 Locutus kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

[Mirrormir]

Sorry Jorge you are right.

 

Okay will shutdown unRAID and switch to ipvlan.

 

Many Thanks

[Mirrormir]

Same old issue after 2 day or so I've attached the syslog again and latest diagnostics.

I was hopping the changing the docer network would help but it appears not.

 

syslog_7-11-23 locutus-diagnostics-20231108-0651.zip

[JorgeB]

Nov  5 17:49:33 Locutus kernel: VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (36028797018963967 < 32768)
Nov  5 17:49:33 Locutus kernel: PANIC at zio.c:341:zio_data_buf_alloc()
Nov  5 17:49:33 Locutus kernel: Showing stack for process 7201
Nov  5 17:49:33 Locutus kernel: CPU: 8 PID: 7201 Comm: dsl_scan_iss Tainted: P        W  O       6.1.49-Unraid #1
Nov  5 17:49:33 Locutus kernel: Hardware name: ASUS All Series/X99-E, BIOS 2101 07/10/2019
Nov  5 17:49:33 Locutus kernel: Call Trace:
Nov  5 17:49:33 Locutus kernel: <TASK>
Nov  5 17:49:33 Locutus kernel: dump_stack_lvl+0x44/0x5c
Nov  5 17:49:33 Locutus kernel: spl_panic+0xd0/0xe8 [spl]
Nov  5 17:49:33 Locutus kernel: ? spl_kmem_cache_alloc+0x45/0x5fd [spl]
Nov  5 17:49:33 Locutus kernel: ? percpu_counter_add_batch+0x85/0xa2
Nov  5 17:49:33 Locutus kernel: ? spl_kmem_cache_alloc+0x5d0/0x5fd [spl]
Nov  5 17:49:33 Locutus kernel: ? spl_kmem_cache_destroy+0x196/0x1b7 [spl]
Nov  5 17:49:33 Locutus kernel: ? percpu_counter_add_batch+0x85/0xa2
Nov  5 17:49:33 Locutus kernel: zio_data_buf_alloc+0x41/0x54 [zfs]
Nov  5 17:49:33 Locutus kernel: abd_alloc_linear+0x63/0x8f [zfs]
Nov  5 17:49:33 Locutus kernel: scan_exec_io+0x58/0x202 [zfs]
Nov  5 17:49:33 Locutus kernel: ? ktime_get_raw_ts64+0x35/0x64
Nov  5 17:49:33 Locutus kernel: scan_io_queue_issue+0x17b/0x2f2 [zfs]
Nov  5 17:49:33 Locutus kernel: scan_io_queues_run_one+0x460/0x586 [zfs]
Nov  5 17:49:33 Locutus kernel: taskq_thread+0x269/0x38a [spl]
Nov  5 17:49:33 Locutus kernel: ? wake_up_q+0x44/0x44
Nov  5 17:49:33 Locutus kernel: ? taskq_dispatch_delay+0x106/0x106 [spl]
Nov  5 17:49:33 Locutus kernel: kthread+0xe7/0xef
Nov  5 17:49:33 Locutus kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov  5 17:49:33 Locutus kernel: ret_from_fork+0x22/0x30
Nov  5 17:49:33 Locutus kernel: </TASK>

 

One of you zfs pools is panicking, you should backup and re-format, also a good idea to run memtest.

[Mirrormir]

Thanks Jorge,

 

Would a "zpool status" help with finding out which pool it is?

 

I've just run the above command and there is someting strange on disk1 again.

zpool status 08-11-2023.rtf

Edited November 8, 2023 by Mirrormir

[JorgeB]

I'm getting an error opening that file, please post directly in the forum using code tags or in a plain text file.

[Mirrormir]

Sorry about that that Mac text editor for you it only make .rtf files

 

New file attached this time .txt

 

zpool status 8-11-23.txt

[JorgeB]

Disk1 is showing some likely metadata corruption, recommend backup and re-formatting that disk, also good idea to run memtest.