1 minute ago, JorgeB said:

Did you save the diags after the read errors? I had asked for them previously.

The current problem looks like a possible filesystem issue, see if you can get the syslog:

cp /var/log/syslog /boot/syslog.txt

Also, it may be a good idea to run memtest to rule out any obvious RAM issues.

Hi. I was not able to save the log because as i mentioned the server hanged and became unresponsive in the morning after leaving it parity-synching overnight. i had to do a shutdown using the power button. i hope i got the correct file you asked for, its from the root dir of the boot flash drive.

syslog.txt

Try starting the array again, and if it GUI hangs, use the CLI to get the syslog, the command above will copy it to the flash drive.

1 minute ago, JorgeB said:

Try starting the array again, and if it GUI hangs, use the CLI to get the syslog, the command above will copy it to the flash drive.

ok will try now, just finishing up with the memtest, no errors or issue so far

10 minutes ago, JorgeB said:

Try starting the array again, and if it GUI hangs, use the CLI to get the syslog, the command above will copy it to the flash drive.

Hi. Here's the file.

syslog.txt

Hi.

I disabled DISK3 and the array is starting now without the error. Just want to know what does that error mean?

Jul 9 19:52:31 McKanor emhttpd: disk3: recover volume

Jul 9 19:52:31 McKanor emhttpd: /sbin/blkid /dev/md3p1 2>&1

Jul 9 19:52:31 McKanor emhttpd: /dev/md3p1: LABEL="disk3" UUID="12670620897066880581" UUID_SUB="4980010982240981706" BLOCK_SIZE="4096" TYPE="zfs_member"

Jul 9 19:52:31 McKanor emhttpd: disk3: zfs recover pool

Jul 9 19:52:31 McKanor emhttpd: shcmd (92): /usr/sbin/zpool import -f -m -N -d /dev/md3p1 12670620897066880581 disk3

Jul 9 19:52:35 McKanor kernel: VERIFY0(dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs, &dbp)) failed (0 == 5)

Jul 9 19:52:35 McKanor kernel: PANIC at dmu.c:1304:dmu_write()

Jul 9 19:52:35 McKanor kernel: Showing stack for process 12394

Jul 9 19:52:35 McKanor kernel: CPU: 3 UID: 0 PID: 12394 Comm: txg_sync Tainted: P O 6.12.24-Unraid #1

Jul 9 19:52:35 McKanor kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE

Jul 9 19:52:35 McKanor kernel: Hardware name: Dell Inc. Precision Tower 5810/0WR1RF, BIOS A34 10/19/2020

Jul 9 19:52:35 McKanor kernel: Call Trace:

Jul 9 19:52:35 McKanor kernel: <TASK>

Jul 9 19:52:35 McKanor kernel: dump_stack_lvl+0x47/0x70

Jul 9 19:52:35 McKanor kernel: spl_panic+0xcf/0xf0 [spl]

Jul 9 19:52:35 McKanor kernel: ? dmu_buf_hold_array_by_dnode+0x240/0x460 [zfs]

Jul 9 19:52:35 McKanor kernel: ? preempt_latency_start+0x2b/0x50

Jul 9 19:52:35 McKanor kernel: ? rawspin_lock+0x13/0x20

Jul 9 19:52:35 McKanor kernel: ? rawspin_unlock+0x14/0x30

Jul 9 19:52:35 McKanor kernel: ? dnode_rele_and_unlock+0x67/0xc0 [zfs]

Jul 9 19:52:35 McKanor kernel: dmu_write+0x7c/0x100 [zfs]

Jul 9 19:52:35 McKanor kernel: ? dmu_buf_will_dirty_impl+0x99/0x180 [zfs]

Jul 9 19:52:35 McKanor kernel: space_map_write+0x1ab/0x600 [zfs]

Jul 9 19:52:35 McKanor kernel: ? space_map_open_impl+0x46/0x80 [zfs]

Jul 9 19:52:35 McKanor kernel: metaslab_flush+0x1ba/0x330 [zfs]

Jul 9 19:52:35 McKanor kernel: ? spa_generate_syncing_log_sm+0x1f1/0x220 [zfs]

Jul 9 19:52:35 McKanor kernel: spa_flush_metaslabs+0x2c3/0x340 [zfs]

Jul 9 19:52:35 McKanor kernel: spa_sync+0x81b/0xd80 [zfs]

Jul 9 19:52:35 McKanor kernel: txg_sync_thread+0x1ea/0x390 [zfs]

Jul 9 19:52:35 McKanor kernel: ? thread_generic_wrapper+0x4b/0x70 [spl]

Jul 9 19:52:35 McKanor kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]

Jul 9 19:52:35 McKanor kernel: thread_generic_wrapper+0x59/0x70 [spl]

Jul 9 19:52:35 McKanor kernel: ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]

Jul 9 19:52:35 McKanor kernel: kthread+0xef/0x100

Jul 9 19:52:35 McKanor kernel: ? __pfx_kthread+0x10/0x10

Jul 9 19:52:35 McKanor kernel: ret_from_fork+0x24/0x40

Jul 9 19:52:35 McKanor kernel: ? __pfx_kthread+0x10/0x10

Jul 9 19:52:35 McKanor kernel: ret_from_fork_asm+0x1a/0x30

Jul 9 19:52:35 McKanor kernel: </TASK>

Seeing the pools back. 😀

i think all my hdd are dying ☹️☠️ one after the other

zfs is crashing on mount, doesn't look like a disk problem for now, but post a SMART report.

IIRC this is not the first disk with filesystem issues, they can be the result of an underlying hardware issue.

3 minutes ago, JorgeB said:

zfs is crashing on mount, doesn't look like a disk problem for now, but post a SMART report.

IIRC this is not the first disk with filesystem issues, they can be the result of an underlying hardware issue.

ok will gather the smart reports for each disk

Hi.

It was an enlightening last 7 hours, but i think i (and with the important help from you guys) finally is closer to cracking this mystery read errors.

I found out that i was using a SATA expansion card whose chip is not in the recommended list for UNRAID 🤯😤🤬😭. I have not replaced it yet, just running 4 drives + 1 parity for now while i am copying the most important files that i can access until i replace this SATA card with a recommended one. I have ordered now on ebay the card, which explicitly says for UNRAID.

I hope there is still hope for the other drives that i have disabled once i get the recommended SATA card.

12-Port SA3112J: ASM1064*1 + JMB575*2 these are the chips on the SATA card i am using now, but will replace later. here's the latest diag.

mckanor-diagnostics-20250710-0643.zip

That could be the issue, also recommend uninstalling the Nvidia driver since it's spamming the log.

Just now, JorgeB said:

That could be the issue, also recommend uninstalling the Nvidia driver since it's spamming the log.

will do. thanks