I had a drive catastrophically fail on me, i saw it coming a few weeks ago and ordered 16TB replacement drives knowing id need to use the parity swap procedure.

I precleared the new drive which took 20 hours, and when swaping parity into drive 3 and the new drive into parity, when i click copy i get this error in the syslog. Ive tried to look up the log error with no luck for a solution. I thought it may be the PCIE controller (SAS2008 using sata cables) so i swaped it to the on board Sata connection with the same result.

any insight on how to fix this would be appreciated.

Diagnostics attached
 

Nov 26 18:44:09 Reaver kernel: md: import_slot: 29 empty
Nov 26 18:44:09 Reaver  emhttpd: import 30 cache device: (nvme1n1) PCIe_SSD_23010350001732
Nov 26 18:44:09 Reaver  emhttpd: import 31 cache device: (nvme2n1) PCIe_SSD_23010350001727
Nov 26 18:44:09 Reaver  emhttpd: import 32 cache device: (nvme4n1) PCIe_SSD_23010350001693
Nov 26 18:44:09 Reaver  emhttpd: import 33 cache device: (nvme0n1) ADATA_SX6000LNP_2K3429Q9J2TF
Nov 26 18:44:09 Reaver  emhttpd: import 34 cache device: (nvme3n1) PCIe_SSD_23010350001738
Nov 26 18:44:09 Reaver  emhttpd: import flash device: sda
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/nvme1n1
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sdg
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/nvme4n1
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sdd
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/nvme0n1
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sde
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sdb
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sdf
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sdc
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/nvme3n1
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/nvme2n1
Nov 26 18:44:09 Reaver  emhttpd: read SMART /dev/sda
Nov 26 18:44:14 Reaver  emhttpd: writing GPT on disk (sdg), with partition 1 byte offset 32KiB, erased: 0
Nov 26 18:44:14 Reaver  emhttpd: shcmd (197): sgdisk -Z /dev/sdg
Nov 26 18:44:14 Reaver root: Problem opening S#004 for reading! Error is 2.
Nov 26 18:44:14 Reaver root: The specified file does not exist!
Nov 26 18:44:14 Reaver root: Warning! Unable to generate a proper UUID! Creating an improper one as a last
Nov 26 18:44:14 Reaver root: resort! Windows 7 may crash if you save this partition table!#007
Nov 26 18:44:14 Reaver root: Problem opening '' for writing! Program will now terminate.
Nov 26 18:44:14 Reaver root: Warning! MBR not overwritten! Error is 2!
Nov 26 18:44:14 Reaver  emhttpd: shcmd (197): exit status: 2
Nov 26 18:44:14 Reaver  emhttpd: shcmd (198): sgdisk -o -a 8 -n 1:32K:0 /dev/sdg
Nov 26 18:44:14 Reaver root: The specified file does not exist!
Nov 26 18:44:14 Reaver root: Warning! Unable to generate a proper UUID! Creating an improper one as a last
Nov 26 18:44:14 Reaver root: resort! Windows 7 may crash if you save this partition table!#007
Nov 26 18:44:14 Reaver root: Information: Creating fresh partition table; will override earlier problems!
Nov 26 18:44:14 Reaver root: Caution! Secondary header was placed beyond the disk's limits! Moving the
Nov 26 18:44:14 Reaver root: header, but other problems may occur!
Nov 26 18:44:14 Reaver root: Unable to open device '' for writing! Errno is 2! Aborting write!
Nov 26 18:44:14 Reaver  emhttpd: shcmd (198): exit status: 4
Nov 26 18:44:14 Reaver  emhttpd: shcmd (199): udevadm settle

 

reaver-diagnostics-20241126-1844.zip

The log indicates an issue with the Parity Swap procedure during the "copy" step. Specifically, the errors suggest that the system is unable to correctly initialize or partition the new 16TB drive intended for parity. Here's a breakdown and steps to address the issue:

Analysis of Errors

 

Key Errors in the Log:

Problem opening S#004 for reading! Error is 2.

This suggests the system is attempting to access a file or device but cannot find it (Error 2 = "No such file or directory").

Unable to generate a proper UUID!

Indicates issues creating a unique identifier for the drive, likely due to initialization errors.

Caution! Secondary header was placed beyond the disk's limits!

This may occur if the drive's geometry or reported capacity is being misinterpreted.

 

Potential Causes:

Drive Initialization Issues:

The new 16TB drive may not be correctly recognized or initialized.

SAS Controller or Cabling Problems:

The LSI SAS2008 controller may be causing compatibility or communication issues.

Formatting or Partitioning Error:

The system is unable to create a partition table or properly format the drive.

File System or Disk Labeling Issues:

The drive may have residual partitioning data causing conflicts during the parity swap.

 

solutions:
 

Power off check cables!

Verify Hardware and Connections

Ensure the 16TB drive is connected securely.

Test the drive on a different SATA port or controller (e.g., move it from the SAS2008 controller to an onboard SATA port, as you've tried).

Swap the SATA or SAS cables to rule out cable-related issues.

 

Test the New Drive

Run a SMART test to check the health of the drive:

smartctl -t long /dev/sdg
smartctl -a /dev/sdg
*can be done in web Ui...

Look for signs of drive failure, such as bad sectors or read errors.

 

 

Manually Zero the Disk

*I never got preclear to work Warrnign DD is great and pwerfull tool! this will wipe data!

Residual data or partitioning could be causing issues. Zero out the first few sectors of the drive:
 

dd if=/dev/zero of=/dev/sdg bs=1M count=100

This will erase any partition tables or file system remnants.

 

Re-Initialize the Drive

Manually create a new partition table:

sgdisk --zap-all /dev/sdg
sgdisk -o /dev/sdg

* looked like sdg... can be dopne in web ui...

 

Then retry the parity swap procedure. and Monitor Logs post berfore after diag...

 

Alternative Troubleshooting Steps

Use a Different Drive Port

If the issue persists, connect the drive to a different port (SATA or another SAS controller) to rule out compatibility issues with the SAS2008 controller.

Test Drive in Isolation

Boot into a live Linux environment or a different Unraid instance, connect only the new drive, and test whether it can be formatted and initialized independently.

Check for Controller Firmware Updates

Ensure your LSI SAS2008 controller has the latest firmware installed. Older firmware versions may cause compatibility issues with newer, high-capacity drives.

Make sure the LSI hba is in IT Mode!

 

Next Steps

Apply the above troubleshooting steps.

Restart the parity swap procedure after resolving the errors.

If issues persist, provide the SMART test results and any additional syslog errors for further assistance.

Thanks for the reply and detailed explanation of the errors, im gonna run the long test and get back to you.

Is there something else different with Exos drives that ive overlooked? (for reference X24 ST16000NM000H 16TB 7.2K RPM SATA 6Gb/s 512e) From my research these should act fairly normal in this setup.

Also is there something in a newer version of unraid that may help here? im still on 6.11.2, though im not sure upgrading unraid right now will have an effect on my array. last thing i need is to lose the array. some of the data is not backed up so im trying not to lose anything in this process. though, the drive that failed didnt have much of importance on it so i guess i could always pull the data from the drives into a new array later if i really needed to, but id like to avoid that if at all possible.

 

i have 2 16TB Exos drives hooked in at the moment, 1 on the sas controller and 1 on the motherboard. gonna run the smart test on both. the short tests all looked ok before i started pre clearing, both seemed to zero out properly like ive normally done in the past.

 

What i cant figure out is no matter the cable i hook in, or new drive i try, i get similar results. I found a thread on Exos drives, and disabled the EPC and Power balance features with the CLI utility, so maybe that will help as well. i ran the commands on both drives for the DD and got the same results. Preclear zeroing worked on both drives, so im kinda at a loss.

I have 2 other drives i got with this order. i will have a try at them tomorrow. it seems wild to me that these drives could potentially all be DOA. I also have a Ironwolf that has some smart errors but isnt completely shot, i may try to see if i get the same issues with it to narrow down the root cause.

 

root@Reaver:~# dd if=/dev/zero of=/dev/sdi bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.478121 s, 219 MB/s
root@Reaver:~# sgdisk --zap-all /dev/sdi
Problem opening S for reading! Error is 2.
The specified file does not exist!
Warning! Unable to generate a proper UUID! Creating an improper one as a last
resort! Windows 7 may crash if you save this partition table!
Problem opening '' for writing! Program will now terminate.
Warning! MBR not overwritten! Error is 2!

 

Edited November 27, 20241 yr by DracoCadendi

I do recommend updating to the latest stable version of 6.12.13

exos can be difficult as this was during the new hardware implementation of smr vs cmr disk drivers. you want cmr drives.
It is hard to tell when buying that line up of which ones are cmr/smr untill the product is in hand.

https://www.buffalotech.com/resources/cmr-vs-smr-hard-drives-in-network-attached-storage-nas-msp#:~:text=With a CMR drive%2C data,handle more random write operations.&text=SMR is a relatively new,read tracks on a drive.

 

 

EDIT 2:

Updating the unraid version has fixed this issue. GPT fdisk (gdisk) version 1.0.9.1 solves this particular issue.

 

EDIT:

Just to document, looks like gdisk 1.0.9 this was a bug with partition generation so hopefully the new version of unraid has the fix version for it.

https://bugs.gentoo.org/844073

 

Original:

8 hours ago, bmartino1 said:

exos can be difficult as this was during the new hardware implementation.
It is hard to tell when buying that line up of which ones are cmr/smr untill the product is in hand.

Not sure i quite understand what you mean by new hardware implementation, but half the point of buying exos drives is they are CMR, I wouldnt put anything thing else but CMR drives into an array at this point anyway.

Either way, was more asking if there were any special flags needed for the drive to be formatted properly.

SMART came back completely clean, so the drives themselves are fine. im having the feeling this is an unraid software/kernel issue, so going to try the upgrade of the OS, hopefully that dosnt have any major effects on my array in its current state.

Edited November 28, 20241 yr by DracoCadendi
Fixed issue and reformating

6.11.2 has a known bug, it won't format any > 2TB drive, just needed to update the OS.

 

https://docs.unraid.net/unraid-os/release-notes/6.11.3/

 

Version 6.11.3 2022-11-08

This release is focused on bug fixes and minor improvements. In particular, we need to revert a base library due to a bug which prevents formatting devices >2TB in size.

Since apparently im unable to create a new topic, adding here.

 

copy step keeps freezing, once at 86% and this time at 54%.

 

Nov 30 12:41:45 Reaver kernel: ------------[ cut here ]------------
Nov 30 12:41:45 Reaver kernel: kernel BUG at fs/buffer.c:2914!
Nov 30 12:41:45 Reaver kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Nov 30 12:41:45 Reaver kernel: CPU: 29 PID: 712 Comm: kswapd0 Tainted: P           O       6.1.118-Unraid #1
Nov 30 12:41:45 Reaver kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS MASTER/TRX40 AORUS MASTER, BIOS F5p 02/19/2021
Nov 30 12:41:45 Reaver kernel: RIP: 0010:free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: Code: e0 d9 7e 85 c0 75 05 0f 1f 44 00 00 48 89 d8 5b e9 94 33 98 00 0f 1f 44 00 00 48 8b 57 48 48 8d 47 48 48 89 fe 48 39 c2 74 02 <0f> 0b 48 8b 3d 56 2b 13 01 e8 3c 74 fa ff bf 01 00 00 00 e8 a0 b8
Nov 30 12:41:45 Reaver kernel: RSP: 0018:ffffc9000628fa78 EFLAGS: 00010202
Nov 30 12:41:45 Reaver kernel: RAX: ffff88964aad8660 RBX: ffff88964aad85b0 RCX: 000000008027001d
Nov 30 12:41:45 Reaver kernel: RDX: ffff88964aad8661 RSI: ffff88964aad8618 RDI: ffff88964aad8618
Nov 30 12:41:45 Reaver kernel: RBP: 0000000000000001 R08: ffff88964aad8680 R09: 000000008027001d
Nov 30 12:41:45 Reaver kernel: R10: ffff88964aad8680 R11: ffff88851bfe6460 R12: ffff88964aad8680
Nov 30 12:41:45 Reaver kernel: R13: ffff88810a181708 R14: 0000000000000001 R15: ffffea005ad1f6c8
Nov 30 12:41:45 Reaver kernel: FS:  0000000000000000(0000) GS:ffff889fcd740000(0000) knlGS:0000000000000000
Nov 30 12:41:45 Reaver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 12:41:45 Reaver kernel: CR2: 0000153ba13f2484 CR3: 000000014d07e000 CR4: 0000000000350ee0
Nov 30 12:41:45 Reaver kernel: Call Trace:
Nov 30 12:41:45 Reaver kernel: <TASK>
Nov 30 12:41:45 Reaver kernel: ? __die_body+0x1a/0x5c
Nov 30 12:41:45 Reaver kernel: ? die+0x30/0x49
Nov 30 12:41:45 Reaver kernel: ? do_trap+0x7b/0xfe
Nov 30 12:41:45 Reaver kernel: ? free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: ? free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: ? do_error_trap+0x6e/0x98
Nov 30 12:41:45 Reaver kernel: ? free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: ? exc_invalid_op+0x4c/0x60
Nov 30 12:41:45 Reaver kernel: ? free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 30 12:41:45 Reaver kernel: ? free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 12:41:45 Reaver kernel: try_to_free_buffers+0x9f/0xc8
Nov 30 12:41:45 Reaver kernel: shrink_folio_list+0x7ae/0x996
Nov 30 12:41:45 Reaver kernel: ? cgroup_rstat_updated+0x21/0xa5
Nov 30 12:41:45 Reaver kernel: shrink_lruvec+0x61a/0x9b5
Nov 30 12:41:45 Reaver kernel: shrink_node+0x31b/0x588
Nov 30 12:41:45 Reaver kernel: balance_pgdat+0x4e9/0x6a2
Nov 30 12:41:45 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 12:41:45 Reaver kernel: ? newidle_balance+0x289/0x30a
Nov 30 12:41:45 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 12:41:45 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 12:41:45 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 12:41:45 Reaver kernel: ? finish_task_switch.isra.0+0x140/0x218
Nov 30 12:41:45 Reaver kernel: kswapd+0x2f0/0x333
Nov 30 12:41:45 Reaver kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Nov 30 12:41:45 Reaver kernel: ? balance_pgdat+0x6a2/0x6a2
Nov 30 12:41:45 Reaver kernel: kthread+0xe7/0xef
Nov 30 12:41:45 Reaver kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov 30 12:41:45 Reaver kernel: ret_from_fork+0x22/0x30
Nov 30 12:41:45 Reaver kernel: </TASK>
Nov 30 12:41:45 Reaver kernel: Modules linked in: nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag it87 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls atlantic igb i2c_algo_bit edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi kvm video btusb btrtl drm_kms_helper crct10dif_pclmul btbcm crc32_pclmul btintel crc32c_intel mxm_wmi gigabyte_wmi wmi_bmof drm bluetooth ghash_clmulni_intel mpt3sas sha512_ssse3 backlight syscopyarea sha256_ssse3 sysfillrect sha1_ssse3 sysimgblt ahci raid_class aesni_intel ecdh_generic crypto_simd cryptd rapl joydev fb_sys_fops scsi_transport_sas ecc libahci nvme i2c_piix4 ccp i2c_core nvme_core k10temp thermal wmi button acpi_cpufreq unix [last unloaded: atlantic]
Nov 30 12:41:45 Reaver kernel: ---[ end trace 0000000000000000 ]---
Nov 30 12:41:45 Reaver kernel: RIP: 0010:free_buffer_head+0x15/0x64
Nov 30 12:41:45 Reaver kernel: Code: e0 d9 7e 85 c0 75 05 0f 1f 44 00 00 48 89 d8 5b e9 94 33 98 00 0f 1f 44 00 00 48 8b 57 48 48 8d 47 48 48 89 fe 48 39 c2 74 02 <0f> 0b 48 8b 3d 56 2b 13 01 e8 3c 74 fa ff bf 01 00 00 00 e8 a0 b8
Nov 30 12:41:45 Reaver kernel: RSP: 0018:ffffc9000628fa78 EFLAGS: 00010202
Nov 30 12:41:45 Reaver kernel: RAX: ffff88964aad8660 RBX: ffff88964aad85b0 RCX: 000000008027001d
Nov 30 12:41:45 Reaver kernel: RDX: ffff88964aad8661 RSI: ffff88964aad8618 RDI: ffff88964aad8618
Nov 30 12:41:45 Reaver kernel: RBP: 0000000000000001 R08: ffff88964aad8680 R09: 000000008027001d
Nov 30 12:41:45 Reaver kernel: R10: ffff88964aad8680 R11: ffff88851bfe6460 R12: ffff88964aad8680
Nov 30 12:41:45 Reaver kernel: R13: ffff88810a181708 R14: 0000000000000001 R15: ffffea005ad1f6c8
Nov 30 12:41:45 Reaver kernel: FS:  0000000000000000(0000) GS:ffff889fcd740000(0000) knlGS:0000000000000000
Nov 30 12:41:45 Reaver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 12:41:45 Reaver kernel: CR2: 0000153ba13f2484 CR3: 000000014d07e000 CR4: 0000000000350ee0
Nov 30 12:41:45 Reaver kernel: ------------[ cut here ]------------
Nov 30 12:41:45 Reaver kernel: WARNING: CPU: 29 PID: 712 at kernel/exit.c:816 do_exit+0x87/0x923
Nov 30 12:41:45 Reaver kernel: Modules linked in: nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag it87 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls atlantic igb i2c_algo_bit edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi kvm video btusb btrtl drm_kms_helper crct10dif_pclmul btbcm crc32_pclmul btintel crc32c_intel mxm_wmi gigabyte_wmi wmi_bmof drm bluetooth ghash_clmulni_intel mpt3sas sha512_ssse3 backlight syscopyarea sha256_ssse3 sysfillrect sha1_ssse3 sysimgblt ahci raid_class aesni_intel ecdh_generic crypto_simd cryptd rapl joydev fb_sys_fops scsi_transport_sas ecc libahci nvme i2c_piix4 ccp i2c_core nvme_core k10temp thermal wmi button acpi_cpufreq unix [last unloaded: atlantic]
Nov 30 12:41:45 Reaver kernel: CPU: 29 PID: 712 Comm: kswapd0 Tainted: P      D    O       6.1.118-Unraid #1
Nov 30 12:41:45 Reaver kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS MASTER/TRX40 AORUS MASTER, BIOS F5p 02/19/2021
Nov 30 12:41:45 Reaver kernel: RIP: 0010:do_exit+0x87/0x923
Nov 30 12:41:45 Reaver kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 1f 47 81 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 21 46 81 00 48 8b 83 d0 06 00 00 83
Nov 30 12:41:45 Reaver kernel: RSP: 0018:ffffc9000628fee0 EFLAGS: 00010286
Nov 30 12:41:45 Reaver kernel: RAX: 0000000000000000 RBX: ffff888108a8de80 RCX: 0000000000000000
Nov 30 12:41:45 Reaver kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff
Nov 30 12:41:45 Reaver kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 3030303030303020
Nov 30 12:41:45 Reaver kernel: R10: 3a34524320303030 R11: 6537306434313030 R12: ffff8881017df800
Nov 30 12:41:45 Reaver kernel: R13: ffff888108a90000 R14: 0000000000000002 R15: ffffffff820b440d
Nov 30 12:41:45 Reaver kernel: FS:  0000000000000000(0000) GS:ffff889fcd740000(0000) knlGS:0000000000000000
Nov 30 12:41:45 Reaver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 12:41:45 Reaver kernel: CR2: 0000153ba13f2484 CR3: 000000014d07e000 CR4: 0000000000350ee0
Nov 30 12:41:45 Reaver kernel: Call Trace:
Nov 30 12:41:45 Reaver kernel: <TASK>
Nov 30 12:41:45 Reaver kernel: ? __warn+0xab/0x122
Nov 30 12:41:45 Reaver kernel: ? report_bug+0x109/0x17e
Nov 30 12:41:45 Reaver kernel: ? do_exit+0x87/0x923
Nov 30 12:41:45 Reaver kernel: ? handle_bug+0x41/0x6f
Nov 30 12:41:45 Reaver kernel: ? exc_invalid_op+0x13/0x60
Nov 30 12:41:45 Reaver kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 30 12:41:45 Reaver kernel: ? do_exit+0x87/0x923
Nov 30 12:41:45 Reaver kernel: make_task_dead+0x11c/0x11c
Nov 30 12:41:45 Reaver kernel: rewind_stack_and_make_dead+0x17/0x17
Nov 30 12:41:45 Reaver kernel: RIP: 0000:0x0
Nov 30 12:41:45 Reaver kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Nov 30 12:41:45 Reaver kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000
Nov 30 12:41:45 Reaver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 30 12:41:45 Reaver kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 30 12:41:45 Reaver kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Nov 30 12:41:45 Reaver kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 30 12:41:45 Reaver kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 30 12:41:45 Reaver kernel: </TASK>
Nov 30 12:41:45 Reaver kernel: ---[ end trace 0000000000000000 ]---
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: sd 11:0:0:0: [sdg] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Nov 30 12:52:44 Reaver kernel: sd 11:0:0:0: [sdg] tag#7 CDB: opcode=0x8a 8a 00 00 00 00 03 e6 8f e9 70 00 00 00 00 00 00
Nov 30 12:52:44 Reaver kernel: I/O error, dev sdg, sector 16753092976 op 0x1:(WRITE) flags 0x104000 phys_seg 1 prio class 2
Nov 30 12:52:44 Reaver kernel: Buffer I/O error on dev sdg1, logical block 16753092911, lost async page write
Nov 30 13:07:30 Reaver kernel: general protection fault, probably for non-canonical address 0x53c095c0ffff8896: 0000 [#2] PREEMPT SMP NOPTI
Nov 30 13:07:30 Reaver kernel: CPU: 22 PID: 6830 Comm: emhttpd Tainted: P      D W  O       6.1.118-Unraid #1
Nov 30 13:07:30 Reaver kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS MASTER/TRX40 AORUS MASTER, BIOS F5p 02/19/2021
Nov 30 13:07:30 Reaver kernel: RIP: 0010:buffer_check_dirty_writeback+0x2c/0x52
Nov 30 13:07:30 Reaver kernel: Code: 44 00 00 c6 06 00 c6 02 00 48 8b 07 a8 01 75 02 0f 0b 48 8b 4f 28 48 85 c9 74 30 48 8b 07 48 c1 e8 0f 88 02 48 89 c8 80 22 01 <48> 8b 38 40 80 e7 04 74 03 c6 02 01 48 8b 38 40 80 e7 02 74 03 c6
Nov 30 13:07:30 Reaver kernel: RSP: 0018:ffffc90025bbb788 EFLAGS: 00010213
Nov 30 13:07:30 Reaver kernel: RAX: 53c095c0ffff8896 RBX: ffffc90025bbb930 RCX: ffff88964aad8138
Nov 30 13:07:30 Reaver kernel: RDX: ffffc90025bbb7b4 RSI: ffffc90025bbb7b3 RDI: 4aad813800000000
Nov 30 13:07:30 Reaver kernel: RBP: ffffc90025bbba70 R08: 0000000000000238 R09: 0000000000000002
Nov 30 13:07:30 Reaver kernel: R10: 0000000000000402 R11: ffff8886e43e8490 R12: ffffc90025bbb8a8
Nov 30 13:07:30 Reaver kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffffea0053c095c8
Nov 30 13:07:30 Reaver kernel: FS:  00001519df9436c0(0000) GS:ffff889fcd580000(0000) knlGS:0000000000000000
Nov 30 13:07:30 Reaver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 13:07:30 Reaver kernel: CR2: 000000c000122000 CR3: 000000014d07e000 CR4: 0000000000350ee0
Nov 30 13:07:30 Reaver kernel: Call Trace:
Nov 30 13:07:30 Reaver kernel: <TASK>
Nov 30 13:07:30 Reaver kernel: ? __die_body+0x1a/0x5c
Nov 30 13:07:30 Reaver kernel: ? die_addr+0x38/0x51
Nov 30 13:07:30 Reaver kernel: ? exc_general_protection+0x30f/0x345
Nov 30 13:07:30 Reaver kernel: ? asm_exc_general_protection+0x22/0x30
Nov 30 13:07:30 Reaver kernel: ? buffer_check_dirty_writeback+0x2c/0x52
Nov 30 13:07:30 Reaver kernel: ? PageUptodate+0x18/0x18
Nov 30 13:07:30 Reaver kernel: shrink_folio_list+0x1c9/0x996
Nov 30 13:07:30 Reaver kernel: ? cgroup_rstat_updated+0x21/0xa5
Nov 30 13:07:30 Reaver kernel: shrink_lruvec+0x61a/0x9b5
Nov 30 13:07:30 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 13:07:30 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 13:07:30 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 13:07:30 Reaver kernel: shrink_node+0x31b/0x588
Nov 30 13:07:30 Reaver kernel: do_try_to_free_pages+0x1b8/0x3ae
Nov 30 13:07:30 Reaver kernel: try_to_free_pages+0x102/0x181
Nov 30 13:07:30 Reaver kernel: __alloc_pages_slowpath.constprop.0+0x2af/0x9a2
Nov 30 13:07:30 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 13:07:30 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 13:07:30 Reaver kernel: ? kmem_cache_alloc+0x122/0x14d
Nov 30 13:07:30 Reaver kernel: ? srso_return_thunk+0x5/0x10
Nov 30 13:07:30 Reaver kernel: __alloc_pages+0x132/0x1e8
Nov 30 13:07:30 Reaver kernel: folio_alloc+0x14/0x35
Nov 30 13:07:30 Reaver kernel: __filemap_get_folio+0x185/0x213
Nov 30 13:07:30 Reaver kernel: ? blkdev_write_begin+0x1c/0x1c
Nov 30 13:07:30 Reaver kernel: pagecache_get_page+0x13/0x63
Nov 30 13:07:30 Reaver kernel: block_write_begin+0x24/0x72
Nov 30 13:07:30 Reaver kernel: generic_perform_write+0xd3/0x1c2
Nov 30 13:07:30 Reaver kernel: __generic_file_write_iter+0x107/0x15e
Nov 30 13:07:30 Reaver kernel: blkdev_write_iter+0xc1/0x12e
Nov 30 13:07:30 Reaver kernel: vfs_write+0x10f/0x1b9
Nov 30 13:07:30 Reaver kernel: ksys_write+0x76/0xc2
Nov 30 13:07:30 Reaver kernel: do_syscall_64+0x65/0x7b
Nov 30 13:07:30 Reaver kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Nov 30 13:07:30 Reaver kernel: RIP: 0033:0x1519e0292bdf
Nov 30 13:07:30 Reaver kernel: Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 32 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c 32 f8 ff 48
Nov 30 13:07:30 Reaver kernel: RSP: 002b:00001519df942c00 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Nov 30 13:07:30 Reaver kernel: RAX: ffffffffffffffda RBX: 00000000005aab6e RCX: 00001519e0292bdf
Nov 30 13:07:30 Reaver kernel: RDX: 0000000000080000 RSI: 00001519df6c2010 RDI: 000000000000000a
Nov 30 13:07:30 Reaver kernel: RBP: 00001519df942ed0 R08: 0000000000000000 R09: 0000000000000000
Nov 30 13:07:30 Reaver kernel: R10: 0000000000000000 R11: 0000000000000293 R12: ffffffffffffff80
Nov 30 13:07:30 Reaver kernel: R13: 0000000000000002 R14: 00007ffca40fc090 R15: 00001519df743000
Nov 30 13:07:30 Reaver kernel: </TASK>
Nov 30 13:07:30 Reaver kernel: Modules linked in: nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag it87 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls atlantic igb i2c_algo_bit edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi kvm video btusb btrtl drm_kms_helper crct10dif_pclmul btbcm crc32_pclmul btintel crc32c_intel mxm_wmi gigabyte_wmi wmi_bmof drm bluetooth ghash_clmulni_intel mpt3sas sha512_ssse3 backlight syscopyarea sha256_ssse3 sysfillrect sha1_ssse3 sysimgblt ahci raid_class aesni_intel ecdh_generic crypto_simd cryptd rapl joydev fb_sys_fops scsi_transport_sas ecc libahci nvme i2c_piix4 ccp i2c_core nvme_core k10temp thermal wmi button acpi_cpufreq unix [last unloaded: atlantic]
Nov 30 13:07:30 Reaver kernel: ---[ end trace 0000000000000000 ]---
Nov 30 13:07:30 Reaver kernel: RIP: 0010:free_buffer_head+0x15/0x64
Nov 30 13:07:30 Reaver kernel: Code: e0 d9 7e 85 c0 75 05 0f 1f 44 00 00 48 89 d8 5b e9 94 33 98 00 0f 1f 44 00 00 48 8b 57 48 48 8d 47 48 48 89 fe 48 39 c2 74 02 <0f> 0b 48 8b 3d 56 2b 13 01 e8 3c 74 fa ff bf 01 00 00 00 e8 a0 b8
Nov 30 13:07:30 Reaver kernel: RSP: 0018:ffffc9000628fa78 EFLAGS: 00010202
Nov 30 13:07:30 Reaver kernel: RAX: ffff88964aad8660 RBX: ffff88964aad85b0 RCX: 000000008027001d
Nov 30 13:07:30 Reaver kernel: RDX: ffff88964aad8661 RSI: ffff88964aad8618 RDI: ffff88964aad8618
Nov 30 13:07:30 Reaver kernel: RBP: 0000000000000001 R08: ffff88964aad8680 R09: 000000008027001d
Nov 30 13:07:30 Reaver kernel: R10: ffff88964aad8680 R11: ffff88851bfe6460 R12: ffff88964aad8680
Nov 30 13:07:30 Reaver kernel: R13: ffff88810a181708 R14: 0000000000000001 R15: ffffea005ad1f6c8
Nov 30 13:07:30 Reaver kernel: FS:  00001519df9436c0(0000) GS:ffff889fcd580000(0000) knlGS:0000000000000000
Nov 30 13:07:30 Reaver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 13:07:30 Reaver kernel: CR2: 000000c000122000 CR3: 000000014d07e000 CR4: 0000000000350ee0
Nov 30 13:07:30 Reaver kernel: ------------[ cut here ]------------
Nov 30 13:07:30 Reaver kernel: WARNING: CPU: 22 PID: 6830 at kernel/exit.c:816 do_exit+0x87/0x923
Nov 30 13:07:30 Reaver kernel: Modules linked in: nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag it87 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls atlantic igb i2c_algo_bit edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi kvm video btusb btrtl drm_kms_helper crct10dif_pclmul btbcm crc32_pclmul btintel crc32c_intel mxm_wmi gigabyte_wmi wmi_bmof drm bluetooth ghash_clmulni_intel mpt3sas sha512_ssse3 backlight syscopyarea sha256_ssse3 sysfillrect sha1_ssse3 sysimgblt ahci raid_class aesni_intel ecdh_generic crypto_simd cryptd rapl joydev fb_sys_fops scsi_transport_sas ecc libahci nvme i2c_piix4 ccp i2c_core nvme_core k10temp thermal wmi button acpi_cpufreq unix [last unloaded: atlantic]
Nov 30 13:07:30 Reaver kernel: CPU: 22 PID: 6830 Comm: emhttpd Tainted: P      D W  O       6.1.118-Unraid #1
Nov 30 13:07:30 Reaver kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS MASTER/TRX40 AORUS MASTER, BIOS F5p 02/19/2021
Nov 30 13:07:30 Reaver kernel: RIP: 0010:do_exit+0x87/0x923
Nov 30 13:07:30 Reaver kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 1f 47 81 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 21 46 81 00 48 8b 83 d0 06 00 00 83
Nov 30 13:07:30 Reaver kernel: RSP: 0018:ffffc90025bbbee0 EFLAGS: 00010286
Nov 30 13:07:30 Reaver kernel: RAX: 0000000000000000 RBX: ffff888108b36e40 RCX: 0000000000000000
Nov 30 13:07:30 Reaver kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff
Nov 30 13:07:30 Reaver kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 3030303030303020
Nov 30 13:07:30 Reaver kernel: R10: 3a34524320303030 R11: 6537306434313030 R12: ffff88814774e800
Nov 30 13:07:30 Reaver kernel: R13: ffff888189d55ac0 R14: 0000000000000000 R15: 0000000000000000
Nov 30 13:07:30 Reaver kernel: FS:  00001519df9436c0(0000) GS:ffff889fcd580000(0000) knlGS:0000000000000000
Nov 30 13:07:30 Reaver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 13:07:30 Reaver kernel: CR2: 000000c000122000 CR3: 000000014d07e000 CR4: 0000000000350ee0
Nov 30 13:07:30 Reaver kernel: Call Trace:
Nov 30 13:07:30 Reaver kernel: <TASK>
Nov 30 13:07:30 Reaver kernel: ? __warn+0xab/0x122
Nov 30 13:07:30 Reaver kernel: ? report_bug+0x109/0x17e
Nov 30 13:07:30 Reaver kernel: ? do_exit+0x87/0x923
Nov 30 13:07:30 Reaver kernel: ? handle_bug+0x41/0x6f
Nov 30 13:07:30 Reaver kernel: ? exc_invalid_op+0x13/0x60
Nov 30 13:07:30 Reaver kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 30 13:07:30 Reaver kernel: ? do_exit+0x87/0x923
Nov 30 13:07:30 Reaver kernel: ? ksys_write+0x76/0xc2
Nov 30 13:07:30 Reaver kernel: make_task_dead+0x11c/0x11c
Nov 30 13:07:30 Reaver kernel: rewind_stack_and_make_dead+0x17/0x17
Nov 30 13:07:30 Reaver kernel: RIP: 0033:0x1519e0292bdf
Nov 30 13:07:30 Reaver kernel: Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 32 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c 32 f8 ff 48
Nov 30 13:07:30 Reaver kernel: RSP: 002b:00001519df942c00 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Nov 30 13:07:30 Reaver kernel: RAX: ffffffffffffffda RBX: 00000000005aab6e RCX: 00001519e0292bdf
Nov 30 13:07:30 Reaver kernel: RDX: 0000000000080000 RSI: 00001519df6c2010 RDI: 000000000000000a
Nov 30 13:07:30 Reaver kernel: RBP: 00001519df942ed0 R08: 0000000000000000 R09: 0000000000000000
Nov 30 13:07:30 Reaver kernel: R10: 0000000000000000 R11: 0000000000000293 R12: ffffffffffffff80
Nov 30 13:07:30 Reaver kernel: R13: 0000000000000002 R14: 00007ffca40fc090 R15: 00001519df743000
Nov 30 13:07:30 Reaver kernel: </TASK>
Nov 30 13:07:30 Reaver kernel: ---[ end trace 0000000000000000 ]---

 

reaver-diagnostics-20241201-1005.zip

10 minutes ago, DracoCadendi said:

Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req
Nov 30 12:52:44 Reaver kernel: ata10.00: WARNING: zero len r/w req

This is the new parity drive, try swapping cables with a different disk, then retry and see where the problem follows.

Are you thinking its a cable issue? or a controller issue? its weird that it happened in different points, and generally when it's to the point of zeroing out the rest of the drive. dosnt feel like any of the connections are loose and this system has been rather stable for 4 years now.

 

Either way, swapping it to the external card as it was on the onboard sata controller.

At this point, i cant see it being the drive given the smart results and ive zeroed the thing twice and nearly had the full disk copied to multiple times. unless its some issue with the kernel, highly doubt that but possible?

 

Edit
I wonder if it's some sort of power instability. i have it on a UPS but we had a bad storm the other day when it was copying and power was fluctuating. i've read some accounts of people saying these drives are a bit more sensitive but not sure if thats true of not.

i guess we will see in another 18 hours.

Edited December 2, 20241 yr by DracoCadendi

the ata error likely mean cable issue this could also be from the power surge/brown out. Some sytems don't take the inital loss of power well and don't recover when still given a lower frequency of steady power. Control is rare but not impossible to be the culprit. it is easier to recommend a cable replacement to test.

I think its something to do with the cable.
 

Well i appreciate y'alls collective hive mind on this one. copy completed and is now rebuilding.

 

I feel like the ATA error was due to the power instability and the older UPS at this point. It gets pretty bad here in winter with snowstorms. ive also replaced the sata cable just to be sure. I have a new UPS i needed to swap in that i got right before the storm so hopefully that will help with any of the odd behavior as well.

 

Prob also should look into a higher PSU closer to 1200 Watts to give a bit more headroom as i add drives and swap out the old 8TB for the 16's.