Having an issue with constant xfs corruption throughout the array.

Memtest didn't find any errors after 7+ passes.

SMART all looks fine despite some disks having been disabled.

I swapped almost every sata/sas cable with new ones and made sure all data and power connections are firmly seated, along with firmly seated HBA.

System is only pulling a max of about 225watts on a Seasonic 650W so I don't believe it's a power issue. Second system with more drives runs fine on the same PSU.

At this point I'm thinking it's an HBA or SAS expander issue (again, same hardware on second system has no issues) due to the corruption across so many disks (all of them at one point or another) but any thoughts/recommendations before I start buying/swapping parts would be extremely helpful.

Diags attached.

Thank you!

c-3po-diagnostics-20241215-1939.zip

I assume you have already repaired the filesystem and more corruptions were found? 

 

Also note that memtest is only definitive if it finds errors, since you have multiple RAM sticks, you can try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

Yes. Fixed numerous times without issue but kept corrupting.

 

I'll try running with individual dimms and see how it goes. Thanks!

Edited December 16, 20241 yr by mmagl

No luck on singling out RAM.

 

I ran each individual DIMM separately and in different slots and got the same corruption as soon as any file transfers began. Unless the DIMM slots are toast, it doesn't appear to be a memory issue. I don't see any physical issues in the slots or on the board.

 

Could it be a bad disk despite no SMART errors dragging the rest down?

 

Going to remove some drives and continue testing.

 

New diags attached for perusal.

c-3po-diagnostics-20241216-1023.zip

If it's not the RAM, my next suspect would be the CPU.

I can't be certain, but I think it's the new Mover Tuning plugin, at least on my media server. It's perfectly fine on my backup server.

 

I ended up temporarily cannibalizing the backup server which has nearly identical parts by swapping them out one at a time. First the CPU, then HBA, then expander. I tested after each swap and it still corrupted each time. Not much left to swap out at this point and I really didn't want to mess with the motherboard or PSU due to the time involved, so I started looking at the software and taking note of when the corruption started and when it occurs.

 

It started when I upgraded to Unraid 7 rc 1. So I downgraded to Beta 4. That didn't help, still corrupted. I tried rsync via CLI. Still corrupted almost immediately. I started using only the GUI rsync. That went well with no immediate corruption. I invoked the mover and it immediately corrupted. So, I removed the mover tuning plugin... and now it's copying fine via GUI rsync and the mover doesn't immediately cause corruption. Time will tell, but so far so good after a few rounds of heavy copying followed by successful moving.

Edited December 17, 20241 yr by mmagl

Do not see how the Mover Tuning plugin can cause corruption.   It might stop mover working correctly but should not cause corruption.

Me either, which is why I immediately assumed a hardware issue, but it's the only time the corruption occurs and that's after swapping in (verified) working parts.

 

Not saying there's something wrong with the plugin b/c it's been working without a hiccup on my other two servers for a couple of months, but there's something finicky with this particular server that only occurs when the mover is invoked with that plugin active. The only real difference between the other two servers and this one is an Nvidia GPU, to the extent that could be relevant.

 

FTR, I tested it more this morning and the same corruption returned immediately upon invoking the mover with the plugin active. Removed the plugin, and it went back to working again without corruption. No idea what's causing it, and doesn't make sense to me, but it's working reliably now so I'll leave it alone unless/until that changes.

Edited December 17, 20241 yr by mmagl