6.12.8 - Segfaults and call traces

Diagnostics required for bug reports. 

@trurl Unfortunately, collecting diagnostics was not possible as the issue rendered system unresponsive, all, using GUI, ssh or kvm connections. As such, only syslog collection was possible up to a point it would become unresponsive - which is in the OP.

14 hours ago, curled said:

only syslog collection was possible up to a point it would become unresponsive - which is in the OP.

Collect diagnostics before the issue occurs, it contains system profile and settings information that may be relevant. It's not a universal issue that effects everyone, so anything that may help find commonalities is needed.

Do you happen to have the Disk Location plugin? It appears that's what was causing the issue for me.

I did have Disk Location plugin installed. However, I have since moved my Unraid installation to another hardware (Motherboard, CPU and RAM) and issue has been resolved. So it appears this was a hardware issue.

 

Since memtest showed no errors for 16 passes, I suspect it was either motherboard or CPU.

Changed Status to Closed

Changed Priority to Other

8 minutes ago, curled said:

Since memtest showed no errors for 16 passes, I suspect it was either motherboard or CPU.

Possibly, but please note that memtest is only definitive when it finds errors, several accounts of users confirming RAM was the problem without memtest finding anything, when there are multiple RAM sticks it's possible to test with just one, and if still issues try a different one, that will basically rule out bad RAM.

Personally, having been mentioned here, i could already identify one bad RAM module through memtest (got several errors in the same exact memory adress and bit, so very high certainty it is the RAM and nothing else). So definitely don't take my post as evidence there's an issue in Unraid itself.

 

I also had to run tests for very long, it took like 30h to get the first error and i'm taking days now to test each single module. Memtest can take a while to find errors, and dozens of passes.

Edited March 11 by river_system

Yes, it appears this is a hardware issue.

 

I've discovered more errors while stress testing with different RAM sticks and I get sporadic errors like this and random segfaults / freezes:

[   28.797207] SQUASHFS error: xz decompression failed, data probably corrupt
[   28.797209] SQUASHFS error: Failed to read block 0x12eefd3c: -5
[   28.797210] SQUASHFS error: Unable to read fragment cache entry [12eefd3c]
[   28.797215] SQUASHFS error: Unable to read fragment cache entry [12eefd3c]
[   28.797215] SQUASHFS error: Unable to read page, block 12eefd3c, size 94e8
[   28.797217] SQUASHFS error: Unable to read fragment cache entry [12eefd3c]
[   28.797217] SQUASHFS error: Unable to read page, block 12eefd3c, size 94e8

 

I've tried at least 6 USB sticks, including one that works properly in another server, all ports on the motherboard. Since all 4 RAM sticks cleared memtest and all of them cause these issues (even with reduced frequency / non-XMP), I can probably narrow this down to a CPU / mobo.

Edited March 11 by curled

So I've been testing this machine with various configurations and narrowed it down to couple of settings in BIOS that made it stable:

 

Asus performance enhancement: Disabled

DRAM Frequency: 4800

DRAM Timings: 40-39-39-77 (I don't trust this motherboard's auto configuration, so that's why I entered manually, according to RAM manufacturer spec)

CPU C-States: Disabled

CPU Short & Long power: 125w

 

I think it's mostly about the last setting. As soon as I change it to auto or 253w, system becomes unstable. For example, even without call traces, with PL1 and PL2 set to 253w, running a non-correcting xfs_repair in a loop would throw errors on a verified correct filesystem after a few loops:

 

$ xfs_repair -n /dev/mapper/md1p1; while true; do xfs_repair -n /dev/mapper/md1p1 2>&1 | grep -E 'rewrite|CRC'; done
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 1
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
bad CRC for inode 268674054, would rewrite
bad CRC for inode 402653313, would rewrite
Metadata CRC error detected at 0x469f20, xfs_agi block 0x15d4dd22/0x200
agi has bad CRC for ag 3
bad CRC for inode 134, would rewrite
bad CRC for inode 402653361, would rewrite
bad CRC for inode 173, would rewrite
Metadata CRC error detected at 0x44228d, xfs_bnobt block 0xe8de8c8/0x1000
bad CRC for inode 402653314, would rewrite
bad CRC for inode 402653340
bad CRC for inode 134217866, would rewrite
bad CRC for inode 128, would rewrite
bad CRC for inode 402653323, would rewrite
bad CRC for inode 402653327, would rewrite
Metadata CRC error detected at 0x47c68f, xfs_sb block 0xe8de8c0/0x200
superblock has bad CRC for ag 2
bad CRC for inode 268674052, would rewrite

 

Same errors disappear under 125w power limit, along with all call traces.

 

I have tested this with 3 power supplies, so I can rule that one out, I have tested with 4 DRAM modules one-by-one and the only thing that has made the system stable was setting a power limit to 125W. I'm not completely sure why that's happening, possibly a faulty CPU, or something in linux kernel that makes it unstable at turbo boost frequencies?

Edited March 18 by curled

I have RMAd the CPU and replacement one does not have the issue. Turns out it was a hardware issue after all.