Hello!

 

I am fairly experienced using Unraid, but this is by far the strangest issue I've encountered.

I am dealing with a system that introduces corruption when dealing with larger files, even locally, both on array and cache.
The system passed full MemTest with no issues (both at 3600 and 3200 MHz).

I have 2 servers with the same motherboard / CPU / RAM combo. The systems were working well for a couple of months. Eventually, they started raising a lot of BTRFS corruption errors. I thought this was a result of using RAM XMP profile (I know, bad idea for a server, but I had no issues using XMP on any other server). I defaulted to RAM timings within spec (3200 MHz). When the BTRFS errors did not stop, I scrapped BTRFS altogether and reformatted them to XFS.

On one server (A), this resolved any issues. On XFS, it runs two VMs with GPU passthrough and advanced USBIP setup for over a month now, with not a single error logged. Windows VMs are in perfect health, with no crashes or problematic system events.
Also, the sha test described below results in correct sha (on server A), so I'd say this server is hopefully cleansed from any corruption issues.

 

However, the other server (B), even running XFS cannot hold it together. When working with larger files (starting at around 1GB), it corrupts them on move / rsync / cp attempts. Even running sha256sum on the same file in a row can yield different result. Copying working VM images from another server to this one (A -> B) not only is not possible, because the image copied with rsync -avz does not arrive intact (sha does not match) but also after some time, not only Windows tools are not able to recover the state of the system, but it also crashes more every time it's run.

 

I was able to detect the system on server B had the wrong sha for bzfirmware. As far as I know, this should be detected at boot, no idea why the system booted normally and did not even raise any log errors. This issue possibly goes back to before the update, as an old flash backup of this server has analogous corruption. Copying fresh bzfirmware to the pendrive did not resolve the issue. All bz* files have correct sha now.

 

Also sometimes when using SMB the server throws Kernel issues, once lead to a complete crash.

 

Showcase of the main problem:

 

`Any other system`
 

root@arch /h/e/builds# dd if=/dev/zero of=8gb_zeros.img bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 3.63427 s, 2.4 GB/s
root@arch /h/e/builds# sha256sum 8gb_zeros.img
ebfb4ef19ae410f190327b5ebd312711263bc7579970e87d9c1e2d84e06b3c25  8gb_zeros.img

 

`Malfunctioning System (here, array but same issue is on cache)`
 

root@LOG-B:/mnt/user/BACKUP_NEW# dd if=/dev/zero of=8gb_zeros.img bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 48.1192 s, 179 MB/s
root@LOG-B:/mnt/user/BACKUP_NEW# cp 8gb_zeros.img cp8gb_zeros.img
root@LOG-B:/mnt/user/BACKUP_NEW# sha256sum 8gb_zeros.img cp8gb_zeros.img 8gb_zeros.img cp8gb_zeros.img 8gb_zeros.img cp8gb_zeros.img
54bece301a2162431b067cd4360c803c53cecd7e63802f1bd302c2a80b3bc19d  8gb_zeros.img
54bece301a2162431b067cd4360c803c53cecd7e63802f1bd302c2a80b3bc19d  cp8gb_zeros.img
3b4c8723c218d93bff840c79dfb3a089bf7c6b2a7592e9d77f49d86f6908730d  8gb_zeros.img
b775253dc3222607e940a73333a270c4c4ef57b8d8c459d0fd349e40488c12b6  cp8gb_zeros.img
3b4c8723c218d93bff840c79dfb3a089bf7c6b2a7592e9d77f49d86f6908730d  8gb_zeros.img
8bab52d97e6e28f87b585469611e20624157a4ccea88b5d4e5fc534c209cef91  cp8gb_zeros.img

 

`After reboot`
 

root@LOG-B:/mnt/user/BACKUP_NEW# sha256sum 8gb_zeros.img cp8gb_zeros.img 8gb_zeros.img cp8gb_zeros.img 8gb_zeros.img cp8gb_zeros.img
ebfb4ef19ae410f190327b5ebd312711263bc7579970e87d9c1e2d84e06b3c25  8gb_zeros.img     <----- Correct SHA
e4c55c999f5560bf706d8303c1464464327c7b186be8ba91c39df792d0fb5a00  cp8gb_zeros.img
dfd0b073462375fa96bde6ec6b4faab121c9c1e232c1fca7bfd4965e27b05e5c  8gb_zeros.img     <----- Gets corrupted ?
4cd75493b0dc36a262f8289ce658ef5c4d2508ddf8a66fe5edf4c85d56f5ef39  cp8gb_zeros.img
ebfb4ef19ae410f190327b5ebd312711263bc7579970e87d9c1e2d84e06b3c25  8gb_zeros.img     <----- Correct SHA again ?
4cd75493b0dc36a262f8289ce658ef5c4d2508ddf8a66fe5edf4c85d56f5ef39  cp8gb_zeros.img

 

`For smaller file`
 

root@LOG-B:/mnt/user/BACKUP_NEW# sha256sum 1gb_zeros.img 1gb_zeros.img 1gb_zeros.img 1gb_zeros.img 1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
root@LOG-B:/mnt/user/BACKUP_NEW# cp 1gb_zeros.img cp1gb_zeros.img
root@LOG-B:/mnt/user/BACKUP_NEW# sha256sum 1gb_zeros.img cp1gb_zeros.img 1gb_zeros.img cp1gb_zeros.img 1gb_zeros.img cp1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  cp1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
e7b2f8cc13543b56922a628630683babb3726f0f0b6b4f9b5a36be451ada27a0  cp1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
e7b2f8cc13543b56922a628630683babb3726f0f0b6b4f9b5a36be451ada27a0  cp1gb_zeros.img
root@LOG-B:/mnt/user/BACKUP_NEW# sha256sum 1gb_zeros.img cp1gb_zeros.img 1gb_zeros.img cp1gb_zeros.img 1gb_zeros.img cp1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
e7b2f8cc13543b56922a628630683babb3726f0f0b6b4f9b5a36be451ada27a0  cp1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
e7b2f8cc13543b56922a628630683babb3726f0f0b6b4f9b5a36be451ada27a0  cp1gb_zeros.img
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  1gb_zeros.img
e7b2f8cc13543b56922a628630683babb3726f0f0b6b4f9b5a36be451ada27a0  cp1gb_zeros.img

 

Given this behavior, I'm no longer sure whether the server A had any corruption issues in the first place, or was it a result of moving corrupted flies from server B during backups / VM migrations.

 

I still have some ideas on how to troubleshoot it, but I'm afraid soon I'll run out of ideas, and the stress levels from dealing with this issue will be too high.

 

I attach diagnostics for the perfectly working server (A) and the corruption-infested one (B). Their configuration is very similar, B has 3 extra HDD drives and A has one different GPU.

 

I attach a current Syslog. I also have a Syslog since January, would it be safe to post it here publicly, or could it have any private information?

 

I'd appreciate any help or hints in diagnosing the issue.

log-b-diagnostics-20230814-2028.zip log-b-syslog-20230814-1827.zip log-a-diagnostics-20230814-2008.zip

3 hours ago, Eksitus said:

I still have some ideas on how to troubleshoot it,

Since you seem to be able to trigger the issue, it should be fairly straightforward. Assuming multiple sticks of RAM, remove half, run test, remove other half and replace with prior removed sticks, repeat test.

 

3 hours ago, Eksitus said:

The system passed full MemTest with no issues

Passing doesn't mean the RAM is good, it means the test didn't trigger an error. Only a failed test is fully conclusive.

 

Also, since the CPU is more directly involved with the RAM sticks on newer builds, loading up the CPU can uncover RAM issues.

 

If it's not clear from my approach, I suspect your issue is certainly within the RAM's chain of custody of the data. Random(ish) corruption is almost always RAM related.

52 minutes ago, JonathanM said:

If it's not clear from my approach, I suspect your issue is certainly within the RAM's chain of custody of the data. Random(ish) corruption is almost always RAM related.

 

I'm so glad I reached out for help on the forum because my current suspect was a faulty pendrive / Unraid system file corruption or even a faulty motherboard.

However, it would all make sense, as the first time I built the system, it came with 2 boxes of 2 sticks of 16GB RAM, where each box had one stick not passing the Memtest. The servers were running with the working half as expected. After going through warranty, sticks that came from RMA passed Memtest, so I thought they are good to go, but turns out they might suffer from exactly the same issue, making only one stick from the box usable (which I assume ended up in server A) and the other one from the bundle currently wreaks havoc in server B.

Lesson learned not to take Memtest for granted.

 

Jonathan, your expertise in solving this issue was immensely valuable. Thank you for the lesson!