Hello,

Since few days/weeks now, I have really weird behavior mainly related to docker, some container stop to work for no obvious reasons.

Initially my docker was using a btrfs vdisk and logs was throwing /dev/loop2 btrfs error, which was related to docker.img

After few time I switch to directory and reinstall all my container but some docker container continue to stop without any warning and sometimes it even refused to restart and I have to restart the NAS

As suggested by discord officel unraid server, I run successfully memtest86+ v9 for a total of 25 hours without any error reported.

XMP is not enable.
tower-diagnostics-20261104-1752.zip
I took the diagnotics zip file last time I got trouble you can find it attached.

Thank you very much for your help

Edited 21 hours ago21 hr by RealZack

Your cache drive is corrupt so has been made read-only. Probably a hardware problem because

Some segfaults in syslog

Nov  3 13:40:10 Tower kernel: .NET ThreadPool[1571238]: segfault at 6 ip 000014d41752d0a0 sp 000014d2c8c815a8 error 6 likely on CPU 2 (core 1, socket 0)
Nov  4 02:10:51 Tower kernel: .NET Tiered Com[2852605]: segfault at 4 ip 000014fd35e38307 sp 000014fbbb102130 error 4 in libcoreclr.so[237307,14fd35da9000+4b6000] likely on CPU 6 (core 3, socket 0)
Nov  4 11:30:59 Tower kernel: AnyIO worker th[2145550]: segfault at 1 ip 000014b82b543045 sp 000014b82298def0 error 4 in libpython3.14.so.1.0[143045,14b82b43b000+27a000] likely on CPU 0 (core 0, socket 0)

11 minutes ago, RealZack said:

run successfully memtest86+

I still suspect RAM.

Thank you for your answer,

RAM could have made the cache drive to be corrupted it's what you mean ?

I understand you suspect the RAM, it's the feedback I got also from discord BUT

I runned memtest three times,
One time for a single pass (no time for more at the moment)
a Second time for 4 pass 10 hours totals those two times using memtest86 v6

And this one 25 hours 10 pass with v9 and if you look at their changelog, v9 bring more tests, more stressfull and suppose to detect error sooner.

considering 15 pass OK, how much pass or time I should test my ram to be 100% sure it is not the RAM ...

Edited 21 hours ago21 hr by RealZack

could have been it corrupted because previously I was running my ram with xmp.
since then I disabled it, but my cache somehow need to be "repaired" ?

Btrfs checks metadata before writing it. On Nov 4 at 09:39 it found a metadata block that was already damaged in memory, refused to write it, and made the pool read-only to protect it. That is why containers stopped and would not restart. The loop2 errors at that time are only a side effect, because loop2 is libvirt.img on the same cache. The SSD itself looks fine, as SMART is clean and there are no SATA errors.

Does the cache need repair? Probably not. The bad block was stopped before it reached the SSD, so I expect the cache to mount normally after a reboot. Please do not run btrfs check --repair.

1. Reboot, because the read-only state does not clear without one, and confirm the cache is writable.

2. Back up appdata and anything else important on the cache.

3. Run a scrub on the pool and post the result.

A clean scrub only proves the data matches its checksums. A file damaged in RAM before it was written would still pass, so if an application database misbehaves, restore it from an older backup.

RAM is still the main suspect, even with those Memtest passes. In about 2 days, unrelated programs crashed with segfaults, a general protection fault and an invalid opcode, and they ran on different CPU threads. One of them was smartctl, which is not a container. That points to memory being corrupted while the server runs, not to one bad application. The cause could be RAM, the CPU memory controller, the board, power, or something in the kernel, and the logs cannot tell these apart.

To answer your question, no number of passes proves it is not the RAM. Memtest does not reproduce a real workload, and I have seen a stick pass 48 hours of Memtest and then fail under real use. A clean Memtest result is useful, but it does not clear the hardware.

What I suggest next:

1. Tell us when you disabled XMP, and whether it was already off on Nov 2 to 4. If it was still on, XMP is a good suspect. If it was already off, XMP cannot explain these crashes.

2. Load BIOS defaults, with no XMP and no CPU overclock or undervolt.

3. Uninstall the CoreFreq plugin for this test. It loads a low-level kernel module and the package dates from 2024, so I want it out of the picture.

4. Run the server with only one stick, or half of them if you have four, and use it normally for 3 or 4 days. Your server was logging a crash about every 8 hours, so that is enough time. Then swap to the other stick and repeat. A real workload is a better test here than more Memtest.

5. After each run, post new diagnostics, and tell us your exact RAM kit and how many sticks you have.

If only one stick crashes, replace it. If both crash alone, the CPU, board or power supply become the next suspects. If neither crashes, try both together again with CoreFreq still removed.

Once the server is stable, I also recommend going back to a btrfs Docker vDisk instead of the directory. The earlier vDisk errors were most likely another symptom of this same problem, not a fault of the vDisk.

Hi and thank you for your long response

1. Tell us when you disabled XMP, and whether it was already off on Nov 2 to 4. If it was still on, XMP is a good suspect. If it was already off, XMP cannot explain these crashes.
I disabled it exactly the 1st of november

2. Load BIOS defaults, with no XMP and no CPU overclock or undervolt.
Beside the xmp no customisation was made on this computer, but I will make sure by setting default again. To be honest after disabling xmp I didn't check if the voltage was set to 1.35 accordingly.

I should review the settings the motherboard set for the ram (beside the lower frequences as expected)

3. Uninstall the CoreFreq plugin for this test. It loads a low-level kernel module and the package dates from 2024, so I want it out of the picture.
Done

4. Run the server with only one stick, or half of them if you have four, and use it normally for 3 or 4 days. Your server was logging a crash about every 8 hours, so that is enough time. Then swap to the other stick and repeat. A real workload is a better test here than more Memtest.

I will try that, by the way by desinstalling coreFreq I figure out docker crashed again, just in case if you want to confirm your diagnostic I put a zip file up to date

5. After each run, post new diagnostics, and tell us your exact RAM kit and how many sticks you have.

G.SKILL TridZ K2 3200mhz 4x16go

I will grab the exact reference if it's needed, I will also try an RMA process in paralelle to figure out if by any chance they can be replaced

tower-diagnostics-20261106-1045.zip

Edited 7 hours ago7 hr by RealZack

Since XMP was already off on November 1, it does not explain the crashes from November 2 onward, so we can rule it out.

The new diagnostics confirm the same problem. This time the Docker service itself crashed with "fatal error: invalid runtime symbol table". That error means Docker found that data it loads from its own program file was damaged in memory. This is a typical sign of memory corruption, not a Docker bug. A PHP process on the host also crashed on the 5th. The good news is that the cache mounted normally after the reboot, with no errors.

About the RAM voltage: with XMP off, the board uses the standard DDR4 settings, which run at 1.2V. That is correct for the lower speed, so you do not need to set 1.35V manually. Just load the BIOS defaults.

Removing CoreFreq does not fully take effect until you reboot, because its kernel module stays loaded until then. It was still loaded when both new crashes happened, so it is not ruled out yet.

Next steps:

1. Load the BIOS defaults and reboot.

2. Confirm the cache is writable, then run a scrub on it and post the result.

3. Since you have 4 sticks, start with two of them, in the slots the board manual recommends for two sticks. Use the server normally for 3 or 4 days. Then replace them with the other two and repeat.

4. If one pair crashes, test those two sticks one at a time to find the bad one.

5. Post new diagnostics after each run, or as soon as something crashes again.

Starting the RMA in parallel is a good idea. If a pair or a single stick crashes, that also gives you something concrete to report to G.Skill.

1. Bios resetted factory

2. Everything start after a reboot, here is the scrub :
   UUID: ffc5eb94-0c3d-43ae-aa1c-f4dabc1d052e Scrub started: Fri Nov 6 14:14:30 2026 Status: finished Duration: 0:06:32 Total to scrub: 161.89GiB Rate: 422.95MiB/s Error summary: verify=8 Corrected: 0 Uncorrectable: 8 Unverified: 0
   
   
   and yes I downloaded the diag before restarting, now the coreFreq should be gone (i'm not seeing it anymore in the plugin page)

Edited 3 hours ago3 hr by RealZack

Thanks for the scrub result. Unfortunately it shows the cache has some damage on the SSD.

"verify" errors are metadata errors, which means the Btrfs internal structures are damaged, not your file data directly. Metadata on this pool is stored twice (DUP). "Uncorrectable" means both copies are bad, so scrub had no good copy to repair from. Most likely this is one damaged metadata block, counted once for each 4 KiB part on each copy, but the log will show that.

So not all the damage was stopped in RAM. The check at write time catches only some types of damage, and this block got past it at some earlier time. The pool mounts and works now, but files or folders that depend on this block can give I/O errors when you read them. If Btrfs needs to change that block, the pool can go read-only again. Scrub cannot fix it, and please do not run btrfs check --repair.

What I suggest:

1. Post new diagnostics now, before you reboot. The scrub wrote the location of each error to the syslog, and that shows which part of the filesystem is affected.

2. Back up everything important on the cache now, while it still mounts. If some files fail to copy with an I/O error, tell us which ones.

3. Continue the RAM test with two sticks at a time, as planned.

4. When you know which sticks are good, reformat the cache and restore your backup. If the cache goes read-only again before that, back up what you can and do it then. For app databases, a backup from before Nov 2 is the safest to restore.

The new diagnostics will also confirm that CoreFreq is not loaded anymore.

so here is the new diag

when i saw the error I took the initiative to backup

Some file was failling to copy but really unrelevant one (0 kb one, log file..), I didn't note them ...

Sorry to ask again but just to be 100% sure, it's failling ram that create those corruption in cache ? or my ssd need to be considered to be replaced too ?

EDIT: i'm doing a save again to cleary identified the files blocking

Question, could I have a situation where my ram with xmp was unstable, create this metadata error.

I then remove xmp, let's assume my ram is now stable, but this corruption is what is still creating issues ?

tower-diagnostics-20261106-1651.zip

Edited 1 hour ago1 hr by RealZack

Thanks, these diagnostics answer most of this, and the news is better than I said in my last post.

The 8 errors are all one 16 KiB block, stored twice, and it belongs to the Btrfs checksum tree. That tree stores the checksums of your data. It does not hold file names or folders. The only fault the log shows is that one 16-byte ID field in the block header is all zeros. Btrfs tests this field only during a scrub, not in normal use. The scrub then read your data, which needs this same tree, without errors. So this block most likely reads normally.

That corrects my last post. If this holds, I do not expect I/O errors on folders from this block, or that it will make the pool go read-only when it changes. A pool with this history is still not one to trust long term, so the plan to reformat stays, but there is no rush.

Does the SSD need to be replaced? Nothing points to that now. SMART is clean, there are no SATA errors, and the same fault is in both copies, 1 GiB apart on the SSD. A random SSD fault is very unlikely to do that. The command below can settle it.

Is it failing RAM? It is still the main suspect, but it is not proven. The memory corruption is clear. The source could still be RAM, the CPU memory controller, the board, power, or a kernel module such as CoreFreq. The stick test will narrow this down.

Could old XMP damage explain everything? Partly, but only for this block. It could be from when XMP was on. But it cannot cause the crashes, for two reasons:

- On Nov 4, three days after you disabled XMP, Btrfs found a different block damaged in RAM.

- Docker itself, PHP and smartctl all crashed, and these run from the Unraid OS, not from the cache. Damage on the cache cannot crash them.

So something was still corrupting memory after XMP was off. The good sign is that there are no crashes since you loaded BIOS defaults and CoreFreq was removed, but that is less than 3 hours so far, which is too short to judge. Your server was logging a crash about every 8 hours.

About the files that failed to copy: the log shows no Btrfs or disk error during your backup, so this block did not cause those failures. They could be files in use or permission problems. Please stop Docker and the VM service before you run the backup again. Then post the file names and the exact error for any file that still fails.

Next recommended steps:

1. Run this now, and post the output. It only reads:

btrfs inspect-internal dump-tree -b 588300288 /dev/sdc1 | head -n 5

If it shows the block with chunk uuid 00000000-..., the block checksum is valid. That means the zeros were already in RAM when the block was written, not caused by the SSD. If it shows checksum verify failed, tell us, because then the block cannot be read normally. The output also has a generation number. That shows if the block was last written before you disabled XMP.

2. Finish the backup, as above.

3. Continue the test with two sticks at a time, and use the server normally for 3 or 4 days per pair. Post diagnostics after each run, or as soon as something crashes.

4. When the server is stable, reformat the cache and restore your backup. Please still do not run btrfs check --repair.

This is the output of your command

root@Tower:~# btrfs inspect-internal dump-tree -b 588300288 /dev/sdc1 | head -n 5

btrfs-progs v7.1

leaf 588300288 items 183 free space 2009 generation 1454853 owner EXTENT_TREE

leaf 588300288 flags 0x1(WRITTEN) backref revision 1

fs uuid ffc5eb94-0c3d-43ae-aa1c-f4dabc1d052e

chunk uuid 453e1b2a-d72a-4bc1-b7d2-f369e8532391

the file that failed : (An unexpected error is keeping you from copying the file. if you continue to receive this error, you can use the error code to search for help bnlabla

error 0x80070780: The file cannot be accessed by the system)

ipc-socket

qbittorent.log

qbittorent.log.bak19

qbittorent.log.bak20

qbittorent.log.bak21

qbittorent.log.bak22

qbittorent.log.bak23

mongodb-27117.sock

ipc-socket and the qbittorrent all came from qbittorrent

mongodb-27117 came from unifi-os

The other one (few) was permissions problem

Edited 19 minutes ago19 min by RealZack

Thanks, but that output does not show what I expected, so I need to correct my last post.

The block we were looking for is no longer at that address. The scrub found a checksum tree block there with a zeroed ID. Your output shows a different block, from the extent tree, with the correct ID and a generation newer than today's mount. Btrfs does not change a block in place. It writes a new copy somewhere else and frees the old spot, and here another block has since used that spot. I should have expected this with the pool in use. So this output says nothing about the old block, and the generation cannot show its age as I said it would. That means we cannot tell if this block dates from when XMP was on. It does not change the plan, because it still cannot explain the crashes.

This is still a good sign. Btrfs had to read that block and verify its checksum before it could replace it. If your cache is still writable and the next diagnostics show no Btrfs errors, the block was valid on the SSD, which means the zeros were already there in RAM when it was written. So the SSD still does not look like the problem, and I do not suggest replacing it.

About the files that failed: error 0x80070780 is not a disk read error.

- ipc-socket and mongodb-27117.sock are sockets. Programs use them to talk to each other, they hold no data, and the app creates them again when it starts. Nothing is lost.

- The qbittorrent log files are not sockets, so I cannot explain those from here, but they are only logs and you can ignore them.

- The permission errors happen because Windows cannot read files owned by the containers. A Windows copy also loses the Linux owners and permissions, which apps like MongoDB need.

You already have the Appdata Backup plugin installed, so please use that instead. It runs on the server, and it keeps owners and permissions. Keep your Windows copy as an extra.

Next steps, in this order:

1. Run Appdata Backup, with the destination on the array, not on the cache. Copy anything else important on the cache to the array too.

2. Run a scrub on the cache again and post the result. If the same 8 errors appear at a different address, that is the same block, moved, because the zeroed ID is copied when Btrfs rewrites a block. Zero errors is also fine. If you get more than 8 errors, or any csum errors, tell us, because that would be new damage.

3. Post new diagnostics before you shut down. These will show if your backups caused any Btrfs errors, and the log is lost on shutdown.

4. Then start the test with two sticks at a time, and use the server normally for 3 or 4 days per pair. Post diagnostics after each run, or as soon as something crashes.

5. When the server is stable, reformat the cache and restore your backup. There is no rush unless the cache goes read-only again. Please still do not run btrfs check --repair.