I have a SABRENT 4-Drive NVMe M.2 SSD to PCIe 3.0 x4 Adapter Card [EC-P3X4] in a lower slot (top is gpu) that has four 4tb WD SN850x's.

They are btrfs mirrored and treated as my fast storage that doesn't go to the array.

Things have been working pretty smoothly but the drives went into RO mode last night, causing (I think) a docker container to go haywire triggering an oom situation. Showing two btrfs corruptions now.

Sabrent support isn't able to give me anything as they only support the unit on mac/windows which is fair - it's documented that way.

I captured the diags (out of unnecessary caution I had it redacted a bit more so if there is stuff missing that is needed to help me, let me know), turned Docker and VMs to Disabled, set "pcie_aspm=off" per google searching and did a shutdown from the terminal. It wouldn't cleanly stop the array due to the Docker service misbehaving I believe.

While shut down I reseated all of the nvme's and the pcie card.

Started it all back up but would like to see if I/we can figure out why this happened.

Temps all looked good and haven't seen any SMART issues. Running extended tests on them one at a time now.

Been running this (AI provided) to keep an eye on changes: grep -Ei "AER|PCIe Bus Error|RxErr|BTRFS|I/O error|read-only|corrupt|forced readonly" /var/log/syslog

And looking to see if the corruption count changes:

diagnostics-20260507-2125-redacted.zip

Edited May 8May 8 by kramttocs
Clarified title

filesystem went read only due to this:

May 7 21:20:03 UNRAIDSERVER kernel: BTRFS error (device nvme0n1p1): block=8118920937472 write time tree block corruption detected

Write time block tree corruption is almost always bad RAM, so the first thing is to run memtest.

I also see that they are using overclocked RAM, don't recommend doing that with a server, if you cannot use ECC, at least don't overclock it.

Thanks @JorgeB . That was fast.

Been using this mobo, ram,cpu combo for a Windows machine for several years now and didn't really change much in the bios when going to unraid so didn't think to mess with the xmp settings. Will give memtest a go.

Edit: The ram is rated at 3600, which is what it's at so I think years ago I turned on XMP 1 in order to get it up there instead of 2400.

Edited May 8May 8 by kramttocs
Horrible spelling.

Since I'll want to run memtest for awhile overnight, any benefit to doing the scrub now before? Or best to do it after the memtest? I could quickly also just turn xmp off as a first action before doing either the memtest or scrub.

Just picked the scrub bit up from here since you gave me something new to search on: https://forums.unraid.net/topic/165719-cache-btrfs-write-time-tree-block-corruption/

I woudl recomend runnin memtest before doing anything else, since if there really is bad RAM, continuing to use the server can cause more issues.

Will do. Reminds me I also need to check the bios for some goofy Asus AI overclock setting that seems to default to On when I make bios changes (which I did to change some pcie settings when going to unraid). While maybe not related to this issue, it's caused me grief in the past.

5 hours ago, kramttocs said:

I think years ago I turned on XMP 1

Turning on XMP means you are overclocking.

2 minutes ago, itimpi said:

Turning on XMP means you are overclocking.

Yep, I don't disagree.

Appreciate JorgeB for pointing it out as my searching was focused on the other errors and didn't make a connection to ram. Fingers crossed that's it.

Edit for random rambling: I will say that the Sabrent card works pretty slick besides this (which hopefully isn't even attributed to it). My mobo doesn't support bifurcation for 4 drives so this was the best option I found outside of losing my gpu and going with a x8 card in the top slot. I don't game much so dropping the gpu to the (s)lower slot would be fine except it hits my psu. That means I'd need to buy a low profile/single slot gpu and haven't found a great option there.

Edited May 9May 9 by kramttocs

Finally got around to memtest. First test with all four showed 2 errors quickly. Testing each stick one by one showed one failing with an error and all others passing. Running multiple passes on two sticks to run with for now and will do the same once I get a replacement and put the four back in.

So fingers crossed the nvme board has no issues at all.

Edit: 5 passes with the two sticks (I'll confess I turned xmp back on first for these passes) and no errors. Server is back up and running with the 32gb for now.

Edited May 14May 14 by kramttocs
spelling

The fun never seems to end.

Really like unraid and my setup but haven't hit stable yet.

A lot of unnecessary detail in this post but wanted to detail it for my own sake if nothing else.

---

Have been running with the 32gb, my windows vm (dedicated nvme on motherboard directly), and a smaller set of docker containers that I need but keeping memory under 80%.

Two days ago everything became unresponsive to the point of having to hard reset. No indication why. Parity finished after 36 hours and was back to functional. I setup a remote syslog server since apparently the bad stick of Ram isn't the only problem.

This morning I get an alert that one of my Core drives has errors. Core is my (4) 4tb sn850x mirrored pool that is on the pciex4 card (non-bifurcation) and doesn't have a secondary location. Sure enough one of the drives was at 10k and then quickly to over 24k errors.

Grabbed a diagnostics real quick and shut it down. Started it back up, started the array (cancelled parity) and watched the count to make sure it didn't change for a few minutes. Applied the boot parameters are mentioned in the error logs (will speak to this later). VM service wouldn't start so disabled it and docker and shutdown. Removed the 4xNVME card and rotated all 4 nvmes to check if a slot or drive problem. Booted. VM service started this time and my vm started. Docker service still wouldn't. Figured now was as good a time as any to switch to directory vs image for Docker so did that, restarted and this time my usb drive wasn't recognized by the bios. Removed it from the internal header adapter and stuck it in a rear port. Bios found it this time and all booted fine. Docker and VM running now (same subset of containers as before).

Watched the error count a bit longer and no changes. Ran a scrub which had no uncorrectable. Ran another with no errors reported.

I haven't cleared the errors yet though probably should.

The boot params I have now are (OS and GUI):

I already had the pcie_aspm=off there from the first read-only issue.

label Unraid OS
menu default
kernel /bzimage
append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

These came from my syslog server at the start of it all. Nothing before this :

Not very familiar besides some googling today on these params but sounding like maybe pcie_aspm=off wasn't enough to keep it alive and happy.

Looked at the card again while I had it out and there doesn't look to be a way to add external power or would try something like that.

The temps on these are never high - 30-40C and just get up to the 60s during the scrub. The nvme's aren't heavily utilized either. Usually single digit MB/s or lower. If they were getting a lot of IOs I probably wouldn't even consider this card approach.

Similar to https://forums.unraid.net/topic/189754-cache-drive-stops-working-nvme-controller-crash/. Hopefully not just a bad hardware combo.

The device is still dropping, if the kernel parameters don't help, best bet is to try using a different brand/model device or a different board.