January 9, 20242 yr Hi, I got "Unable to write to cachesystem Drive mounted read-only or completely full" warning. Attached is diagnostics. I had the warning also earlier this week, but at that time thought It might be from misconfigured docker. I deleted docker image and went through docker settings. It happened again, so I stopped all dockers for the time being incase I had missed something (to be checked later). Earlier today I got a missing cache pool nvme disk warning. I shutdown the server and reseated the ssd in question (nvme1n1). It came online and all was good for a couple of hours before the "unable to write" warning popped up again. Some background: I changed hardware this week and relocated the server to another room at the same time. Only PSU, HDDs and sata SSDs came from the old server. Case, MB, memory, GPU, 2x1TB nvme are new and were running in a test unraid rig for 1,5 months flawlessy. When changing to new hardware I also changed the location of appdata, systemdata and VM's from the old cache pool 2x sata SSD to new pool of 2x nvme "cachesystem". I did originally have something pointing to old cache instead of the new, so that's why I thought I might still have some misconfiguration regarding the dockers and consequently shut down all dockers. And to muddy the waters even more, after running the new rig for day or two I also upgraded from 6.11.5 to 6.12.6. and had some work to do to get wireguard and rdp to VMs working, but I think I got that sorted out. I'm still not totally convinced that I don't have anything pointing wrongly, but who knows. So, how to proceed from here? ziggy-nas-diagnostics-20240109-1420.zip
January 9, 20242 yr Community Expert Jan 9 11:33:28 ZIGGY-NAS kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1762907, rd 6707, flush 75929, corrupt 1218576, gen 474 Jan 9 11:33:28 ZIGGY-NAS kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 1914208, rd 70669, flush 16426, corrupt 844313, gen 55 Diags show that both devices dropped offline in the past, depending on the sequence of events the pool may be beyond repair, but try running a correcting scrub and post the results.
January 9, 20242 yr Author UUID: 12a695f9-224c-47c2-bd89-24f6658e137a Scrub started: Tue Jan 9 16:08:34 2024 Status: aborted Duration: 0:00:00 Total to scrub: 719.38GiB Rate: 0.00B/s Error summary: no errors found
January 9, 20242 yr Community Expert Solution Scrub is aborting, suggest copying what you can from the pool and reformat, also see here for better pool monitoring in case the devices drop again.
January 12, 20242 yr Author Damned, I managed to save pretty much everything from the old configs, and had the server running for a couple of days now. And just now got errors again. Diag attached. Reboot brought cachesystem devices back to life. Scrub finished this time: UUID: d395a597-ffad-45b7-9e24-22d7d319ed29 Scrub started: Fri Jan 12 15:36:28 2024 Status: finished Duration: 0:02:20 Total to scrub: 629.04GiB Rate: 4.49GiB/s Error summary: verify=2545 csum=2180566 Corrected: 2183111 Uncorrectable: 0 Unverified: 0 VM's are running, docker service failed to start. I have to go showel loads of snow, will work on this later.. ziggy-nas-diagnostics-20240112-1509.zip
January 12, 20242 yr Community Expert Jan 11 12:16:13 ZIGGY-NAS kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 Jan 11 12:16:13 ZIGGY-NAS kernel: nvme nvme0: Does your device have a faulty power saving mode enabled? Jan 11 12:16:13 ZIGGY-NAS kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug Jan 11 12:16:13 ZIGGY-NAS kernel: nvme 0000:02:00.0: enabling device (0000 -> 0002) Jan 11 12:16:13 ZIGGY-NAS kernel: nvme nvme0: Removing after probe failure status: -19 Jan 11 12:16:13 ZIGGY-NAS kernel: nvme0n1: detected capacity change from 1953525168 to 0 Jan 11 12:16:13 ZIGGY-NAS kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Jan 11 12:16:13 ZIGGY-NAS kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Jan 11 12:16:13 ZIGGY-NAS kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 NVMe dropped offline, this can sometimes help with that: On the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 pcie_aspm=off e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off Reboot and see if it makes a difference.
January 12, 20242 yr Author Thanks, I'll try that and report back hopefully after several days with good news.
January 20, 20242 yr Author Nope. /var/log started to get full with btrfs errors. Fix common errors plugin did not fing anything. I ran Memtest for 24h, no errors found. When booting back online, I'm greeted again with "Unable to write to cachesystem". Scrub aborts right away. ziggy-nas-diagnostics-20240120-1318.zip
January 20, 20242 yr Community Expert Assuming the btrfs stats were reset last time both device dropped offline again: Jan 20 13:15:16 ZIGGY-NAS kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 15687166, rd 8853, flush 101765, corrupt 2224093, gen 2545 Jan 20 13:15:16 ZIGGY-NAS kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 4914818, rd 13045, flush 180631, corrupt 0, gen 0
January 20, 20242 yr Author Any advice how to proceed? I did append the line you suggested before this happened, and I'm running Smart tests now.
January 20, 20242 yr Community Expert It would be good to see the diags from when the devices dropped, but if that line doesn't help, best best would be to use different brand/model devices (or a different board).
January 20, 20242 yr Community Expert Syslog had already rotated so can't see when they dropped, but most likely it would have been the same issue as before.
January 20, 20242 yr Author I can try to see if I can pull the whole syslog for the kast week from the drive. edit: the whole syslog for the last ten days is 160Mb... I searched the syslog for when the controller last time dropped. Attached is partial syslog: from the last reboot until the controller dropping and then some. syslog.log Edited January 20, 20242 yr by hawgorn
January 20, 20242 yr Author I had ”spare” nvme installed intended for VM passtrough use, I moved the cachesystem there for the time being.
January 21, 20242 yr Community Expert Jan 16 22:55:34 ZIGGY-NAS kernel: nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 Jan 16 22:55:34 ZIGGY-NAS kernel: nvme nvme1: Does your device have a faulty power saving mode enabled? Jan 16 22:55:34 ZIGGY-NAS kernel: nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug Jan 16 22:55:34 ZIGGY-NAS kernel: nvme 0000:03:00.0: enabling device (0000 -> 0002) Jan 16 22:55:34 ZIGGY-NAS kernel: nvme nvme1: Removing after probe failure status: -19 As suspected issue is the same as before, try the other device to see if it works better.
February 1, 20242 yr Author I updated the FW for the NVMEs and now they've been running problem free for 1,5 weeks <knocks on wood>
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.