September 27, 20241 yr About 2 weeks ago I replaced my Cache Pool. 2x 970 Evo Plus 1TB replaced by 2x 990 Pro 4TB. (NVME) They are in a RAID 1 configuration. That all went well, until my drives suddenly went 'missing'. After a bios reset they were available again but I could not start my Docker Service. I removed vDisk from docker.img and then rebuilt everything. Now I had the same problem again. Firmware of NVME is up-to-date. Anyone have any ideas? nas-diagnostics-20240927-1152.zip
September 27, 20241 yr Community Expert Docker image is corrupt, delete and re-create: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
September 30, 20241 yr Author On 9/27/2024 at 12:34 PM, JorgeB said: Docker image is corrupt, delete and re-create: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks Unfortunately I have done this before, hence the question how this can be solved permanently.
September 30, 20241 yr Community Expert If the image keeps getting corrupt there can be an underlying hardware issue, like bad RAM
October 7, 20241 yr Author The problem only started after I replaced my cachepool with 2x 990 Pro 4TB NVME from Samsung. Now everything worked for 2 days and the same again.
October 8, 20241 yr Community Expert The devices are not being detect, that's a different issue, post new diags.
October 9, 20241 yr Author After a normal reboot both cache drives are recognized again. Here are my logs. nas-diagnostics-20241009-2028.zip
October 10, 20241 yr Community Expert The syslog-previous already had both devices missing, I assume they disappear when using the server?
October 10, 20241 yr Author The problem occurs sporadically, at random times. All my docker containers stop one by one. When I reboot the system, my two cache drives are gone. After another reboot, the disks come back online, but I can't start the docker service. My Ubuntu VM also gets corrupted. When I perform a scrub on my Cache pool. it sometimes works again. Not always. If I now delete docker.img and then recreate my containers from the template, it works again. After a while I have the same problems again. Is this the same problem? https://forums.unraid.net/topic/154783-samsung-990-pro-4tb-disappeared-for-the-3e-time/
October 12, 20241 yr Author It's happened again. Two containers stopped, but all my dockers stopped working. After a reboot, both SSDs are missing. Attached are the logs. nas-diagnostics-20241012-2258.zip
October 12, 20241 yr Community Expert Solution Oct 12 04:40:03 NAS kernel: BTRFS warning (device nvme1n1p1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nvme devices dropped offline, a power cycle, not just a reboot, should bring them back online, then try the below. Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 pcie_aspm=off e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off Reboot and see if it makes a difference.
October 14, 20241 yr Author I made the adjustment. I turned the power off and back on. Removed the Docker image and rebuilt it. Now to wait, thanks so far.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.