December 21, 20241 yr Good-day to All, This is my first post and I'm going to try to offer as much relevant information as possible concerning the issue i've run into. Running Unraid 6.12.13 and had what seemed like a pretty stable setup. Back on November 17th, when I had some time off, I relocated the server in a new room. When I went to restart the server in its new location, I got multiple errors during boot-up that were stating something was wrong with the nvme / PCI-express devices. I snapped some pics with my phone and have attached one of the images to this post. Among the error messages, there was the text "AER: Multiple Corrected error message received from 0000:09:00.0" and "PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)". After many lines scrolled by with those messages, the server managed to start. I logged in to the WebUI and saw that one of the NVMe drives in my cache pool [2 NVMe drives total] was not seen by Unraid. Of course, nothing would launch. No Docker and no VMs. I reboot the server twice and the same errors were seen during startup but strangely enough, Unraid saw both NVMe drives again. I decided to run the BTRFS scrub on the cache pool and there were no errors. Despite this, Docker will not launch and VMs will not start. I left the server running and just walked away for a month to take care of work and personal matters. Just earlier this week, I had some time again to try to troubleshoot the server, so I logged in to the WebUI again and had Unraid generate the zip file with all the logs [attached]. As far as I can tell Unraid still sees both NVMe drives in the cache pool. Furthermore, there do not appear to be any errors with the NVMe drives based on the SMART tests I found in the logs, so I am really stuck as to what to do next. Are there any command-line utils I could use to check the health / status of each NVMe drive in the cache pool? Should I take out the NVMe drives and check them in another system? Thanks in advance! tower-smart-20241216-1314.zip tower-diagnostics-20241216-1310.zip
December 23, 20241 yr Author @JorgeB: started the server and started the array this time. Ran the diagnostics again. File is attached. tower-diagnostics-20241223-0409.zip in the syslog I found this line: Dec 23 04:09:14 Tower kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 62078695, rd 573, flush 0, corrupt 14937, gen 614 which looks like there are both write and read errors with a number of corrupt blocks!! Any advice on how to proceed / what to do next is most appreciated! Edited December 23, 20241 yr by Dosage278 updated information
January 1, 20251 yr Author @JorgeB Attached you should see a screen snippet after I ran the BTRFS scrub on the cache pool. It reports no errors found! This is very frustrating because at some level, Unraid recognized there are errors but now the test does not find any errors 😞 Any advice is greatly appreciated!
January 2, 20251 yr Community Expert Since no error were found clear the stats and keep monitoring: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-700582
January 3, 20251 yr Author Hi again and Happy New Year @JorgeB... Appreciate all the guidance thus far. Today I followed the steps as outlined in the post you linked to and managed to clear the BTRFS stats. See attached screenshots. Listed the stats, cleared the stats, the same values were reported, reboot the server, listed the stats again, and finally the stats were cleared. Unfortunately, Docker engine still failed to start. I have attached diagnostics again [with array started!]... Going to look at the logs now to see if I can find anything obvious. Question: how sensitive are connectors for U.2 NVMe drives? My cache pool is composed of 2 identical U.2 NVMe drives and this all began when I moved the server to another location. Put differently, should I go ahead and buy another cable and see if that solves the matter? If there is a problem with the U.2 cable / connector, why would Unraid report that it sees both cache drives? tower-diagnostics-20250103-1659.zip
January 4, 20251 yr Community Expert Solution Docker image is corrupt, delete and recreate: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
January 4, 20251 yr Author @JorgeBThanks, that did the trick! I guess whatever happened with the cache pool corrupted the docker image file...?? Based on what you said in the other thread [which is from 2018!], I take it that Limetech has still not adopted checking the BTRFS dev stats into Unraid natively? So the best practice is to set up that script which is outlined there? Thanks again for all the help!
January 5, 20251 yr Community Expert 16 hours ago, Dosage278 said: So the best practice is to set up that script which is outlined there? Yep
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.