November 11, 20241 yr After updating and reboot, docker doesn't start anymore. In the docker tab I see the message "Docker Service failed to start." In the docker settings there's no place to start it. What to do? supermicro-diagnostics-20241111-1832.zip
November 11, 20241 yr Community Expert Both pool devices dropped offline in the past, there could be data corruption, run a correcting scrub and post the results.
November 11, 20241 yr Author 18 minutes ago, JorgeB said: Both pool devices dropped offline in the past, there could be data corruption, run a correcting scrub and post the results. Sorry for my ignorance. How to run a correcting scrub. And what could cause this? Why does it happen after reboot for upgrade? Shouldn’t it just work after a simple reboot? Feels a bit unsettling to me
November 11, 20241 yr Community Expert Click on the pool, then check "repair"and click on scrub. 17 minutes ago, eXorQue said: And what could cause this? 36 minutes ago, JorgeB said: Both pool devices dropped offline in the past Typically, a power/connection issue, or with the NVMe, an APST issue.
November 11, 20241 yr Author It's still not starting unfortunately. I've added the diagnostics. supermicro-diagnostics-20241111-2141.zip
November 12, 20241 yr Hi, I have a similar problem but my docker crashed while the server was running. Can I ask someone knowledgeable to take a look at my diagnostics? I'll try the scrub in a moment. aelothtower-diagnostics-20241112-0905.zip
November 12, 20241 yr Community Expert 12 hours ago, eXorQue said: It's still not starting unfortunately That's expected since the image was corrupt, but first: 15 hours ago, JorgeB said: run a correcting scrub and post the results.
November 12, 20241 yr Community Expert 55 minutes ago, Aeloth said: Hi, I have a similar problem but my docker crashed while the server was running Since this thread is still active, and to avoid confusion, please start your own thread.
November 12, 20241 yr Author 3 hours ago, JorgeB said: That's expected since the image was corrupt, but first: 18 hours ago, JorgeB said: run a correcting scrub and post the results. My bad. I expected the diagnostics to contain the results. Here the results: ``` UUID: 38dc75e4-1cf6-45cd-bb76-e15d45da80d9 Scrub started: Mon Nov 11 20:46:49 2024 Status: finished Duration: 0:13:29 Total to scrub: 671.41GiB Rate: 849.77MiB/s Error summary: no errors found ```
November 12, 20241 yr Community Expert There's something weird with the diags, it's repeating most of the lines multiple times, but the docker service appears to be starting.
November 12, 20241 yr Author When I go to the Docker tab, the message still says "Docker Service failed to start." When I run `docker ps` there's none started ``` docker ps Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ```
November 12, 20241 yr Author I just did a reboot, just to verify it's really not working. supermicro-diagnostics-20241112-1359.zip
November 12, 20241 yr Community Expert Docker was running in the previous diags, maybe the wrong ones? It's not in the latest ones, docker image is corrupt, delete and recreate: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
November 12, 20241 yr Author So after scrub and repair it got broken? I'll recreate. It's getting annoying though. I've had this issue earlier. I don't blame anyone, but I'd like to know what I can do to prevent this Edited November 12, 20241 yr by eXorQue
November 12, 20241 yr Community Expert 5 minutes ago, eXorQue said: but I'd like to know what I can do to prevent this Monitor the pool, if the devices keep dropping there's still a problem: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582
November 12, 20241 yr Author 51 minutes ago, JorgeB said: Monitor the pool, if the devices keep dropping there's still a problem: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582 They dropped out once. On start of UnRAID after rebooting for upgrade it couldn't find one of the pool drives. So it wasn't even started yet. The array hadn't been started either. I then rebooted and the array booted just fine, except for the docker service. The whole idea of having multiple disks in the pool is to not loose data on the "cache" pool. I don't understand why a device dropping out would cause these issues. Should I add another disk to the pool? Addition: is there a way to not have to reinstall all containers and images and everything when the docker.img is corrupted? Some of the images are created from source on my pc (when I use a remote DOCKER_HOST to my server) and I need to recreate them all...... PS2: I have older syslogs if that's useful Edited November 12, 20241 yr by eXorQue older syslogs
November 12, 20241 yr Author syslog-20241111-103928.txt ``` Nov 11 10:00:02 supermicro kernel: btrfs_dev_stat_inc_and_print: 337 callbacks suppressed Nov 11 10:00:02 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 335938737, rd 2125849244, flush 5057466, corrupt 72, gen 0 ```
November 12, 20241 yr Community Expert 56 minutes ago, eXorQue said: They dropped out once. On start of UnRAID after rebooting for upgrade it couldn't find one of the pool drives. So it wasn't even started yet. That wasn't the problem, both devices dropped while online, or there wouldn't be the btrfs filesystem errors, just can't tell if they dropped one or more times, or if they one at a time, or both at the same time. 35 minutes ago, eXorQue said: wr 335938737, rd 2125849244, flush 5057466, These are the filesystem device errors, that show that they dropped offline in the past, see the link for how to reset the btrfs stats, then keep monitoring
November 24, 20241 yr Author @JorgeBI had notifications on the Cache drive. I just got errors Nov 24 20:47:59 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:47:59 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:47:59 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:47:59 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900898, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900899, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900900, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900901, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900902, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900903, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900904, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900905, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900906, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1900907, rd 194623731, flush 68123, corrupt 3, gen 0 Nov 24 20:48:07 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:07 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:07 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:07 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:10 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:10 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:11 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:11 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:11 supermicro kernel: BTRFS warning (device sdf1): lost page write due to IO error on /dev/nvme0n1p1 (-5) Nov 24 20:48:12 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1901436, rd 194623731, flush 68130, corrupt 3, gen 0 Nov 24 20:48:12 supermicro kernel: BTRFS error (device sdf1): bdev /dev/nvme0n1p1 errs: wr 1901437, rd 194623731, flush 68130, corrupt 3, gen 0 I included diagnostics supermicro-diagnostics-20241124-2053.zip
November 25, 20241 yr Community Expert NVMe device dropped offline, see if this helps: On the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off Reboot and see if it makes a difference.
November 25, 20241 yr Author I've applied the changes. Let's see how it goes. Thank you for the quick support
November 25, 20241 yr Author I've restarted. Docker didn't start again. I have scrubbed and repaired Scrub Status btrfs scrub status: UUID: 38dc75e4-1cf6-45cd-bb76-e15d45da80d9 Scrub started: Mon Nov 25 11:27:13 2024 Status: finished Duration: 0:32:15 Total to scrub: 1.61TiB Rate: 870.75MiB/s Error summary: verify=6424 csum=408435 Corrected: 414859 Uncorrectable: 0 Unverified: 0 I then tried restarting docker by setting "Enabled: True" to false and back to true in settings > Docker. That didn't start it. So I restarted the array (not the server itself). Still no restart. I made a Diagnostics file that I added to this post. supermicro-diagnostics-20241125-1203.zip Starting service with `/etc/rc.d/rc.docker start` Gives: `no image mounted at /var/lib/docker` So I restarted to see if it would start docker afterwards, but it didn't... Do I need to delete the image again?
November 25, 20241 yr Author supermicro-diagnostics-20241125-2322.zipAgain cache issues. Should I replace my nvme disk?
November 26, 20241 yr Community Expert If the device keeps dropping, you can try a different brand/model
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.