daftblazer Posted February 13 Share Posted February 13 I've been having issues where docker will randomly stop working. I have the auto update plugin and I've woken up to none of my docker containers appearing in the docker tab. I've had to reboot to get it to appear once again. I've also had it where the server is on but nothing works and not even the unraid GUI is accessible. I started monitoring logs and this is the last thing that was logged before it crashed. Feb 12 19:15:01 Vault kernel: BTRFS warning (device loop2): checksum verify failed on logical 60047360 mirror 1 wanted 0x186c3ef4 found 0x33436545 level 0 Feb 12 19:15:01 Vault kernel: BTRFS info (device loop2): read error corrected: ino 0 off 60047360 (dev /dev/loop2 sector 133664) Feb 12 19:15:01 Vault kernel: BTRFS info (device loop2): read error corrected: ino 0 off 60051456 (dev /dev/loop2 sector 133672) Feb 12 19:15:01 Vault kernel: BTRFS info (device loop2): read error corrected: ino 0 off 60055552 (dev /dev/loop2 sector 133680) Feb 12 19:15:01 Vault kernel: BTRFS info (device loop2): read error corrected: ino 0 off 60059648 (dev /dev/loop2 sector 133688) I'm not sure if this is related but I also did find these errors in the log Feb 11 12:10:32 Vault kernel: BTRFS info (device sdb1): scrub: started on devid 1 Feb 11 12:10:36 Vault kernel: BTRFS warning (device sdb1): checksum error at logical 10649088000 on dev /dev/sdb1, physical 11731218432, root 5, inode 6567155, offset 45056, length 4096, links 1 (path: appdata/radarr/MediaCover/422/fanart-360.jpg) Feb 11 12:10:36 Vault kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Feb 11 12:10:36 Vault kernel: BTRFS warning (device sdb1): checksum error at logical 10649092096 on dev /dev/sdb1, physical 11731222528, root 5, inode 6567155, offset 49152, length 4096, links 1 (path: appdata/radarr/MediaCover/422/fanart-360.jpg) Feb 11 12:10:36 Vault kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Feb 11 12:11:41 Vault kernel: BTRFS info (device sdb1): scrub: finished on devid 1 with status: 0 Quote Link to comment
trurl Posted February 13 Share Posted February 13 Attach Diagnostics to your NEXT post in this thread. Quote Link to comment
daftblazer Posted February 13 Author Share Posted February 13 vault-diagnostics-20240213-1557.zip Quote Link to comment
JorgeB Posted February 13 Share Posted February 13 Btrfs is detecting corruption on the docker image and pool, I would start by running memtest. Quote Link to comment
daftblazer Posted February 14 Author Share Posted February 14 I ran a memtest on both sticks and then each one individually. All the tests passed Quote Link to comment
JorgeB Posted February 14 Share Posted February 14 Recreate the docker image, delete the corrupt files on the pool, reset the pool stats and keep monitoring, if new corruptions are found there's likely still an underlying hardware issue. Quote Link to comment
trurl Posted February 14 Share Posted February 14 https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Quote Link to comment
daftblazer Posted February 15 Author Share Posted February 15 I've switched the docker settings from macvlan to ipvlan and so far it seems better. I recently upgraded to 6.12.6 from 6.11 and thats when a lot of these problems appeared. But also only a few days after I upgraded I had a brief power outage. I do not have a UPS. I'm also now getting random shutdowns. In the log it shows that the system is shutting down so I'm not sure if this is a hardware or software issue. This is what I could find this morning: Feb 15 04:00:46 Vault Docker Auto Update: No updates will be installed Feb 15 07:30:30 Vault webGUI: Successful login user root from 192.168.1.246 Feb 15 07:31:59 Vault monitor: Stop running nchan processes Feb 15 08:04:01 Vault shutdown[13219]: shutting down for system halt Feb 15 08:04:01 Vault init: Switching to runlevel: 0 It just decides to randomly shutdown at 8am Quote Link to comment
itimpi Posted February 15 Share Posted February 15 1 minute ago, daftblazer said: I've switched the docker settings from macvlan to ipvlan and so far it seems better. I recently upgraded to 6.12.6 from 6.11 and thats when a lot of these problems appeared. But also only a few days after I upgraded I had a brief power outage. I do not have a UPS. I'm also now getting random shutdowns. In the log it shows that the system is shutting down so I'm not sure if this is a hardware or software issue. This is what I could find this morning: Feb 15 04:00:46 Vault Docker Auto Update: No updates will be installed Feb 15 07:30:30 Vault webGUI: Successful login user root from 192.168.1.246 Feb 15 07:31:59 Vault monitor: Stop running nchan processes Feb 15 08:04:01 Vault shutdown[13219]: shutting down for system halt Feb 15 08:04:01 Vault init: Switching to runlevel: 0 It just decides to randomly shutdown at 8am that log snippet suggests software initiated the shutdown. Do you have something like S3 sleep plugin installed as I believe that can shutdown the server unexpectedly. You might want to have the syslog server enabled (if you do not already have it enabled) to capture syslog entries leading up to he shutdown. Quote Link to comment
daftblazer Posted February 15 Author Share Posted February 15 I don't have any sleep plugins installed. The syslog server is already enabled Quote Link to comment
JorgeB Posted February 15 Share Posted February 15 Something is invoking a shutdown, if it's not software it could be hardware, like a damaged power button, or a cat/child pressing it. Quote Link to comment
daftblazer Posted February 15 Author Share Posted February 15 I never even thought of that. It might be my cats. I will be covering the power button. Not sure if I should just completely disconnect the power button so it doesn't happen again Quote Link to comment
JonathanM Posted February 15 Share Posted February 15 15 minutes ago, daftblazer said: I never even thought of that. It might be my cats. I will be covering the power button. Not sure if I should just completely disconnect the power button so it doesn't happen again Depending on the case, sometimes the reset button is better suited to be the power button, and the power button can just stay disconnected. Quote Link to comment
trurl Posted February 15 Share Posted February 15 1 hour ago, daftblazer said: It might be my cats. Definitely happened to me before, but on my PC not my Unraid server. That case had a big power button on top just the right size for a cat paw. Quote Link to comment
daftblazer Posted February 15 Author Share Posted February 15 Yeah, my server tower has a Fractal Design Define 7 case, so its a large button on top thats easy for a cat to press. Pretty sure that's what happened. I have it covered now, but It's easy enough to open since the sides are magnetic. so I might just get a button I can put internally in the case Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.