MarianKoniuszko Posted August 29, 2023 Share Posted August 29, 2023 (edited) Hello, I noticed my docker service keep fails almost everyday. I see on logs that there was some btrfs errors. I check scrub option on my cache ssd drive and I see 1 uncorrectable error. I format that drive and move previously copied appdata folder there. Another issue was that my docker.img was very big (107GB) It was on cache drive. After format it is not on cache drive but still have 107GB. I reinstall all docker apps and still have that issue. Now when I run scrub test on cache drive I have 6 uncorrectable errors. To make my dockers work I have to reboot unraid server. I start array and I noticed starting apps take very loong. I see some radarr errors at this time. Radarr was only one app that don't work after I reinstall all apps. I have to delete db files and restore from radarr backup It's possible that it make that issue? Aug 28 23:39:06 unRAID kernel: br-79a8b0407e6b: port 1(vethfc5808c) entered disabled state Aug 28 23:39:06 unRAID kernel: veth32a9667: renamed from eth0 Aug 28 23:39:07 unRAID kernel: br-79a8b0407e6b: port 1(vethfc5808c) entered disabled state Aug 28 23:39:07 unRAID kernel: device vethfc5808c left promiscuous mode Aug 28 23:39:07 unRAID kernel: br-79a8b0407e6b: port 1(vethfc5808c) entered disabled state Aug 28 23:39:08 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered blocking state Aug 28 23:39:08 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered disabled state Aug 28 23:39:08 unRAID kernel: device veth945a11a entered promiscuous mode Aug 28 23:39:10 unRAID kernel: eth0: renamed from veth6f43dc4 Aug 28 23:39:10 unRAID kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth945a11a: link becomes ready Aug 28 23:39:10 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered blocking state Aug 28 23:39:10 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered forwarding state Aug 28 23:40:16 unRAID kernel: Radarr[12366]: segfault at 0 ip 0000150d08d523ee sp 00007fff02d54db0 error 4 in libcoreclr.so[150d08a75000+2fb000] likely on CPU 4 (core 8, socket 0) Aug 28 23:40:16 unRAID kernel: Code: 83 c0 08 48 89 05 52 6c 3a 00 48 39 3d 23 64 38 00 77 57 48 39 3d 5a 6c 3a 00 73 07 48 89 3d 51 6c 3a 00 48 8b 37 48 83 e6 f8 <8b> 16 8b 46 04 85 d2 78 49 31 f6 48 01 c6 48 01 35 6d 5b 3a 00 f7 Aug 28 23:40:18 unRAID kernel: Radarr[12473]: segfault at 1530528c540c ip 0000152cdeeefb4b sp 00007ffca8995bf0 error 6 in libclrjit.so[152cdee17000+1e2000] likely on CPU 4 (core 8, socket 0) Aug 28 23:40:18 unRAID kernel: Code: 10 1d d9 94 10 00 4c 89 e3 e9 0d 02 00 00 41 8b 4b 28 41 8b b7 20 08 00 00 0f b3 ce ba ff ff ff ff 41 89 b7 20 08 00 00 89 c9 <41> 89 94 8f 24 08 00 00 41 80 fd 04 0f 84 76 01 00 00 41 80 fd 10 Aug 28 23:40:19 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered disabled state Aug 28 23:40:19 unRAID kernel: veth6f43dc4: renamed from eth0 Aug 28 23:40:19 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered disabled state Aug 28 23:40:19 unRAID kernel: device veth945a11a left promiscuous mode Aug 28 23:40:19 unRAID kernel: br-79a8b0407e6b: port 1(veth945a11a) entered disabled state Aug 28 23:40:20 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered blocking state Aug 28 23:40:20 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered disabled state Aug 28 23:40:20 unRAID kernel: device veth278f6ca entered promiscuous mode Aug 28 23:40:22 unRAID kernel: eth0: renamed from vethdd839d2 Aug 28 23:40:22 unRAID kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth278f6ca: link becomes ready Aug 28 23:40:22 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered blocking state Aug 28 23:40:22 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered forwarding state Aug 28 23:40:23 unRAID kernel: vethdd839d2: renamed from eth0 Aug 28 23:40:23 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered disabled state Aug 28 23:40:23 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered disabled state Aug 28 23:40:23 unRAID kernel: device veth278f6ca left promiscuous mode Aug 28 23:40:23 unRAID kernel: br-79a8b0407e6b: port 1(veth278f6ca) entered disabled state Aug 28 23:40:23 unRAID kernel: .NET Finalizer[12843]: segfault at f040 ip 000014e7bbee0b4f sp 000014e721218420 error 6 in libcoreclr.so[14e7bbd71000+1dd000] likely on CPU 4 (core 8, socket 0) Aug 28 23:40:23 unRAID kernel: Code: 48 89 48 08 b8 01 00 00 00 41 87 87 ac 01 00 00 85 c0 75 3c 49 83 bf f8 00 00 00 00 74 1d e8 b8 db 00 00 49 8b 9f f8 00 00 00 <83> 43 40 ff 74 47 e8 66 dc 00 00 eb 19 31 db eb 20 41 83 bf a8 01 Aug 28 23:40:24 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered blocking state Aug 28 23:40:24 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered disabled state Aug 28 23:40:24 unRAID kernel: device vethd74aa22 entered promiscuous mode Aug 28 23:40:24 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered blocking state Aug 28 23:40:24 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered forwarding state Aug 28 23:40:24 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered disabled state Aug 28 23:40:26 unRAID kernel: eth0: renamed from veth224ce06 Aug 28 23:40:26 unRAID kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethd74aa22: link becomes ready Aug 28 23:40:26 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered blocking state Aug 28 23:40:26 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered forwarding state Aug 28 23:40:29 unRAID kernel: traps: immich_microser[13142] trap int3 ip:1e74f12 sp:7fffcc8ab550 error:0 in node[400000+4d68000] Aug 28 23:40:29 unRAID kernel: veth224ce06: renamed from eth0 Aug 28 23:40:29 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered disabled state Aug 28 23:40:29 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered disabled state Aug 28 23:40:29 unRAID kernel: device vethd74aa22 left promiscuous mode Aug 28 23:40:29 unRAID kernel: br-79a8b0407e6b: port 1(vethd74aa22) entered disabled state Aug 28 23:41:12 unRAID kernel: BTRFS warning (device loop2): csum failed root 575 ino 3617 off 143360 csum 0xfc1b0b5e expected csum 0x8aacc678 mirror 1 Aug 28 23:41:12 unRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Aug 29 00:00:01 unRAID Plugin Auto Update: Checking for available plugin updates Aug 29 00:00:01 unRAID Docker Auto Update: Community Applications Docker Autoupdate running Aug 29 00:00:01 unRAID Docker Auto Update: Checking for available updates Aug 29 00:00:01 unRAID Docker Auto Update: No updates will be installed unraid-diagnostics-20230829-0809.zip Edited August 29, 2023 by MarianKoniuszko Quote Link to comment
JorgeB Posted August 29, 2023 Share Posted August 29, 2023 If you've recreated the docker image and it keeps getting corrupt start by running memtest. Quote Link to comment
MarianKoniuszko Posted August 29, 2023 Author Share Posted August 29, 2023 8 minutes ago, JorgeB said: If you've recreated the docker image and it keeps getting corrupt start by running memtest. Hi, thank You for answer. I'm not docker expert so can You tell me what do You mean recreate docker image? I Format my ssd cache disk and copy there appdata folder. Then go to appstore and previous apps and reinstall my apps. Folder system/docker with docker.img file was on array now not on my cache drive. But still have 107GB. Do I have to delete it? Quote Link to comment
JorgeB Posted August 29, 2023 Share Posted August 29, 2023 See here: https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564309 Quote Link to comment
MarianKoniuszko Posted August 29, 2023 Author Share Posted August 29, 2023 23 minutes ago, JorgeB said: See here: https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564309 Thanks, I make it that way. Now I set docker image size to 30GB. See that have 36,4GB but on dashboard I see there's 60% used. I don't move image to `/mnt/cache/`. I see it is in `/mnt/user/system/docker/`. It will be better to move to cache? Quote Link to comment
MarianKoniuszko Posted August 29, 2023 Author Share Posted August 29, 2023 5 hours ago, JorgeB said: If you've recreated the docker image and it keeps getting corrupt start by running memtest. Hi, I run memtest grom USB and at start I got many errors. Do I have to buy new RAM? Quote Link to comment
JorgeB Posted August 29, 2023 Share Posted August 29, 2023 25 minutes ago, MarianKoniuszko said: Do I have to buy new RAM? Yep. Quote Link to comment
itimpi Posted August 29, 2023 Share Posted August 29, 2023 You could try setting the RAM to a slower clock speed to see if that helps. If you have multiple RAM sticks you could try testing them individually. Even 1 error is too many when testing the RAM. Quote Link to comment
MarianKoniuszko Posted August 29, 2023 Author Share Posted August 29, 2023 45 minutes ago, itimpi said: You could try setting the RAM to a slower clock speed to see if that helps. If you have multiple RAM sticks you could try testing them individually. Even 1 error is too many when testing the RAM. Hi, thanks for tip. I slow down to 2133mhz but still fails. Now I remove one 32GB and make test. So far there are no errors. Quote Link to comment
MarianKoniuszko Posted August 30, 2023 Author Share Posted August 30, 2023 Hi, Test still working. I have 7 pass and 0 errors on one 32gb stick. How long it should work? Quote Link to comment
itimpi Posted August 30, 2023 Share Posted August 30, 2023 1 hour ago, MarianKoniuszko said: Hi, Test still working. I have 7 pass and 0 errors on one 32gb stick. How long it should work? I would think that is more than enough. Quote Link to comment
MarianKoniuszko Posted August 30, 2023 Author Share Posted August 30, 2023 35 minutes ago, itimpi said: I would think that is more than enough. OK, I make return in Amazon and will order new. But when I make a scrub test on my SSD cache drive I still have 6uncorrectable errors. When I try on that good one 32GB ram. It's because btrfs? Or I have to format again? Quote Link to comment
JorgeB Posted August 30, 2023 Share Posted August 30, 2023 See the syslog after a scrub for the list of the corrupt files, then delete/restore from a backup. Quote Link to comment
MarianKoniuszko Posted September 4, 2023 Author Share Posted September 4, 2023 Hello, I add new 2 x 32GB RAM and format ssd cache to XFS. Now I don't have docker fails error but I get notifications that docker utilisation grows. Now is: I set docker image to 30GB. Now it's 32,2GB. In main dashboard I see: What can I do with that? It's because i have many apps installed? Quote Link to comment
JorgeB Posted September 4, 2023 Share Posted September 4, 2023 See if this helps: https://forums.unraid.net/topic/57181-temporary-docker-faq/#comment-564310 Quote Link to comment
MarianKoniuszko Posted September 4, 2023 Author Share Posted September 4, 2023 1 hour ago, JorgeB said: See if this helps: https://forums.unraid.net/topic/57181-temporary-docker-faq/#comment-564310 Hi, I think my both deluge and deluge-vpn I use for downloading have wright paths now. I set that incomplete folder in deluge web UI before I format and delete docker.img (there was about 100gb). I see both download to Incomplete folder as it set. Quote Link to comment
MarianKoniuszko Posted September 15, 2023 Author Share Posted September 15, 2023 Hi, @JorgeB can You tell me one more thing? I format my cache sdd, recrete docker.img and now have only 48%, docker set to 30GB. Don't occure docker errors so far. But before I have docker.img like 109GB and now when I search for big images I noticed that my: ``` du -h -d 1 /var/lib/docker/ 84G /var/lib/docker/btrfs ``` there was 109GB few days ago. Are that some old subvolumes in that folder I can remove? Regards. Quote Link to comment
JorgeB Posted September 15, 2023 Share Posted September 15, 2023 I don't think you can remove that, also note that du is not very reliable with btrfs (or zfs). Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.