January 16, 20233 yr **I searched the forums for this log entry and while there were a few, they mostly were in regards to a corrupt docker.img file. This is not the case here. While troubleshooting (and trying to figure out how Docker networks work) I have run into this multiple times. Every time I need to stop Docker to make a change to network settings, it will not start back up again and the log file says that the docker image is in use still. docker.img' is in-use, cannot mount. The only way I have found to fix this is to stop the array (or reboot the whole server but that's dramatic). It seems that something is not unmounting the docker image when you initially stop the docker service. I think this could be a bug.
January 16, 20233 yr Community Expert When you have the problem, get diagnostics and attach to your NEXT post in this thread
January 19, 20233 yr Author @trurl Came up again, here's diagnostics knoxx-diagnostics-20230119-1227.zip Edited January 20, 20233 yr by aglyons
January 20, 20233 yr Community Expert Not entirely clear what the problem is. Your appdata has files on disk4, and system share is completely on disk2. And system share is set to be moved to the array. Ideally, appdata, domains, system shares would all be on fast pool (cache) and set to stay there so Docker/VM performance isn't impacted by slower parity array, and so array disks can spin down since these files are always open. Might as well clean that up. Nothing can move open files. Disable Docker and VM Manager in Settings. Set system share to cache:prefer. Run mover to get appdata and system moved to cache. Mover won't replace files so if any are already on cache you might have to use Dynamix File Manager to clean up any duplicates. Before enabling Docker again delete and recreate docker.img to see if that helps anything. https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file https://wiki.unraid.net/Manual/Docker_Management#Re-Installing_Docker_Applications
January 5, 20242 yr I recently ran into a similar issue on 6.12.6. Discovered that my dockers were not running and saw a message "Docker service failed to start" in my docker tab. Saw a bunch of errors in the logs like: ``` kernel: I/O error, dev loop2, sector 2764512 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 2 kernel: BTRFS error (device loop2: state EA): bdev /dev/loop2 errs: wr 67, rd 0, flush 0, corrupt 0, gen 0 kernel: loop: Write error at byte offset 5152833536, length 4096. ``` Sounds like there is some issue writing to the docker image? So I try to shut down the array, it won't go down, gets stuck on unmounting disks. Tried the suggestions in this thread, unfortunately `umount /var/lib/docker` did not seem to work. Messages in the logs make it look like the docker image isn't the issue, the system also cannot unmount my cache drive and my drive in the array. I wound up going into the Open Files plugin and saw that several appdata folders were in use by `shfs`. I killed that process via the terminal, still no luck. Wound up hitting the Shutdown button and that worked cleanly but still not sure why. Seems like maybe a disk I/O issue? My plan is to go into my docker configs and change `/mnt/user` to `/mnt/cache` where possible to hopefully lighten the load on the fuse filesystem. My only significant recent changes to the system were adding an additional NVME ZFS pool (however no data has been added to that yet so I'm doubtful it's the culprit) and also setting up a Graylog stack with docker-compose... I'm a little worried that writes from Graylog mucked things up, but I only have syslog and plex feeding into there, so the writes shouldn't be *too* excessive. Anyone have any thoughts?
January 6, 20242 yr Community Expert 18 hours ago, shaihulud said: Seems like maybe a disk I/O issue? My plan is to go into my docker configs and change `/mnt/user` to `/mnt/cache` where possible to hopefully lighten the load on the fuse filesystem. You can also eliminate the fuse overhead by making the share an Exclusive share.
January 6, 20242 yr 1 hour ago, trurl said: Attach diagnostics to your NEXT post in this thread. Thanks for your help, here they are. Things seem to be working after a reboot. Dockers are back, I've disabled Graylog and DelugeVPN for now until I have a better idea of what was going on. I did have a large torrent that was downloading and my cache drive got 80% full, so maybe it was just a space issue, although I would have expected things to fail a little more gracefully if my cache disk filled up (altho maybe that was an incorrect assumption). shai-hulud-diagnostics-20240106-1046.zip Edited January 6, 20242 yr by shaihulud
January 6, 20242 yr 1 hour ago, itimpi said: You can also eliminate the fuse overhead by making the share an Exclusive share. Oh this sounds like it may be exactly what I need, thank you. I will look into it further.
January 6, 20242 yr Community Expert 3 hours ago, shaihulud said: I did have a large torrent that was downloading and my cache drive got 80% full, so maybe it was just a space issue, although I would have expected things to fail a little more gracefully if my cache disk filled up (altho maybe that was an incorrect assumption). All of your shares that use cache are configured to overflow to the array if cache gets below Minimum Free. But... Your cache Minimum Free is zero, so it will never overflow, it will just fail if cache fills completely. And btrfs seems particularly fragile if you fill it up.
January 7, 20242 yr 5 hours ago, trurl said: All of your shares that use cache are configured to overflow to the array if cache gets below Minimum Free. But... Your cache Minimum Free is zero, so it will never overflow, it will just fail if cache fills completely. And btrfs seems particularly fragile if you fill it up. Oh wow thank you, this may be just what I needed. Set it to 30GB, let's see how it goes.
December 31, 20241 yr I fear I may be having similar issues, and they are progressively getting worse. Issues I see consistently; 1. Appdata not backing up successfully 2. Intermittent Mover issues where the mover doesn't do anything 3. Docker crashing 4. Entire Unraid system unresponsive to pings / webgui, hard reset fixes Mover example: mvlogger: Log Level: 1 mvlogger: *********************************MOVER -SHARE- START******************************* mvlogger: Tue Dec 31 06:18:57 PST 2024 mvlogger: Share supplied TVShows Sharecfg: /boot/config/shares/TVShows.cfg mvlogger: Cache Pool Name: cache mvlogger: Share Path: /mnt/cache/TVShows mvlogger: Complete Mover Command: find "/mnt/cache/TVShows" -depth | /usr/local/sbin/move -d 1 mvlogger: Tue Dec 31 06:18:57 PST 2024 mvlogger: ********************************Mover Finished***************************** tower-diagnostics-20241231-0829.zip
December 31, 20241 yr Community Expert 2 hours ago, nurollinsiii41 said: Intermittent Mover issues where the mover doesn't do anything Uninstall the plugin and retest 2 hours ago, nurollinsiii41 said: Entire Unraid system unresponsive to pings / webgui, hard reset fixes Enable the syslog server and post that after a crash
January 1, 20251 yr 9 hours ago, JorgeB said: Uninstall the plugin and retest Enable the syslog server and post that after a crash just tried the uninstall and re-install to no avail. Same logs as mentioned above in my example. I’m on 7.0.0 beta if that makes a difference
January 1, 20251 yr Community Expert 18 hours ago, JorgeB said: Enable the syslog server and post that after a crash
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.