[SOLVED] Docker Containers not Starting on 6.8.3 (stable) - Says docker img file is read only


abhi.ko

Recommended Posts

None of the docker containers were able to be accessed, and this just came to my notice, I believe it might have started sometime today morning. Everything was working fine about a couple of hours ago, I even updated Plex with the latest beta server version today morning, and have been getting pushbullet notifications from radarr and tautulli etc as recent as 45 minutes back. Now none of the containers can load because the docker image is inaccessible (read-only) it looks like, from what I can see in the logs.

 

I tired stopped the docker service and tried to restart it but that fails, saying docker image location is read only - excerpt from the log below. No idea how and why the permissions got changed. I am on 6.8.3 (stable)

Nov 23 11:29:42 Tower emhttpd: shcmd (1708): umount /var/lib/docker
Nov 23 11:30:09 Tower login[15119]: ROOT LOGIN on '/dev/pts/0'
Nov 23 11:34:46 Tower ool www[27234]: /usr/local/emhttp/plugins/dynamix/scripts/emcmd 'cmdStatus=Apply'
Nov 23 11:34:46 Tower emhttpd: Starting services...
Nov 23 11:34:46 Tower emhttpd: shcmd (1723): /etc/rc.d/rc.samba restart
Nov 23 11:34:48 Tower root: Starting Samba: /usr/sbin/smbd -D
Nov 23 11:34:48 Tower root: /usr/sbin/nmbd -D
Nov 23 11:34:48 Tower root: /usr/sbin/wsdd
Nov 23 11:34:48 Tower root: /usr/sbin/winbindd -D
Nov 23 11:34:48 Tower emhttpd: shcmd (1736): /usr/local/sbin/mount_image '/mnt/user/system/docker.img' /var/lib/docker 100
Nov 23 11:34:48 Tower root: truncate: cannot open '/mnt/cache/system/docker.img' for writing: Read-only file system
Nov 23 11:34:48 Tower kernel: BTRFS info (device loop2): using free space tree
Nov 23 11:34:48 Tower kernel: BTRFS info (device loop2): has skinny extents
Nov 23 11:34:48 Tower root: ERROR: unable to resize '/var/lib/docker': Read-only file system
Nov 23 11:34:48 Tower root: Resize '/var/lib/docker' of 'max'
Nov 23 11:34:48 Tower emhttpd: shcmd (1738): /etc/rc.d/rc.docker start
Nov 23 11:34:48 Tower root: starting dockerd ...
Nov 23 11:35:03 Tower emhttpd: shcmd (1740): umount /var/lib/docker

 Will post diagnostics soon.

 

In the meanwhile can someone please provide some guidance as to why this happened and what caused this?

 

Thanks.

Edited by abhi.ko
Link to comment
1 minute ago, JorgeB said:

Without the diags we can only guess, possibly a problem with the cache filesystem.

Thanks @JorgeB. Trying to post it, it has been downloading for a long time (>20 min), not sure if something is wrong. Hopefully will be done soon.

1740147609_2020-11-2312_22_18-Tower_Diagnostics.thumb.png.5dfca5fa9a9dda0d67a7d2ff02c319c8.png

The funny thing is I haven't done anything to my cache drive recently.

 

Did change the drive about 2-3 months back and everything has  been working fine till now.

 

 

Link to comment
1 hour ago, trurl said:

On mobile now so can't look at Diagnostics yet, but likely you corrupted docker.img by filling it. Further evidence in that syslog snippet shows you want it at 100G when 20G should be more than enough. Making it larger won't fix anything it will just make it take longer to fill.

 

40 minutes ago, JorgeB said:

There's a problem with the cache filesystem, you should backup and re-format, then restore the data and also best recreate the docker image.

Thank you @JorgeB and @trurl !

 

How do I achieve this.

  1. I am planning to manually cp over all the cache folders to a location on the array. Is there a faster or better way to do this?
  2. I have the appdata and USB backed up as recently as yesterday via the CA Backup/Restore plugin. so that should hopefully not be corrupt.
  3. How do I go about formatting an already assigned drive?

Sorry for the questions, just don't want to do something I think is right and make this problem worse.

 

Recreating the docker image and reducing size to 20G - do I just recreate a new image and reinstall all the dockers from my templates? The templates should still be available right?

Edited by abhi.ko
Link to comment
3 minutes ago, trurl said:

Your system share where docker.img lives has some files on disk15. And as mentioned there is not any good reason to have 100G docker.img. Have you had problems filling it?

 

What do you get from the command line with this?


ls -lah /mnt/disk15/system

 

thanks @trurl . Was typing a reply to both of you. But here is the output of the ls command:

root@Tower:~# ls -lah /mnt/disk15/system
total 0
drwxrwxrwx  3 root   root   21 Nov  7 13:15 ./
drwxrwxrwx 12 nobody users 201 Nov 23 00:17 ../
drwxrwxrwx  2 root   root   25 Nov  7 13:15 libvirt/

 

Link to comment
17 minutes ago, abhi.ko said:

What do you mean by 'problems filling it'?

You can see how much of docker.img is used on the Dashboard, and we can also see when you post diagnostics.

 

Currently you have no docker.img, but when you recreate it you should set it to 20G, not 100G. I have 17 dockers and they use less than half of 20G.

 

The usual reason for filling docker.img is an app writing to a path that isn't mapped. One common mistake is specifying a path with an app that doesn't exactly match the mapped container path in upper/lower case.

 

1 hour ago, abhi.ko said:

Recreating the docker image

 

Link to comment
6 minutes ago, trurl said:

Do you actually have any VMs?

Yes total of 4 VM's - only one that I do care about. My hassos VM, and maybe the Win 10 VM.

The rest are rarely run, so no big deal if there are gone. I just copied over the domains and systems share contents to the array (share). 

Link to comment
11 minutes ago, trurl said:

You can see how much of docker.img is used on the Dashboard, and we can also see when you post diagnostics.

 

Currently you have no docker.img, but when you recreate it you should set it to 20G, not 100G. I have 17 dockers and they use less than half of 20G.

 

The usual reason for filling docker.img is an app writing to a path that isn't mapped. One common mistake is specifying a path with an app that doesn't exactly match the mapped container path in upper/lower case.

No - never filled. Docker is not currently running - so I couldn't tell you what the status is currently.

 

I think it was a stupid mistake some years back when I changed the default size of the file to 100G, not understanding what I was doing. But never had it filled up.

Link to comment

So an update on this:

 

Thanks @JorgeB - I did what you suggested and everything seems to be working from a docker perspective. But my VM's are not starting up now, not a big deal for me because I have a snapshot of my hassos VM saved up.

 

What I did:

  1. Run CA Backup & Restore to backup appdata and and other files.
  2. Copy over systems and domain shares (they were set to Cache Prefer) manually to location on the array. Did not copy over the data since there was nothing valuable to me in cache at the time. Just some media files. 
  3. Stop the array.
  4. For formatting the cache SSD - Changed the filesystem for the Cache Drive to xfs from btrfs, and clicked apply & done. I am not planning to use a Cache pool anytime soon, not sure if that corruption was any fault of the btrfs. (my knowledge on linux filesystems is very limited, so this is just google/unraid wiki based.
  5.  Cache Drive showed up to be formatted -  clicked checkbox and format.
  6. Once done, restarted the array.  
  7. Restored appdata using the CA backup/restore
  8. Restarted the server as suggested by the plugin.
  9. Copied everything over other than the docker image.
  10. Re-enabled docker in settings and set the image size to 20G - thanks @trurl
  11. Rebuilt docker image from existing templates in my templates.
  12. All containers are now back up and running fine!
  13. Re-enabled VM service, with the copied over image and the VM's showed up. But they are not starting, so not sure what is wrong there, unsure if the filesystem change caused that?

That's it now. Dockers are all back and shares are intact, so I'm happy. Will have to try fixing VM again later, any suggestions here would be appreciated.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.