Docker will not start after reboot


Recommended Posts

When I woke up this morning, I noticed my internet wasn't working. I run pihole on Unraid, so that's where I went. None of my docker containers are running. I'm not sure what happened because everything was fine when I went to bed.

 

I was previously writing my syslog to flash but Fix Common Problems told me to disable it... unfortunately I did. Also unfortunately, when I tried to copy the /var/log files to another drive to survive a reboot, it didn't work right so I don't have any logs that lead up to this. Lovely. I did reboot. I can browse my array and everything SEEMS ok so far, but docker still won't start.

 

Quote

root@Tower:/var/lib# df -h
Filesystem        Size  Used Avail Use% Mounted on
rootfs            7.7G  931M  6.8G  12% /
tmpfs              32M  512K   32M   2% /run
devtmpfs          7.7G     0  7.7G   0% /dev
tmpfs             7.8G     0  7.8G   0% /dev/shm
cgroup_root       8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs             128M  584K  128M   1% /var/log
/dev/sda1          15G  504M   14G   4% /boot
/dev/loop0        8.7M  8.7M     0 100% /lib/modules
/dev/loop1        5.9M  5.9M     0 100% /lib/firmware
tmpfs             1.0M     0  1.0M   0% /mnt/disks
/dev/mapper/md1   7.3T  7.2T  176G  98% /mnt/disk1
/dev/mapper/md2   7.3T  6.8T  583G  93% /mnt/disk2
/dev/mapper/md3   7.3T  6.7T  604G  92% /mnt/disk3
/dev/mapper/md4   7.3T  6.7T  613G  92% /mnt/disk4
/dev/mapper/sdg1  120G   79G   41G  66% /mnt/cache
shfs               30T   28T  2.0T  94% /mnt/user0
shfs               30T   28T  2.0T  94% /mnt/user
/dev/loop2         20G   19G     0 100% /var/lib/docker
//VEIN/incoming   1.9T  663G  1.2T  36% /mnt/disks/VEIN_incoming
//VEIN/drivepool   15T   14T  745G  96% /mnt/disks/VEIN_drivepool
//VEIN/tv          15T   14T  745G  96% /mnt/disks/VEIN_tv
//VEIN/movies      15T   14T  745G  96% /mnt/disks/VEIN_movies

 

I see my loop0, 1 and 2 are all 100% full. Can someone give me some pointers before I do something dumb?

 

Thanks!

 

tower-diagnostics-20200108-1537.zip

Link to comment
35 minutes ago, lotekjunky said:

When I woke up this morning, I noticed my internet wasn't working. I run pihole on Unraid, so that's where I went. None of my docker containers are running. I'm not sure what happened because everything was fine when I went to bed.

 

I was previously writing my syslog to flash but Fix Common Problems told me to disable it... unfortunately I did. Also unfortunately, when I tried to copy the /var/log files to another drive to survive a reboot, it didn't work right so I don't have any logs that lead up to this. Lovely. I did reboot. I can browse my array and everything SEEMS ok so far, but docker still won't start.

 

 

I see my loop0, 1 and 2 are all 100% full. Can someone give me some pointers before I do something dumb?

 

Thanks!

 

tower-diagnostics-20200108-1537.zip 92.07 kB · 1 download

It is normal for loop0 and loop1 to be full as they are files loaded by Unraid as part of the boot process.

 

Loop2 (the docker image) being full is very unusual.   It may well mean that a docker is mis-configured and is writing internally to the docker image.   Normally docker containers are configured so that all variable data is mapped externally.

Link to comment
2 minutes ago, itimpi said:

It may well mean that a docker is mis-configured and is writing internally to the docker image.   Normally docker containers are configured so that all variable data is mapped externally.

Yes that is likely. You will have to delete and recreate docker image, then you can install your dockers just as they were using the Previous Apps feature on the Apps page.

 

But just as before is not good enough, since something is wrong and filling the docker image.

 

The usual way docker image gets filled is if an application is writing to a path that isn't mapped. This would be settings within the application itself. Common mistakes are not using an absolute path (beginning with /) or not using the same upper/lower case as specified in the mappings.

 

Link to comment

Thanks for your replies!

 

This makes total sense. I think my plex MetaData directory was not mapped correctly so it was probably filling the docker image with that junk. I moved the old docker image elsewhere (just in case), restarted docker with a fresh image, and added some of the containers back. They are all working now!

 

BUT, I have other issues like my unassigned devices are not working any longer. It's gotta be related because everything had been fine for months until last night. Should I open another thread to discuss other issues?

Link to comment
10 minutes ago, lotekjunky said:

BUT, I have other issues like my unassigned devices are not working any longer. It's gotta be related because everything had been fine for months until last night. Should I open another thread to discuss other issues?

How are they not working?   What formats are they?

 

Are you aware that Unassigned Devices is now split into 2 plugins.      The basic one provides the commonest features while if you install the Plus variant as well you get some additional options added.

Link to comment
13 minutes ago, lotekjunky said:

This makes total sense. I think my plex MetaData directory was not mapped correctly so it was probably filling the docker image with that junk. I moved the old docker image elsewhere (just in case), restarted docker with a fresh image, and added some of the containers back. They are all working now!

Your solution doesn't make total sense. The point I was making is that the problem is likely a setting within an application, not an incorrect mapping. I'm not entirely sure what you mean by the plex MetaData directory and its mapping.

 

And just because they seem to be working doesn't mean they aren't filling docker image again.

Link to comment

For Plex, I just didn't install that container yet. I had the meta data directory mapped to /mnt/user0/appdata which didn't exist, it was supposed to be going into user instead of user0. Does that make more sense?

 

For unassigned devices, the mapped shares are no longer present. I haven't touched that app in months. I was going to uninstall the plug-in and then reinstall it again, but like I said, I didn't want to do anything dumb that I can't revert.

Link to comment
2 hours ago, lotekjunky said:

For Plex, I just didn't install that container yet. I had the meta data directory mapped to /mnt/user0/appdata which didn't exist, it was supposed to be going into user instead of user0. Does that make more sense?

 

For unassigned devices, the mapped shares are no longer present. I haven't touched that app in months. I was going to uninstall the plug-in and then reinstall it again, but like I said, I didn't want to do anything dumb that I can't revert.

user0 is just the user shares excluding cache. Not sure why you would want to do that since the best place for appdata is the cache drive.

 

For Unassigned Devices, are you referring to shares it has mounted from other computers? Just the base UD plugin is enough for that.

 

I generally recommend keeping everything up-to-date.

Link to comment
Just now, trurl said:

user0 is just the user shares excluding cache. Not sure why you would want to do that since the best place for appdata is the cache drive.

 

For Unassigned Devices, are you referring to shares it has mounted from other computers? Just the base UD plugin is enough for that.

 

I generally recommend keeping everything up-to-date.

Yes, that's where I wanted metadata. The metadata for my library is over 500GB and it takes up too much space on my cache drive.

 

For UD, if I go to Shares, it now says "There are no exportable disk shares". If I go into Apps and look for UD, I can get to the UD page that shows they ARE there and they do work... and I can get there from the cli too, but they are not showing on the shares page like they were before everything went sideways this morning.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.