[SOLVED] After Reboot most Dockers gone?!

[cirkator]

Hi, I was watching a plex stream and it stopped, i tried to see whats up and found nothing suspisious.

Stopped the Plex Docker, tried to restart it via DockerMan but it didnt start.

Restarted unRAID clean via the Web-UI and upon restart i saw that most of my dockers are not shown in DockerMan.

 

Help please

 

Beta 10a

 

Syslog

 

pics

 

Edit: I think these Lines are the error happening:

 

Nov 17 21:04:53 conchulio logger: starting docker ...
Nov 17 21:04:53 conchulio kernel: BTRFS: new size for /dev/loop8 is 10737418240
Nov 17 21:04:53 conchulio kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Nov 17 21:04:54 conchulio avahi-daemon[2655]: Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.42.1.
Nov 17 21:04:54 conchulio avahi-daemon[2655]: New relevant interface docker0.IPv4 for mDNS.
Nov 17 21:04:54 conchulio avahi-daemon[2655]: Registering new address record for 172.17.42.1 on docker0.IPv4.
Nov 17 21:04:54 conchulio logger: CrashPlan: Error response from daemon: Cannot start container CrashPlan: write /var/lib/docker/execdriver/native/e979fb4d02e21650236d2c27096752be872451faa601802d1b4f8a9301588ca5/container.json: no space left on device 2014/11/17 21:04:54 Error: failed to start one or more containersPlexMediaServer: Error response from daemon: Cannot start container PlexMediaServer: write /var/lib/docker/execdriver/native/5ddb53d286272a48a7326fac855a6d741e35c8c49ae39604b3ed1466032fd574/container.json: no space left on device 2014/11/17 21:04:54 Error: failed to start one or more containersXBMCServer: Error response from daemon: Cannot start container XBMCServer: write /var/lib/docker/execdriver/native/f422f835515f81dd2167a17c4db2319bb51d9aa2f1354bb5f9f53a7ddd9982c2/container.json: no space left on device 2014/11/17 21:04:54 Error: failed to start one or more containers

[sureguy]

Is your docker share set to cache only?

[cirkator]

My docker share is on an ssd mounted and shared via snap outside of the array.

I dont see the sharename under my shares and i did quite a few reboots before this one without any failures.

[BRiT]

My docker share is on an ssd mounted and shared via snap outside of the array.

I dont see the sharename under my shares and i did quite a few reboots before this one without any failures.

 

What likely is happening is docker is doing its startup before SNAP has the external drive mounted.

[cirkator]

So, how can i delay the docker startup?

[sureguy]

I don't know that you can, you might be able to restart docker after snap is up via your go file.

[cirkator]

Ok, if i understand you correct, by stopping docker and then starting it again, it should get back to normal?

I just tried that and i am still getting the same error:

 

Nov 17 22:54:06 conchulio php: /etc/rc.d/rc.docker stop
Nov 17 22:54:06 conchulio php: stopping docker ...
Nov 17 22:54:07 conchulio php: unmounting docker loopback
Nov 17 22:54:09 conchulio php: 
Nov 17 22:54:15 conchulio php: /etc/rc.d/rc.docker start
Nov 17 22:54:15 conchulio kernel: BTRFS info (device loop8): disk space caching is enabled
Nov 17 22:54:15 conchulio php: Resize '/var/lib/docker' of 'max'
Nov 17 22:54:15 conchulio php: starting docker ...
Nov 17 22:54:15 conchulio kernel: BTRFS: new size for /dev/loop8 is 10737418240
Nov 17 22:54:17 conchulio php: CrashPlan: Error response from daemon: Cannot start container CrashPlan: write /var/lib/docker/execdriver/native/e979fb4d02e21650236d2c27096752be872451faa601802d1b4f8a9301588ca5/container.json: no space left on device 2014/11/17 22:54:17 Error: failed to start one or more containers
Nov 17 22:54:17 conchulio php: PlexMediaServer: Error response from daemon: Cannot start container PlexMediaServer: write /var/lib/docker/execdriver/native/5ddb53d286272a48a7326fac855a6d741e35c8c49ae39604b3ed1466032fd574/container.json: no space left on device 2014/11/17 22:54:17 Error: failed to start one or more containers
Nov 17 22:54:17 conchulio php: XBMCServer: Error response from daemon: Cannot start container XBMCServer: write /var/lib/docker/execdriver/native/f422f835515f81dd2167a17c4db2319bb51d9aa2f1354bb5f9f53a7ddd9982c2/container.json: no space left on device 2014/11/17 22:54:17 Error: failed to start one or more containers
Nov 17 22:54:17 conchulio php: 

[cirkator]

I tried something:

Since the error message talks about "no space left on device" i checked with "df -h" and got this:

root@conchulio:/mnt# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop8       10G  8.9G   16K 100% /var/lib/docker

 

so obviously there is not enough free space.

I then stopped docker, made a copy of my docker.img to docker.img.bak and set the size for the docker.img to 15GB and hit start.

 

Now i get this:

 

root@conchulio:/mnt# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop8       15G  8.9G  5.0G  64% /var/lib/docker

 

So i should have enough space left now, BUT it is still showing me just the few dockers and this is in the log:

 

Nov 17 22:59:11 conchulio php: /etc/rc.d/rc.docker stop
Nov 17 22:59:11 conchulio php: stopping docker ...
Nov 17 22:59:12 conchulio php: unmounting docker loopback
Nov 17 22:59:15 conchulio php: 
Nov 17 23:00:21 conchulio sshd[14236]: Accepted password for root from 192.168.1.60 port 61625 ssh2
Nov 17 23:01:02 conchulio php: /etc/rc.d/rc.docker start
Nov 17 23:01:02 conchulio kernel: BTRFS info (device loop8): disk space caching is enabled
Nov 17 23:01:02 conchulio kernel: BTRFS: new size for /dev/loop8 is 16106127360
Nov 17 23:01:02 conchulio php: Resize '/var/lib/docker' of 'max'
Nov 17 23:01:02 conchulio php: starting docker ...
Nov 17 23:01:02 conchulio php: CrashPlan: started succesfully!
Nov 17 23:01:06 conchulio php: PlexMediaServer: started succesfully!
Nov 17 23:01:06 conchulio kernel: device vetha7dc entered promiscuous mode
Nov 17 23:01:06 conchulio kernel: docker0: port 1(vetha7dc) entered forwarding state
Nov 17 23:01:06 conchulio kernel: docker0: port 1(vetha7dc) entered forwarding state
Nov 17 23:01:06 conchulio avahi-daemon[2655]: Withdrawing workstation service for veth8ef7.
Nov 17 23:01:06 conchulio php: XBMCServer: started succesfully!
Nov 17 23:01:06 conchulio php: 
Nov 17 23:01:21 conchulio kernel: docker0: port 1(vetha7dc) entered forwarding state

 

 

So the question now is, do i have to reinstall and reconfigure all the dockers that did not autostart and seem to be gone?

 

Edit: I can now connect to Plex just fine, as the log shows, it started succesfully. But the missing dockers are not showing up.....

Edit2: Still the same behaviour after a reboot. Fresh Syslog

 

But since all the configs for each docker are stored outside the docker.img, would it be enough to start with a fresh docker.img, redownload all my dockers and just point them to the respective config folders. I would image, that all the apps are then properly configured, right?

[cirkator]

Totally forgot yesterday that the dockerMan plugin stores your my-xxxx templates for each docker. That along with the configs stored in seperate folders from the docker.img brought all my dockers back in less then half an hour!

 

Thanks again for this awesome plugin!!! I love unRAID even more now

 

 

I guess my error came from the full docker image. There are some threads reporting about this problem in the defect reports subforum.