Docker img won't start when stored on unassigned device

[www_creedthoughts]

Hi All,

After my flash drive corrupted and caused my server to do a hard shutdown, I reinstalled and upgraded (I was on 6.0, now on 6.2) on a new flash drive and now I can't seem to get the docker started. I have the unassigned devices plugin installed, and an SSD mounted and shared with a folder on it storing my docker.img file. When I point the docker to the image file on that drive (or any drive mounted by unassigned devices), the docker won't start. The docker starts fine when pointed to any drive in the array. How can I point docker to the img on the unassigned devices drive?

 

Thanks!

[Bjonness406]

Storing docker.img on a unassigned device is not support on 6.2

[www_creedthoughts]

Out of curiosity, is there a reason why? That seems like reduced functionality, and functionality that I depended on, so understanding why would be helpful.

[Squid]

I think it's a race condition between unraid and UD.  Works fine for some people

 

Sent from my LG-D852 using Tapatalk

 

 

[settings]

I'm having the same issue, my little proliant server is overloaded with 6 disks so I always ran docker on an external drive not in the array. You don't know what you got 'til it's gone 

[www_creedthoughts]

I think it's a race condition between unraid and UD.  Works fine for some people

 

Sent from my LG-D852 using Tapatalk

 

Interesting. Were there major changes in the upgrade that have caused this race condition to appear? What is the likelihood of this change being reversed?

[Squid]

First it has to be found the conditions under which it succeeds and fails

 

Sent from my SM-T560NU using Tapatalk

 

 

[dlandon]

First it has to be found the conditions under which it succeeds and fails

 

Sent from my SM-T560NU using Tapatalk

 

I don't understand why it works for some and not others.  Post a UD and Docker log when it fails and let me see if I can find a cause.

[settings]

One of the prerequisites is that there be a share created called "appdata" (https://lime-technology.com/docker-guide/).

 

Using the web interface it's only possible to set up a share on the raid or recognised cache and because I am using an unassigned drive I can't make that share (outside the array).

 

Could this be the barrier preventing the docker from starting up?

[www_creedthoughts]

First it has to be found the conditions under which it succeeds and fails

 

Sent from my SM-T560NU using Tapatalk

 

I don't understand why it works for some and not others.  Post a UD and Docker log when it fails and let me see if I can find a cause.

 

I am not sure how to find the UD log or Docker log, any suggestions (sorry for being dense)?

 

EDIT: Found the UD log, still not sure how to get the docker log. Give me a couple minutes and I'll upload the UD log. I've got to recreate this scenario now.

[Squid]

First it has to be found the conditions under which it succeeds and fails

 

Sent from my SM-T560NU using Tapatalk

 

I don't understand why it works for some and not others.  Post a UD and Docker log when it fails and let me see if I can find a cause.

Here's at least a diagnostics from a guy with this problem.  http://lime-technology.com/forum/index.php?topic=51890.msg498066#msg498066

 

Maybe it wouldn't be a bad idea to have UD log into the syslog something like "mounting disk whatever", etc

[Squid]

EDIT: Found the UD log, still not sure how to get the docker log. Give me a couple minutes and I'll upload the UD log. I've got to recreate this scenario now.

Docker log is included in the standard diagnostics

[www_creedthoughts]

Alrighty here we go. Attached is the system log and UD log. I restarted the server so the log wouldn't be as obnoxious. Currently, the docker image is set to a hard drive mounted as an unassigned device and the docker won't start.

 

Ignore unassigned device "ST380815AS_9RW1KHZV", that is an unused hard drive in my machine.

 

I'm excited to hear back from anyone, let me know if there is anything else I can do to help.

tower-syslog-20160921-1041.zip

tower-unassigned.devices-20160921-1043.zip

[dlandon]

Alrighty here we go. Attached is the system log and UD log. I restarted the server so the log wouldn't be as obnoxious. Currently, the docker image is set to a hard drive mounted as an unassigned device and the docker won't start.

 

Ignore unassigned device "ST380815AS_9RW1KHZV", that is an unused hard drive in my machine.

 

I'm excited to hear back from anyone, let me know if there is anything else I can do to help.

 

The syslog does not have enough information for me.  Please post the following when you have this issue:

- UD log.

- System diagnostics so I can also see the configuration settings.

[www_creedthoughts]

Hi dlandon, the UD log was attached to my previous post. For completeness, I'm going to restart my server, upload a new UD log and the system diagnostics (which I meant to do last time). Hope this helps solve the issue!

 

EDIT: Let me know if there is anything else you need.

 

EDIT2: I tried to manually start the docker after restarting my machine, and the system diagnostics after attempting that are also attached. That diagnostic log is the "tower-diagnostics-20160921-1419.zip" file.

tower-diagnostics-20160921-1410.zip

tower-unassigned.devices-20160921-1410.zip

tower-diagnostics-20160921-1419.zip

[dlandon]

The UD device did mount correctly at: /mnt/disks/Temporary

 

In your case here is something I see that doesn't look correct.

 

This is from your docker.cfg:

DOCKER_IMAGE_FILE="/mnt/disks/Temporary/Docker/docker.img"
DOCKER_APP_CONFIG_PATH="/mnt/disks/Temporary/Docker/"

 

I would do this:

 

Docker Storage Location: /mnt/disks/Temporary/Docker/docker.img

 

Default appdata storage location: /mnt/disks/Temporary/appdata/

 

Make sure the folders 'Docker', and 'appdata' both exist on the UD device.  You might have to correct permissions on the UD drive for those folders.

[www_creedthoughts]

The UD device did mount correctly at: /mnt/disks/Temporary

 

In your case here is something I see that doesn't look correct.

 

This is from your docker.cfg:

DOCKER_IMAGE_FILE="/mnt/disks/Temporary/Docker/docker.img"
DOCKER_APP_CONFIG_PATH="/mnt/disks/Temporary/Docker/"

 

I would do this:

 

Docker Storage Location: /mnt/disks/Temporary/Docker/docker.img

 

Default appdata storage location: /mnt/disks/Temporary/appdata/

 

Make sure the folders 'Docker', and 'appdata' both exist on the UD device.  You might have to correct permissions on the UD drive for those folders.

 

Hi dlandon, I tried what you suggested and the docker did not start successfully. I attached diagnostics from the tower immediately after attempting to start the docker with the new paths as you've suggested. I'm not sure what you mean by "correcting permissions", nor am I sure how to do that. Any ideas or advice would be much appreciated. Thanks in advance!

tower-diagnostics-20160924-2013.zip

[dlandon]

The UD device did mount correctly at: /mnt/disks/Temporary

 

In your case here is something I see that doesn't look correct.

 

This is from your docker.cfg:

DOCKER_IMAGE_FILE="/mnt/disks/Temporary/Docker/docker.img"
DOCKER_APP_CONFIG_PATH="/mnt/disks/Temporary/Docker/"

 

I would do this:

 

Docker Storage Location: /mnt/disks/Temporary/Docker/docker.img

 

Default appdata storage location: /mnt/disks/Temporary/appdata/

 

Make sure the folders 'Docker', and 'appdata' both exist on the UD device.  You might have to correct permissions on the UD drive for those folders.

 

Hi dlandon, I tried what you suggested and the docker did not start successfully. I attached diagnostics from the tower immediately after attempting to start the docker with the new paths as you've suggested. I'm not sure what you mean by "correcting permissions", nor am I sure how to do that. Any ideas or advice would be much appreciated. Thanks in advance!

 

What does the Docker did not start successfully mean?  Your log shows that the Docker started and PleyPy did not have enough space and your cache drive is full.  It looks like Docker did start, but then was stopped.

Sep 23 10:46:21 Tower shfs/user: cache disk full
Sep 23 10:46:21 Tower shfs/user: err: shfs_create: assign_disk: Programs/Docker/PlexPy/plexpy.db-wal (28) No space left on device

 

You have several things to work on:

- Your cache drive is very full:

/dev/sdg1        60G   54G  5.2G  92% /mnt/cache

Empty it.

 

- Why don't you have auto start on for the array?

 

I would try this for a start:

- Start your array and set the docker to not run.

- Turn off auto start on all your Dockers so they will not start automatically.

- Start Docker and see if it starts.

 

If Docker does not start and stay running. The work on getting it fixed before moving on.

 

- Go to the Dashboard and check the Docker utilization.  I suspect that your 10GB Docker is not large enough.

- Scrub and confirm that your Docker image does not have any errors.

- Adjust the size of the Docker image as necessary.

- Start Dockers one at a time and check utilization after starting each Docker.

 

You need to take things one step at a time.  Get Docker running first and then try to start the individual Dockers and troubleshoot as you go.  There are changes in 6.2 to Docker and it is probably causing you some problems.

[www_creedthoughts]

What does the Docker did not start successfully mean?  Your log shows that the Docker started and PleyPy did not have enough space and your cache drive is full.  It looks like Docker did start, but then was stopped.

 

That means that I go to docker settings, and the docker status in the top right corner is "stopped". I click the dropdown menu next to enable docker, choose yes, and click apply. The page refreshes but the docker status stays as "stopped". The unassigned device has 368GB free on it.

 

Your cache drive is very full

 

My cache drive is where I actually store my VMs and Docker information since storing it on the unassigned device is not working in 6.2. When you ask me to try something I change the docker location from the cache drive to an unassigned device in order to test what you've suggested. The docker runs normally when pointed to the cache drive.

 

Go to the Dashboard and check the Docker utilization.  I suspect that your 10GB Docker is not large enough.

 

I have the exact same img on my cache drive. The docker works normally when pointed to the cache drive but does not start when pointed to the unassigned device. I have only used 4.02GB of my 10GB image.

 

Start your array and set the docker to not run.

Turn off auto start on all your Dockers so they will not start automatically.

 

Auto start is off on all dockers.

 

Start Docker and see if it starts.

 

It still does not start.

[dlandon]

What disk format are you using on the UD device?  You should use xfs.

 

Remove the docker.img from the UD device and start over.  Create a new docker.img on the UD device and map appdata to the UD device.  Then start docker.  Get docker running on the UD device.  If it does not start with a new docker.img file on the UD device, post your diagnostics.  I can't follow what you are doing and I need you to do this one simple thing to see if it works first.  I don't know if your UD docker.img is the same as the cache docker.img or not and it looks like you mapped the docker.img and appdata to the UD device in the previous diagnostics.  Try that first and let's see if docker will start.

 

Your UD device has to have Docker and appdata folders already created.

 

I will also try a test later today to see that a UD device will work with dockers.  I see no reason it shouldn't.

[dlandon]

What disk format are you using on the UD device?  You should use xfs.

 

Remove the docker.img from the UD device and start over.  Create a new docker.img on the UD device and map appdata to the UD device.  Then start docker.  Get docker running on the UD device.  If it does not start with a new docker.img file on the UD device, post your diagnostics.  I can't follow what you are doing and I need you to do this one simple thing to see if it works first.  I don't know if your UD docker.img is the same as the cache docker.img or not and it looks like you mapped the docker.img and appdata to the UD device in the previous diagnostics.  Try that first and let's see if docker will start.

 

Your UD device has to have Docker and appdata folders already created.

 

I will also try a test later today to see that a UD device will work with dockers.  I see no reason it shouldn't.

 

I did a quick test and the docker.img mapped to a UD device fails to start with a "no image mounted at /var/lib/docker" error.  This is an LT issue that I have no control over.  I had to go to the command line to see this error.  It would be better to log this so it can be easier to find.

 

Bottom line is it doesn't look like you can't use a UD device for the docker.img at this time.  Appdata can be mapped to a UD device.

 

For the time being you can map the docker.img to an array device (cache?) and the appdata to a UD device.  The docker.img is small compared to the appdata folder so it could move a potentially large amount the data off a array (cache?) drive.

[www_creedthoughts]

Thank you for all of the research dlandon! That is disappointing that you can't use an UD device for the docker. Hopefully that functionality will be restored in the future.

 

Thanks again!

[dlandon]

I posted a defect report and LT is looking into it.

[DZMM]

On 9/29/2016 at 4:07 AM, dlandon said:

I posted a defect report and LT is looking into it.

I think I've been running into this problem since I moved my docker and appdata to ud last week.  The main problem I have is when I try to edit a docker it disappears.  Has there been any updates on docker running on ud?

 

Thanks

[Squid]

16 minutes ago, DZMM said:

The main problem I have is when I try to edit a docker it disappears

Shouldn't have anything to do with where the image is mounted.  You should post the docker Run command that appears after you finish editing the container.