Array stuck trying to start up

[superiorx]

Hi guys,

 

Having some issues with my UnRaid server when trying to start the array. Any insight or help you can provide would be greatly appreciated. After restarting, the array is unable to start. I have attached the most concerning errors in the system logs below, and the system log in its entirety is available also. This issue has only started to occur today. Prior to being unable to start the array, when attempting to update dockers, I was getting an error stating "UnRaid execution error 403". Before today, everything was running smoothly.

Thanks again for your help, guys!

 

 

 

Edited December 23, 2019 by superiorx

[trurl]

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your NEXT post.

[superiorx]

Done. Thank you.

deep-thought-diagnostics-20191224-0222.zip

[trurl]

Your docker image is corrupt. And the fact that you have given it 80G makes me suspect you have been filling it or it has been growing. I always recommend only 20G for docker image. Making it larger will not fix anything, it will just make it take longer to fill.

 

Most likely you have one or more of your docker applications writing into the docker image instead of to mapped storage. If any application is writing to any path that is not in a mapped container path, then it will write into the docker image, filling it up. All applications must be configured to use paths that are absolute and correspond to a container path, case-sensitive.

 

Also, your cache disk is completely full, possibly that is corrupt also.

 

Unfortunately, your old version of Unraid doesn't give us all of the useful information about your configuration that more recent versions do. For example, I can't tell what user shares have files on your full cache. I see you have most shares set to not use cache, but I suspect you just changed them to that because you realized you were filling it. And setting them to not use cache won't get anything moved off cache.

 

Go to Settings - Docker, disable docker service and delete the docker image. Leave it disabled until we get room on your cache.

 

Go to Shares - User Shares and click the Compute All button. Wait for the result, it will likely take several minutes. Then post a screenshot of that page.

[superiorx]

Thank you for the info!

I have disabled the docker service and deleted the docker image.

I went to the "User Shares" section and clicked "Compute All", but after 15 minutes it was still loading. I opened the same page in another tab thinking that it may not have refreshed the information after completing this task automatically. I've attached the screenshot, but if this was done incorrectly and some information has been omitted, please let me know.

 

[trurl]

Set appdata, domains, and system shares to cache-only. That won't get them moved to cache where they belong, but it will get them ignored by mover until we can make space on cache.

 

Set all other shares to cache-yes and run mover. Wait for it to complete, then post another Compute All screenshot.

[superiorx]

I've adjusted the settings to what you've recommended, run the mover and compute all.

 

[trurl]

OK that looks good. Do you actually have an VMs?

 

Post a new diagnostic.

[superiorx]

No, I don't have any VMs.

 

Also, should I do a restart before I post the diagnostics?

[trurl]

1 minute ago, superiorx said:

should I do a restart before I post the diagnostics?

No need, and in general diagnostics are better if you don't reboot, since syslog, included in diagnostics, doesn't survive a reboot.

[superiorx]

deep-thought-diagnostics-20191224-0707.zip

 

No problems. Diagnostics attached.

[trurl]

Looks like you have plenty of space on cache. I don't understand why you have so much space used in the domains share if you have no VMs.

 

Go to Settings - VM Manager and disable.

 

Then set appdata, domains, and system shares to cache-prefer if they aren't already.

 

Then run mover again and do another Compute All screenshot.

[superiorx]

When I go to "Settings - VM Manager", I get the following notice:

 

 

I have set appdata, domains, and system shares to cache-prefer, but when I run the mover, it finishes instantly without making any changes.

 

[trurl]

I don't know how you got so much space use in domains without VM capability. And libvirt is mounting. I know there is a bug that prevents working with that page, but I don't see how you could have gotten any of that enabled in the first place. Did this configuration come from another installation or something?

 

I guess we will just let that go for now. You really should upgrade Unraid.

 

As for your dockers, go to Settings - Docker and change the docker image to only 20G, then enable dockers.

 

You can add your dockers back just as they were by using the Previous Apps feature on the Apps page. But just as they were is probably not good enough since you had created a problem the way they were.

 

What dockers did you have? Probably best to take them one at a time and see if we can figure out where you went wrong.

[superiorx]

I don't believe I had ever attempted to use VMs previously, so I'm confused as to why I've got so much space allocated to it also.

I've been hesitant to update unRaid for awhile now due to 6.7.X causing my dockers to become corrupted, which I think was an issue a few users were also having. This caused me to have to roll back from 6.7.X to 6.5.X as I was having dockers become corrupted every other day. Honestly, I am yet to try 6.8.X though to see if this issue was resolved as I wanted to check the patch notes first.

As for my dockers, I was using: SABnzbd, Deluge VPN, JDownloader, Rader, Sonarr, Plex, Ombi, Ubooquity, Calibre Web, Calibre Server, Jackett, Krusader, Organizr, Organizr V2. I *think* that's all.

 

I have set the docker image to 20gb, and enabled dockers. How do I determine which docker/s are incorrectly configured when adding them back individually?
 

Thanks again for your help, @trurl

[superiorx]

bump

[trurl]

There are 2 ways you can get into trouble with the paths a docker application uses.

1. If you have an application set to write to a path that isn't part of a mapped container volume, then those writes will go into the docker image. Typical mistakes are not using an absolute path (beginning with /), or using different upper/lower case than specified in the mapping (linux is case-sensitive).
2. If you have a mapping which specifies a host path that isn't mounted storage, then writes to the corresponding container path will be in RAM, since the Unraid OS is in RAM. Note that a host path that specifies an Unassigned Device that isn't mounted is in RAM.

[superiorx]

I've checked the docker paths and all seem to be correctly configured. Since making the adjustments you've recommended, everything appears to be acting normally again. Thanks again for your assistance, @trurl!