Unraid removing docker containers during parity check

[KyleCo]

Hello,

 

I'm having a strange issue during my parity checks. Anytime a parity check starts on my system, I end up losing some of the docker containers I have setup. I did not have a parity drive until recently but since I installed it, this has been happening.

 

Between the times of the parity check running and before I installed in the parity drive, I have 0 issues. It is always random docker containers that seem to get removed, but it is at least 3 containers removed every time, if not more. I've tried searching for an answer but I haven't found anyone else having this issue before. I am completely up to date with version 6.8.3

 

I've only been using Unraid for about 8 months so I am still kind of new, so sorry if I should have provided more info. Happy to provide any logs or other info that could be helpful.

 

Thanks!!

[itimpi]

I do not see how a parity check could cause docker containers to disappear unless it is somehow triggering a drive containing them to be dropped.    You should provide your system diagnostics zip file (obtained via Tools->Diagnostics) covering a period where this happens to get any sort of informed feedback.

[Lothar]

I think that I am having simular issue. Every 2  - 3 months I loose a docker container. I have to reinstall making sure to use the same folder in appdata and everything is good again. I just lost 2 yesterday during a parity sync. I have attached my diagnostics file.  

server-diagnostics-20201102-1808.zip

[KyleCo]

I was doing a parity check when I created this topic and it has now finished. I have lost 3 docker containers(after already losing and replacing 2 that prompted me to create this topic). I've attached my diagnostic file as well.

 

IDK if it's the same for Lothar, but none of the appdata files are lost when I lose these containers. I reinstall them and they are back to normal, though they always say 'Update Available' after being reinstalled, even though I am installing from the apps tab and I would assume it would install the latest version from there.

tower-diagnostics-20201102-1933.zip

[Squid]

Both of you are having updates to the system (docker auto update) being done in and around the same time as your parity checks.  There is however no fundamental reason why the system would "lose" the containers.  What do you exactly mean by "lose"

[KyleCo]

Like I stated in my previous comment, for me "lose" means that unraid no longer knows that the container should be there, it does not show up in the docker tab. It is also not running, as trying to navigate to any of the pages for the docker containers says no connection. The appdata stays in place, but docker loses track of ever having that installed. 

[trurl]

13 hours ago, KyleCo said:

I created this topic

Shouldn't cause this particular issue, but why is your system share on the array?

[trurl]

13 hours ago, Lothar said:

I am having simular issue

1 minute ago, trurl said:

Shouldn't cause this particular issue, but

Your appdata and system shares both have files on the array.

 

[trurl]

Maybe flash problems? If the templates for your installed containers can't be read from flash then those containers will still exist but they can't be worked with in the webUI.

 

Go to the Docker page and move the slider at upper right from Basic to Advanced. Do you see any orphan containers in the list?

[Lothar]

1 hour ago, trurl said:

Your appdata and system shares both have files on the array.

 

I have those shares set to prefer cahce, should I set them to only?

57 minutes ago, trurl said:

Maybe flash problems? If the templates for your installed containers can't be read from flash then those containers will still exist but they can't be worked with in the webUI.

 

Go to the Docker page and move the slider at upper right from Basic to Advanced. Do you see any orphan containers in the list?

I have my flash drive set as private, do I need to set it to public for container access? 

 

Yes I can see the orphaned docker images, thanks fopr the head up. I forgot that you can see these in the advanced settings.

 

For now I am turning off auto update for docker.

[trurl]

7 hours ago, Lothar said:

I have those shares set to prefer cahce, should I set them to only?

prefer is the only setting that will move them to cache, but they can't be moved while the files are open, so you have to go to Settings - Docker and disable then run mover before enabling Docker again

 

7 hours ago, Lothar said:

I have my flash drive set as private

That only controls how it is accessed over the network, has nothing to do with dockers.

 

7 hours ago, Lothar said:

see the orphaned docker images

You should delete them.

[Lothar]

Thank you trurl. 

[Squid]

Are any of the lost containers using any custom networks, or running the network through another container?

[KyleCo]

1 hour ago, Squid said:

Are any of the lost containers using any custom networks, or running the network through another container?

Yes and no. Some of the containers that go missing are on a custom network while others are set to host or bridge.

 

6 hours ago, trurl said:

prefer is the only setting that will move them to cache, but they can't be moved while the files are open, so you have to go to Settings - Docker and disable then run mover before enabling Docker again

I have my appdata set to prefer cache as well. I do this because my drives are not the fastest and I have plenty of cache, so I prefer to have that speed to the appdata folder. Disabling Docker, is that for when you're updating containers?

[trurl]

10 minutes ago, KyleCo said:

Disabling Docker, is that for when you're updating containers?

When mover runs, it can't move open files. The reason I said to disable dockers for him was so appdata and system shares could be moved to cache.

 

 In your case, appdata is already on cache but system share is on the array. System share is where docker.img lives, which contains the executables for all your containers. You want this share to be on cache for performance and so array disks won't stay spunup. But you have system share as cache-yes instead of prefer, which means move to array, and in fact, it is already completely on the array. 

 

16 minutes ago, KyleCo said:

containers that go missing are on a custom network

When you recreate docker.img custom networks need to be recreated.

[KyleCo]

14 hours ago, trurl said:

 In your case, appdata is already on cache but system share is on the array. System share is where docker.img lives, which contains the executables for all your containers. You want this share to be on cache for performance and so array disks won't stay spunup. But you have system share as cache-yes instead of prefer, which means move to array, and in fact, it is already completely on the array. 

Ok, so shutdown containers and disable docker, set the system share to prefer cache, and run mover. After that is completed I should be able to re-enable docker and restart containers. Do I have that correct?

 

Thanks for looking into this btw, I was at a total loss

[trurl]

18 hours ago, KyleCo said:

Do I have that correct?

Yes, not clear that it will help with the problem that started this thread though. Frankly your description of that problem seems impossible since parity check doesn't do anything to any files. Possibly a hardware problem as suggested by itimpi. Parity check does cause all disks to run at the same time, so maybe a power or controller issue is causing a problem.

 

Also

On 11/3/2020 at 9:46 AM, trurl said:

Maybe flash problems? If the templates for your installed containers can't be read from flash then those containers will still exist but they can't be worked with in the webUI.

 

Go to the Docker page and move the slider at upper right from Basic to Advanced. Do you see any orphan containers in the list?

 

[KyleCo]

I've made the change to the System Share and I've initiated a parity check. I'm quite certain it has something to do with parity(Not directly for certain but possibly power like you stated) because I have ran fine for over 4 months without a parity drive and it's not until I installed the parity that these occurred. So it's the parity check or something that doesn't like the parity check running haha. I've got my fingers crossed that this check completes without issues and I'll report back.

 

I'm also hoping not the flash drive! It's quite new haha. And I did check the full drive before using Unraid, so again, fingers crossed!

 

Thanks everyone!

[KyleCo]

So the parity check is still running, but when I got home from work tonight I saw that I am missing 2 containers now. 

 

EVEN WORSE, I AM NOW MISSING ABOUT 20TB of data off my array!!!! Almost an entire folder in one of my shares was completely blown away.

*EDIT* I realized what happened, it was a rouge container of mine. Is there anyway to roll back the array or is there a trash bin by chance to recover stupidly deleted files? *EDIT*

 

And when I look at the docker tab, a lot of containers show stopped, others say they are running but fail to connect in the web browser and I tried to restart one for Krusader and got an error. I've attached the diagnostic file I just downloaded.

 

Please help!! I don't know what is going on, everything was working perfectly for so long before I installed the parity drive.

 

Thank you again!

tower-diagnostics-20201106-1953.zip

Edited November 7, 2020 by KyleCo
I'm dumb

[trurl]

docker.img corrupt. Why have you allocated 50G for docker.img anyway? 20G is usually much more than enough. I am running 17 dockers and they use less than half of 20G docker.img

 

Have you had problems filling docker.img? Making it larger won't fix anything, it will just make it take longer to fill.

 

Why are your disks so hot? Maybe you affected cooling in some way when you added parity disk.

[KyleCo]

1 hour ago, trurl said:

docker.img corrupt. Why have you allocated 50G for docker.img anyway? 20G is usually much more than enough. I am running 17 dockers and they use less than half of 20G docker.img

 

Have you had problems filling docker.img? Making it larger won't fix anything, it will just make it take longer to fill.

I had an issue early on with filling the docker.img due to wrong mapping for a container and increased the size. I've fixed the issue but just never changed the size back.

1 hour ago, trurl said:

Why are your disks so hot? Maybe you affected cooling in some way when you added parity disk.

What do you consider hot? I rarely see them spike above 35. They were performing a mass deletion today while I had stupidly setup a handbrake watch folder incorrectly and it proceeded to delete 20TB of data haha. That may have generated some heat with them all running simultaneously.

[trurl]

1 minute ago, KyleCo said:

What do you consider hot? I rarely see them spike above 35.

Nov  3 07:48:26 Tower autofan: Highest disk temp is 51C, adjusting fan speed from: 130 (50% @ 1295rpm) to: FULL (100% @ 1294rpm)
Nov  3 08:58:37 Tower autofan: Highest disk temp is 43C, adjusting fan speed from: FULL (100% @ 1293rpm) to: 205 (80% @ 1284rpm)
Nov  3 09:03:42 Tower autofan: Highest disk temp is 45C, adjusting fan speed from: 205 (80% @ 1294rpm) to: FULL (100% @ 1295rpm)
Nov  3 09:08:47 Tower autofan: Highest disk temp is 43C, adjusting fan speed from: FULL (100% @ 1280rpm) to: 205 (80% @ 1285rpm)
Nov  3 09:13:52 Tower autofan: Highest disk temp is 46C, adjusting fan speed from: 205 (80% @ 1294rpm) to: FULL (100% @ 1288rpm)
Nov  3 09:18:58 Tower autofan: Highest disk temp is 43C, adjusting fan speed from: FULL (100% @ 1289rpm) to: 205 (80% @ 1285rpm)
Nov  3 09:24:03 Tower autofan: Highest disk temp is 45C, adjusting fan speed from: 205 (80% @ 1286rpm) to: FULL (100% @ 1284rpm)

And lots more like that. Also, if you look at SMART for each of your disks in those diagnostics you can see max lifetime temp in the reports

[KyleCo]

3 minutes ago, trurl said:

And lots more like that. Also, if you look at SMART for each of your disks in those diagnostics you can see max lifetime temp in the reports

Oh thanks! I'll check into that and add some more fans. I appreciate the heads up, full load or not I would rather the temps stay lower than that.

[trurl]

To fix docker.img, you will have to delete and recreate it. Might as well make it 20G while you're at it. Then if you had any custom docker networks you will have to recreate them. Then, you can reinstall your dockers exactly as they were using the Previous Apps feature on the Apps page.

[KyleCo]

13 minutes ago, trurl said:

To fix docker.img, you will have to delete and recreate it. Might as well make it 20G while you're at it. Then if you had any custom docker networks you will have to recreate them. Then, you can reinstall your dockers exactly as they were using the Previous Apps feature on the Apps page.

What does it do if you just disable docker in the settings and then change the size in there? That's how I've done it every other time haha.