Memory Max and Usable sizes and Docker

[SiliconFalcon]

 

Hi friends,

Why do I only show 30GB of usable memory and the Docker allocation is full?

How do I fix this?

 

Thanks in advance

[Michael_P]

20 minutes ago, SiliconFalcon said:

Why do I only show 30GB of usable memory

 

Your board supports up to 64G, you only have 32G installed

 

20 minutes ago, SiliconFalcon said:

Docker allocation is full

 

You have a docker that's writing to the docker image instead of a disk share

[SiliconFalcon]

6 hours ago, Michael_P said:

 

Your board supports up to 64G, you only have 32G installed

 

 

You have a docker that's writing to the docker image instead of a disk share

I guess I don't understand. Is the docker image stored someplace else? I have 18TB free on my array.

[trurl]

29 minutes ago, SiliconFalcon said:

Is the docker image stored someplace else?

Wherever you have it set to be stored in Docker Settings.

 

As mentioned, the usual cause of filling docker.img is an application writing to a path that isn't mapped to host storage.

 

Attach diagnostics to your NEXT post in this thread.

 

Also, go to Docker page, click Container Size button at bottom, post the results.

[Vr2Io]

9 hours ago, SiliconFalcon said:

 

Hi friends,

Why do I only show 30GB of usable memory and the Docker allocation is full?

How do I fix this?

 

Thanks in advance

Docker utiliize not means memory usage, it ref. to docker image file usage for my understanding.

Edited February 24, 2023 by Vr2Io

[SiliconFalcon]

On 2/23/2023 at 10:30 AM, Michael_P said:

 

Your board supports up to 64G, you only have 32G installed

 

 

You have a docker that's writing to the docker image instead of a disk share

I guess I don't understand. Is the docker image stored someplace else? I have 18TB free on my array.

tower-diagnostics-20230224-1119.zip

[Michael_P]

10 minutes ago, SiliconFalcon said:

Is the docker image stored someplace else?

 

Your docker image file is stored on your 'docker' share in /mnt/user/docker/

 

If your docker image file is getting filled, you have a container that's writing to that file instead of a disk share - you need to verify that whatever folders (downloads, temp and such) are configured in your docker containers point to a folder in one of your shares or disks.

[itimpi]

13 minutes ago, SiliconFalcon said:

I guess I don't understand. Is the docker image stored someplace else? I have 18TB free on my array.

tower-diagnostics-20230224-1119.zip 139.16 kB · 0 downloads

You have your system set to use the ‘docket’ share configured to store docker containers and currently that is stored on disk1 (which is not ideal from a performance perspective).   You also have the Minimum Free Space set to 0 which can cause problems if the array disk gets really full (as is the case for disk1).

 

Did you mean for that share to be on the array which is where it ends up if you have Use Cache=Yes set for it?   If you really wanted it to end up on the ‘cache’ pool for better performance then you should have Use Cache=Prefer set for that share so that mover puts the files on the pool if space permits.   Just a note that if that is what you want you will want the docker service disabled while mover is transferring files to the cache pool as mover will not transfer open files.

[trurl]

In your case, you have specified a folder for docker instead of a docker.img file. And, incredibly, you have filled that up. 9TB is easily the most I have ever seen by several orders of magnitude.

Filesystem      Size  Used Avail Use% Mounted on
overlay         9.1T  9.0T  105G  99% /var/lib/docker/overlay2/03842b0e87ad0dc324aeb22aa97f69b34801edda4321d816c7b7c4a55c340c62/merged

 

I never recommend a folder, it just seems like a way to avoid fixing the actual problem.

 

Usually, 20G docker.img file is more than enough, maybe a little more depending on how many containers you have. 9TB means you are doing it wrong.

 

18 hours ago, trurl said:

the usual cause of filling docker.img is an application writing to a path that isn't mapped to host storage.

 

[trurl]

18 hours ago, trurl said:

go to Docker page, click Container Size button at bottom, post the results.

  

I don't know how long that will take or even if it will work with 9TB

[SiliconFalcon]

I have been doing a ton of guess-work on this thing as I don't understand it all.

[SiliconFalcon]

I don't see that I am restricting this share to a single drive. What am I doing wrong... please don't say "EVERYTHING"!

 

[itimpi]

2 hours ago, SiliconFalcon said:

I don't see that I am restricting this share to a single drive. What am I doing wrong... please don't say "EVERYTHING"!

 

You are not per se, but the trigger for switching to another drive will be the free space falling below the Minimum Free Space size you have set.   At the point the file is created and the target disk is selected it has a size of 0 so is probably succeeding, then failing when the file grows to exceed the space available.   The recommended value for the Minimum Free Space setting is at least the size of the largest file you expect to have and 2 x that if you can afford it to give some headroom.

[SiliconFalcon]

On 2/24/2023 at 2:31 PM, itimpi said:

You are not per se, but the trigger for switching to another drive will be the free space falling below the Minimum Free Space size you have set.   At the point the file is created and the target disk is selected it has a size of 0 so is probably succeeding, then failing when the file grows to exceed the space available.   The recommended value for the Minimum Free Space setting is at least the size of the largest file you expect to have and 2 x that if you can afford it to give some headroom.

Well now I went and done it...
I used Unballance to move my docker files to a drive with more space now a couple of apps don't run. I tried reinstalling  Lidarr-Extended but get an error. 

docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/7475db062a75fb20eff1db54beb341e7bd7261ce4eabc9f50da58e5b6028be51-init/merged: no such file or directory.

 

I removed it from previous apps but it still will not reinstall.

Edited February 26, 2023 by SiliconFalcon

[SiliconFalcon]

Now none of my Dockers are working.

 

This sucks so bad 😭

 

Edited February 26, 2023 by SiliconFalcon

[trurl]

1 hour ago, SiliconFalcon said:

move my docker files to a drive with more space

You are going about this all wrong. There is absolutely no reason you need more than 9TB for docker. You may not even need 20G.

 

On 2/24/2023 at 11:37 AM, trurl said:

I never recommend a folder, it just seems like a way to avoid fixing the actual problem.

Go to Settings - Docker, disable Docker, delete docker and set it to use an img file instead of a folder. Go ahead and give it 40G just in case and we can go from there.

 

You can reinstall your containers using Previous Apps on the Apps page. I recommend you only install one at a time and figure out what might need to be corrected before installing the next.

 

On 2/23/2023 at 5:21 PM, trurl said:

the usual cause of filling docker.img is an application writing to a path that isn't mapped to host storage.

Each application must only write to container paths that you have mapped to host paths.

 

https://wiki.unraid.net/Manual/Docker_Management#Volume_Mappings

 

[SiliconFalcon]

Is there any clean up that I should do before reinstalling?

System settings that I have screwed up that I should fix.

Permissions ect...

[SiliconFalcon]

Since I screwed up the first time, I don't want to reinstall anything with the same settings. I will just rebuild.

I have gone in and removed all previous apps.

After going to the docker settings and deleting the docker file, I'm wondering if I should take a look and make sure that the Docker and Appdata folders are completely gone? If not, should I manually remove them?

 

[trurl]

If you want to be more "standard", there shouldn't be a docker share. docker.img by default is in system share along with libvirt.img

https://wiki.unraid.net/Manual/Shares#Default_Shares

 

If you don't mind starting over then it might be simpler to get rid of appdata just in case you put a bunch of stuff in there that shouldn't be.

 

I notice you said Appdata and Docker. Linux is case-sensitive, so Appdata and appdata are completely different. Be sure you don't make this mistake when setting anything up. Could be that is exactly what you have done, by specifying paths in an application that is different than the container paths you have mapped to host paths.

[SiliconFalcon]

ok I got Docker back up and 3 apps installed.

I think I did it right this time 🤔

 

tower-diagnostics-20230226-2246.zip

[trurl]

OK so far, but your docker share has files on disk7 as well as cache.

What do you get from command line with this?

ls -lah /mnt/disk7/docker

and this?

ls -lah /mnt/cache/docker

 

[SiliconFalcon]

root@Tower:~# ls -lah /mnt/disk7/docker
total 41G
drwxrwxrwx 2 nobody users  36 Feb 26 18:33 ./
drwxrwxrwx 8 nobody users 137 Feb 26 18:33 ../
-rw-rw-rw- 1 nobody users 40G Feb 27 18:08 docker-xfs.img

root@Tower:~# ls -lah /mnt/cache/docker
total 0
drwxrwxrwx 2 nobody users  6 Feb 26 18:33 ./
drwxrwxrwx 5 nobody users 49 Feb 26 18:23 ../

 

[trurl]

Nothing can move open files.

Disable Docker in Settings.

Run Mover so your docker-xfs.img is moved to cache.