[SOLVED] aneely - docker image filling up

[aneelley]

UGH.  I am now having this exact issue.  I got a warning that the docker image file was filling up.  Sure enough, log file on sonarr says it is looking in /usr/local/bin/nzbget/downloads/completed/shows/ for the completed shows.  The actual path where they are located is mnt/user/downloads/completed/series.  Not sure what I need to change on the nzbget interface to have it let sonarr properly proces.

[trurl]

3 minutes ago, aneelley said:

I got a warning that the docker image file was filling up.

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your NEXT post.

 

Also, post the docker run command for your Sonarr docker as explained at this very first link in the Docker FAQ:

 

 

[Squid]

9 minutes ago, aneelley said:

UGH.  I am now having this exact issue.  I got a warning that the docker image file was filling up.  Sure enough, log file on sonarr says it is looking in /usr/local/bin/nzbget/downloads/completed/shows/ for the completed shows.  The actual path where they are located is mnt/user/downloads/completed/series.  Not sure what I need to change on the nzbget interface to have it let sonarr properly proces.

You paths don't match between nzbget and sonarr, and additionally, you're telling nzbget to save the downloads within it's /config folder.  See here for another ongoing example  https://forums.unraid.net/topic/91138-cache-not-being-moved/?tab=comments#comment-846536

 

[aneelley]

unraid-diagnostics-20200419-1758.zipI have been reworking mine for a couple of hours now.  I can still see this in the Sonarr log:

20-4-19 15:43:35.2|Error|DownloadedEpisodesImportService|Import failed, path does not exist or is not accessible by Sonarr: /usr/local/bin/nzbget/downloads/completed/shows/

 

Sonarr container path for /data is /mnt/user/downloads/

Sonarr container path for /media is /mnt/user/shows/

nzbget container path for /data is /mnt/user/downloads

Inside nzbget, the MainDir is set to /data

Inside nzbget, the DestDir is set to ${MainDir}/dst

Inside nzbget, the InterDir is set to ${MainDir}/inter

Edited April 19, 2020 by aneelley
inserting diagnostics.

[trurl]

Since you already had this on 2 different threads (I won't say they were hijacks exactly since one was dead and the other resolved), I have collected all the posts related to your issue into this thread.

 

You have literally the largest docker image I have ever seen, 2TB, and you are still filling it up.

 

I always recommend 20G for docker image. That should be more than enough, and if it is growing you have something wrong.

 

Of course, you know you have something wrong, but I suspect you have had something wrong for a while now. I sort of looked back at some of your post history and you have had problems with your dockers for years and your fix seems to be mostly just deleting docker image and starting over.

 

How long have you had a docker image larger than 20GB?

 

The usual reason for filling docker image is an application writing to a path that isn't mapped. Typical mistakes are using a different upper/lower case than specified as the container path in the mappings, or specifying a relative instead of an absolute path.

 

[aneelley]

41 minutes ago, trurl said:

Since you already had this on 2 different threads (I won't say they were hijacks exactly since one was dead and the other resolved), I have collected all the posts related to your issue into this thread.

 

You have literally the largest docker image I have ever seen, 2TB, and you are still filling it up.

 

I always recommend 20G for docker image. That should be more than enough, and if it is growing you have something wrong.

 

Of course, you know you have something wrong, but I suspect you have had something wrong for a while now. I sort of looked back at some of your post history and you have had problems with your dockers for years and your fix seems to be mostly just deleting docker image and starting over.

 

How long have you had a docker image larger than 20GB?

 

The usual reason for filling docker image is an application writing to a path that isn't mapped. Typical mistakes are using a different upper/lower case than specified as the container path in the mappings, or specifying a relative instead of an absolute path.

 

I have had it for years.  I moved in November and after having the system off for a year even then, I am spinning it back up and trying to get things in order.  I had it originally set to 1TB but it started filling up so today or yesterday, I gave it another TB as a stop gap and now I am re-investigating to try and make sure I have everything configured correctly for the long haul.  FWIW (funny or not), I am a storage engineer.  Been messing with Unraid for years but as things change with the containers, it seems things break.  I can provide you whatever diags you ask.  Just trying to get the environment stable and any old plugins removed that could now be defunkt, etc.  Thanks!

[trurl]

46 minutes ago, aneelley said:

as things change with the containers, it seems things break. 

I use some of these same dockers, and nothing has changed that will break them in this manner. Your setup has been wrong from the beginning, as illustrated by your filling an incredibly huge docker image.

 

Somewhat surprisingly, your appdata, domains, and system shares are all on cache as they should be.

 

Your docker image, which is going to be the first thing we deal with since it is ridiculous, is not on any user share. It is not even specifically on any disk. You have configured it to be at the root of the user shares, /mnt/user/docker.img.

 

Go to Settings - Docker, disable dockers, and delete the docker image. Change the path of docker image to /mnt/user/system/docker.img, and set its size to 20GB. Then enable docker, but don't install any dockers yet. That will put your docker image on the system share where it belongs, and since the system share is cache-prefer, it will wind up on cache where it belongs.

 

When you have done that much post new diagnostics so I can verify. Then we can begin working on getting your dockers set up correctly. Maybe after we work through one or two of them you will understand how this is supposed to work.

[aneelley]

I am on it and I seriously appreciate your help!  I will reply shortly.

[trurl]

1 hour ago, aneelley said:

I am spinning it back up and trying to get things in order. 

Looking at the SMART folder in your diagnostics, there are some other things you might consider. I'll just put this here for later discussion if it seems appropriate.

 

You are using a controller that isn't passing the serial number or SMART information from your disks. Disk controllers are not my area of expertise so I am going to see if @johnnie.black has any recommendations. Fixing that may be more trouble than it is worth, but I don't think Unraid can monitor your disk health the way it is, and you have a lot of disks and only single parity.

 

I always recommend fewer, larger disks instead of more, smaller disks. Larger disks perform better and are less expensive TB/$ than smaller disks, and each additional disk is an additional point of failure that requires more hardware to support it

.

[aneelley]

Ok, I have done as you requested and please find the diagnostics below.

unraid-diagnostics-20200419-2030.zip

[aneelley]

8 minutes ago, trurl said:

Looking at the SMART folder in your diagnostics, there are some other things you might consider. I'll just put this here for later discussion if it seems appropriate.

 

You are using a controller that isn't passing the serial number or SMART information from your disks. Disk controllers are not my area of expertise so I am going to see if @johnnie.black has any recommendations. Fixing that may be more trouble than it is worth, but I don't think Unraid can monitor your disk health the way it is, and you have a lot of disks and only single parity.

 

I always recommend fewer, larger disks instead of more, smaller disks. Larger disks perform better and are less expensive TB/$ than smaller disks, and each additional disk is an additional point of failure that requires more hardware to support it

.

Understood.  I am running some HP 3TB shelves on it and they were free.  Except for the power consumption of course!

[trurl]

That looks good.

 

Since you are a storage engineer, I assume you know about paths generally, and the difference between a relative and an absolute path. I will also mention that Linux is case sensitive, so "Downloads" and "downloads" are different paths. As I mentioned above

2 hours ago, trurl said:

The usual reason for filling docker image is an application writing to a path that isn't mapped. Typical mistakes are using a different upper/lower case than specified as the container path in the mappings, or specifying a relative instead of an absolute path.

These application paths I mention above are the paths set within the application itself, not in the mappings.

 

Do you understand docker volume mappings? That is the thing that often confuses people in the beginning with dockers, and they wind up getting things sort of working by blindly following someone else.

 

I think the word "map" may not make sense to some people in this context, though it is often used in this somewhat abstract way in other contexts such as mathematics.

 

A mapping is just a correspondence. Just as a place on a road map corresponds to a place in the world. A docker volume mapping is a correspondence between a path on the host (Unraid) and a path within the container.

 

Port mapping is similar, a correspondence between ports on the host and ports within the container. If you understand mappings you are most of the way there to setting up any docker.

 

Another piece of this puzzle is the paths to the user shares. The user shares are at /mnt/user/... The Unraid webUI helps you out with this when setting up a docker since it gives you these paths to choose from when setting up a docker, for example, /mnt/user/appdata.

 

Which dockers have you been using?

 

[aneelley]

3 minutes ago, trurl said:

Do you understand docker volume mappings? That is the thing that often confuses people in the beginning with dockers, and they wind up getting things sort of working by blindly following someone else.

Yes, I think so.  I know there are the paths that unRAID knows about (absolute I think those are?) and the relative (which I believe are the paths that you set on the container configuration (they are relative to what you put in there)).

5 minutes ago, trurl said:

A mapping is just a correspondence. Just as a place on a road map corresponds to a place in the world.

Nice, I like that.

5 minutes ago, trurl said:

Which dockers have you been using?

nzbget, sonarr, radarr and plex.  I have tried other misc file managers, etc.

[trurl]

1 minute ago, aneelley said:

I know there are the paths that unRAID knows about (absolute I think those are?) and the relative (which I believe are the paths that you set on the container configuration (they are relative to what you put in there)).

Absolute and relative paths are a general concept and not really about Unraid specifically.

 

An absolute path is any path beginning with / on Linux. On windows, an absolute path begins with \ since that is the path separator used by Windows. An absolute path begins at the root of the filesystem, such a C:\ in Windows.

 

In Unraid, an absolute path will be a path like /mnt/user/... That path begins at the root and goes down into the mnt folder, then from there to the user folder, etc.

 

A relative path is any path relative to an already specified or assumed directory, often to the current directory. A relative path does not begin with /

 

For example, if you are already in the /mnt/user directory, then just specifying the relative path appdata is enough to refer to /mnt/user/appdata.

 

When setting up an application, there are usually paths that you must specify, such as where to download files. If you specify a relative path instead of an absolute path, that path will be inside the docker image. Relative paths are not allowed in docker mappings, so any container path is an absolute path within the container that corresponds to an absolute host path.

[aneelley]

Makes sense.

[trurl]

With plex, are you trying to do anything fancy with transcoding, such as hardware transcoding or transcoding in RAM?

[aneelley]

1 minute ago, trurl said:

With plex, are you trying to do anything fancy with transcoding, such as hardware transcoding or transcoding in RAM?

I think I am using it's defaults pretty much.

[trurl]

There are several different plex containers available in Community Applications, they have different defaults, and some of those defaults are incomplete.

 

Review this very first link in the Docker FAQ:

 

https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564345

Go to Apps and use the Previous Apps feature. Reinstall your plex docker and post the resulting docker run command.

 

[aneelley]

I just noticed I was using the linuxserver version of the container.  Ok if I use BinHex now since that is what I use with my other containers?

[trurl]

Whichever you prefer. I am more familiar with the linuxserver containers.

[trurl]

But Previous Apps is going to let you reuse the settings you already had for a container you were already using.

[aneelley]

13 minutes ago, trurl said:

There are several different plex containers available in Community Applications, they have different defaults, and some of those defaults are incomplete.

 

Review this very first link in the Docker FAQ:

 

https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564345

Go to Apps and use the Previous Apps feature. Reinstall your plex docker and post the resulting docker run command.

 

Here you go:

Command:root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='binhex-plex' --net='host' -e TZ="America/New_York" -e HOST_OS="Unraid" -e 'TRANS_DIR'='/config/transcode' -e 'UMASK'='000' -e 'PUID'='99' -e 'PGID'='100' -v '/mnt/user':'/media':'rw' -v '/mnt/user/appdata/binhex-plex':'/config':'rw' 'binhex/arch-plex' 

d75812f6fdc779644de9016d6b06c9813e3dd63846edf7c30aadc6370b4ae287

The command finished successfully!

[trurl]

That looks OK. Is it working as you expect? Since you changed dockers and presumably the appdata path, it will be a completely new install of plex and you will have to set up your media again.

 

[trurl]

Or you can use Previous Apps to install the linuxserver plex you had before.

[trurl]

If you go with the binhex plex, then according to your mappings

4 minutes ago, aneelley said:

-v '/mnt/user':'/media':'rw'

plex is going to see all of your user shares (/mnt/user) in the /media folder.