Hey Friends,

 

Will try to be brief and just let you review the diagnostics as you do.

 

Long story short -- my Cache seems to fill up and sort of stall out the system for the day until I force move, or it hits its nightly move.  Do I just need to get a bigger cache drive?  Do I need to limit the quantity of downloads at any given time?  Does this make sense for it to basically lock up the rest of my server once it hits capacity?  Happy to share screenshots of my shares cache settings if needed. 

 

I also seem to be filling up my docker image way to regularly.  I ran about the forums here for a bit trying to find a solution and tried a handful, including limiting the log size for all of my dockers.  It seemed to work at first but has since went back to filling up.  Again do I just need a bigger cache drive? 

 

Lastly, Radarr/Sonarr doesn't seem to always move my files to the appropriate media folder after finishing downloads.  I am thinking this will be solved after I get the above two issues solved, but it just leaves me scratching my head and plenty of files being left in Downloads/completed.

 

Any and all help would be appreciated.

Thanks!

 

 

tower-diagnostics-20190509-1818.zip

May  6 04:40:29 Tower root: Fix Common Problems: Error: Invalid folder cach contained within /mnt

One of your docker containers is referencing on the template /mnt/cach instead of /mnt/cache.  This is resulting in your ram being filled up very quickly

 

Probably also, one or more of your apps is writing into the docker.img.  Common mistake would be say within your download client in it's settings, telling it to put files at downloads (instead of /downloads), or at /Downloads (when only /downloads exists) -> all assuming you've mapped /mnt/user/Downloads to /downloads

There are a number of things we need to work through, and unfortunately it is very late for me so we might not get to all of them tonight.

 

Your docker image is 20G, the recommended size, and it isn't close to filling currently. It is, however, corrupt. Maybe you filled it in the past. We might have to get into the details of your applications to see how you might be filling it, but basically the reason an application writes into the docker image instead of into Unraid storage is because the application somewhere in its settings specifies a path that isn't mapped to Unraid storage in the volume mappings. Common mistakes are not using the same upper/lower case when specifying paths, or using a path that isn't absolute (beginning with /).

 

In addition to the corrupt cache, Fix Common Problems indicates you have filled /var/log, which is true, but your syslog itself isn't that large so you must have something else writing there. A common culprit for this seems to be atop, which people install with Nerd Pack. We would have to take a look at the contents of /var/log to know better. You should probably reboot now to get this under control at least temporarily.

 

Another thing FCP is giving you an error for is a folder in /mnt named cach. This is almost certainly due to a typo in one of your container mappings. /mnt/cach is not a path to mounted Unraid storage, and so is actually a location in RAM.

 

And, of course, you have filled the cache drive, probably with your Downloads share. You have also nearly filled disk1, but I don't notice anything in your share settings that would cause it to keep putting stuff there instead of disk2, so maybe you added disk2 more recently.

 

We probably need to start with making some room on cache before we can get the other things fixed.

1. Reboot to get /var/log emptied, and remove atop if you have it.
2. Disable docker service and delete the docker image. Disable VMs also.
3. Set appdata to cache-only. It has some things not on cache, but since there isn't room yet we need to have Mover leave it for now.
4. Run Mover and wait for it to complete.
5. Post a new diagnostic.

 

 

 

 

Thanks for the help, no worries on getting it busted out tonight.  I am going to try and be overly detailed here with my steps and responses as to hopefully alleviate the need for too much back and forth just for you to gather the correct information from me.

 

Bare with me if I am not seeing things or still not quite getting what you need -- I am fairly new to Unraid, but would like to say I am competent

 

1 hour ago, trurl said:

You have also nearly filled disk1, but I don't notice anything in your share settings that would cause it to keep putting stuff there instead of disk2, so maybe you added disk2 more recently.

 

 

 

 

 

Speaking to the above really quickly -- I just added the new drive less than a week ago as I had filled the first almost entirely.  Is it recommended to have Unraid balance the storage a bit more??

 

 

You and Squid both mention somewhere I may have a container pointing to "cach" instead of "cache" -- unfortunately I am not seeing any of my mappings pointing to "cach" or "cache" when I go in to edit them.  This is a new problem within the last week or so.

 

Here are my download paths in qBit's WebUI, and then the container mappings in the GUI.  This seems to be operating well from what I can tell but maybe not.

 

 

Anyways, on to the steps you provided.  I did a clean reboot. Docker services stopped and image deleted.

 

 

VMs Disabled

 

 

Appdata was "prefer" now "Only"

 

Mover is running now -- will report with fresh diagnostics when it finishes

Edited May 10, 20197 yr by WhatPlantsCrave

New diagnostics attached here.

 

Also atop was showing as "uninstalled" here.  Not sure what it was before going through the steps above.

 

tower-diagnostics-20190510-0108.zip

Edited May 10, 20197 yr by WhatPlantsCrave

7 hours ago, WhatPlantsCrave said:

You and Squid both mention somewhere I may have a container pointing to "cach" instead of "cache" -- unfortunately I am not seeing any of my mappings pointing to "cach" or "cache" when I go in to edit them.  This is a new problem within the last week or so.

 

Here are my download paths in qBit's WebUI, and then the container mappings in the GUI.  This seems to be operating well from what I can tell but maybe not.

This is not the complete volume mappings for that container, and it is not your only container according to your previous diagnostics. Since dockers are disabled currently we will have to look into this later.

 

These new diagnostics show you still have some of Downloads share on cache, but cache has a lot of room now. The last thing in the syslog is mover starting. Are you sure it completed?

 

You can see which disks are used by each share by going to User Shares and clicking on Compute... or use the Compute All button. It may take a while to get a result. If you don't get any results after a while try refreshing the screen.

 

We can go ahead with the next part though, which is to get appdata moved to cache where it belongs.

 

Set appdata to cache-prefer then run mover. Wait for it to complete, check the user shares to see which disks they are using, and post new diagnostics.

2 hours ago, trurl said:

These new diagnostics show you still have some of Downloads share on cache, but cache has a lot of room now. The last thing in the syslog is mover starting. Are you sure it completed?

 

You can see which disks are used by each share by going to User Shares and clicking on Compute... or use the Compute All button. It may take a while to get a result. If you don't get any results after a while try refreshing the screen.

 

Here is a quick screenshot of that.  I have to head off to work, but I have appdata cache-prefer and mover is running.  Will report back when I get home.

 

Ok, decided to come home on my lunch break to run those Diagnostics for you.

 

See attached here.

 

tower-diagnostics-20190510-1229.zip

Edited May 10, 20197 yr by WhatPlantsCrave

appdata is all on cache now, but Downloads still has some on cache. Possibly these are duplicates of files on the array. You might turn on Mover Logging and try to move again so we can see what it says about that stuff that won't move.

1 hour ago, trurl said:

appdata is all on cache now, but Downloads still has some on cache. Possibly these are duplicates of files on the array. You might turn on Mover Logging and try to move again so we can see what it says about that stuff that won't move.

Hey Trurl,

 

I took a look at what was in Downloads that was on cache and there was a lingering incomplete movie.  It was nothing crucial so I opted to just delete it and I can re-download it once we are sorted.

 

Once again another Diagnostics attached.

tower-diagnostics-20190510-1737.zip

Edited May 11, 20197 yr by WhatPlantsCrave

That looks OK now.

 

Before putting things back together the way they were, it might be a good idea to reconsider how you are using cache. Why were you filling it up? 

I am thinking the cache was getting filled by downloads being sent there before being moved to the array -- the solution I have in mind is to limit the number of active torrents qBit allows at any given time to a lower number.  

 

Am I still showing an incorrect mapping to "cach" anywhere in my diagnostics?  Or will that pop-up once I enable dockers?

 

I don't really use my cache for much else other than routing the downloads to it, and then from my understanding all of the appdata, vms (not running any currently), etc.  The Appdata I assume contains plex media files?  I deleted all of the video thumbnails from plex to help save some space.

Since /mnt/cach isn't actual mounted storage, it is in RAM, as already mentioned. So, it doesn't survive a reboot. If one of your dockers has a mapping for that folder, then it will be created when the container is started.

 

appdata is the "working" data for any of your docker applications. For example, your media files are not in appdata, but the plex database that keeps track of your media files is in appdata.

 

Go ahead and enable the docker service to recreate the docker image. Also, on that same page, make sure you have log rotation on.

 

You can reinstall your dockers exactly as they were using the Previous Apps feature on the Apps page. Take a look at each of your dockers to see if you see that cach mapping.

25 minutes ago, WhatPlantsCrave said:

I am thinking the cache was getting filled by downloads being sent there before being moved to the array -- the solution I have in mind is to limit the number of active torrents qBit allows at any given time to a lower number. 

Another thing to consider is that Mover can't move any open files. This is why I had you disable docker before trying to get things moved.

 

If you are seeding torrents from cache, they may not get moved. NZBs often need post-processing, and so there is some performance reasons to have them on cache. But torrents often don't need any post-processing, so it might make sense to have those go directly to the array instead of caching them. Then you wouldn't have these cache problems with torrents.

Ok I will get back to setting things up as they were -- and be sure to watch out for "cach" anywhere.  I didn't see it when rolling through any, I am wondering if it was from an old Deluge docker that was giving me trouble a while back.

 

Will report back if there are further problems.

 

Note: Docker log rotation was disabled.

Edited May 11, 20197 yr by WhatPlantsCrave

8 hours ago, WhatPlantsCrave said:

watch out for "cach" anywhere.  I didn't see it when rolling through any, I am wondering if it was from an old Deluge docker that was giving me trouble a while back.

But if you had ever rebooted (that path is RAM as noted) since getting rid of that old deluge then it wouldn't have been in those diagnostics.

 

Make Fix Common Problems do a scan after you get things going again and it will tell you. That is why we saw it in the diagnostics before.

 

Don't ignore errors or warnings from Fix Common Problems, unless you understand exactly why it is OK to ignore them in your specific use.

 

Also, don't ignore SMART warnings if Unraid ever shows you any on the Dashboard.

 

Set up Notifications to alert you immediately by email or other agent as soon as any problems are detected. This is especially important for disk problems. If you allow disk problems to accumulate you will lose data.