Hey, I have a newish setup with 2 x 128GB SSDs (mirrored) as cache drives. It's clearly not enough for my daily usage as the drives are regularly filling resulting in a corrupted docker image. Mover is trying to keep up but every couple of days the server will be super busy and the cache will fill up. Short of buying more cache drives, is there a way I can protect the docker image from a full cache drive, or at least make mover a little more efficient?

I've attached my diagnostics and a screenshot of my mover settings.

 

Thanks

tobor-server-diagnostics-20180816-1113.zip

What do you have set as the Min Free Space for the cache drive?    The recommended value is twice the size of the largest file you are likely to want to write to the array.   When the free space on the cache drive drops below this value unRAID will stop writing new files to the cache drive and instead start writing new files directly to the array drives bypassing the cache drive.

In addition to Min Free you might also reconsider how you use cache.

 

There is no requirement to cache user share writes. If you write directly to the array, no cache space is needed, nothing needs to be moved, and those writes are immediately protected by parity. Some of us only use cache for dockers and VMs. With no more cache space than you have that might be something for you to consider, or maybe you could cache only certain shares.

 

I have 2x275GB cache pool. No user shares are set to cache-yes. I don't have any VMs since dockers do everything I need. My cache is used for dockers and for a few cache-only user shares. My cache-only user shares have a copy of my frequently accessed music and photos so other network devices can read them without spinning up any array disks.

 

I also have a cache-only share for DVR recording. This makes for better performance for multiple recordings / playbacks. Usually these are deleted after watching. If I want to keep any I manually move them to array shares.

 

So, mover never has anything to do on my system.

Thanks for your reply. That may be the issue, however you mention "min free space for the cache drive", my understanding is the min free space is set for each share.

docker.img is in the "system" share which has 0KB min free space

"media" share has about 20GB min free space.

"appdata" doesn't get much of a workout, but that's also on 0KB

1 minute ago, trurl said:

In addition to Min Free you might also reconsider how you use cache.

 

That's probably a good solution for my paltry 128GB cache, however:

• downloads are much much faster when using cache

• typically a movie/tv will get watched shortly after downloading, so having it on cache is pretty neat.

• uploading to the swarm is much quicker while downloading to cache

 

I'd also like to get an idea of how things are working under the hood so I have a better idea of how to plan any upgrades.

6 minutes ago, enmesh-parisian-latest said:

Thanks for your reply. That may be the issue, however you mention "min free space for the cache drive", my understanding is the min free space is set for each share.

docker.img is in the "system" share which has 0KB min free space

"media" share has about 20GB min free space.

"appdata" doesn't get much of a workout, but that's also on 0KB

No.   The Min Free Space for shares refers to the Minimum Free Space on an array drive that needs to be present for the drive to be considered for writing new files to that drive.   It is nothing to do with the Min Free Space on the cache drive which is used separately to decide if files can be written to the cache drive when a share is set to Use Cache=Yes.before later being written to the array by mover.

2 hours ago, itimpi said:

No.   The Min Free Space for shares refers to the Minimum Free Space on an array drive that needs to be present for the drive to be considered for writing new files to that drive.

 

I assume I need to stop the array to access this setting? I can't see it in the cache drive settings...

 

EDIT: Stopped array but no setting ?

Edited August 16, 20187 yr by enmesh-parisian-latest

Not at my unRAID server at the moment, but I seem to remember it being under the Global Share settings.

4 hours ago, itimpi said:

Not at my unRAID server at the moment, but I seem to remember it being under the Global Share settings.

 

Yes you're right, it's set currently to 2,000,000.

 

There's certainly something seriously wrong still. After disabling cache for "media" and deleting docker.img several times today, the unraid docker tab is very sluggish to load and docker containers are very slow to load if they load at all (most don't). I'd really appreciate if someone with some more experience than myself could take a quick look at my diagnostics, I've spent a few days trying to work this out without any results so far.

 

EDIT: After more troubleshooting, I think there may've been a corrupted file created by deluge. I've deleted the last few days of deluge data and nuked the appdata.

 

EDIT 2: Was working for a while but nope, using Krusader and triggered something. Like day and night, suddenly all containers are lagging, can't refresh. The exact use which triggered the behaviour (after 2 hours of perfect use) was; open krusader tab > navigate to a "media/downloads" share bookmark, no response, all containers now unresponsive, unraids docker page now takes ~20 secs to load (usually instant). I stopped containers, deleted docker image, started docker and loaded plex again, now running smoothly again. Can someone shed some light on what's wrong here?

 

tobor-server-diagnostics-20180816-1848.zip

Edited August 16, 20187 yr by enmesh-parisian-latest

Most of your disks are very full. And most of your User Shares don't have any Minimum Free set. And no shares are set to exclude any of those full disks, so you are perfectly setup to get out-of-space when trying to write to your shares.

 

Minimum Free for each user share should be set to larger than the largest file you expect to write to the share. unRAID has no way to know how large a file will become when it chooses a disk to write it to. If a disk has less than Minimum Free, unRAID will choose a different disk. There is also a Minimum Free setting for cache in Global Share Settings.

 

 

And many of your shares are set to cache-prefer. This setting means "move files from array to cache if their is room". This is a perfect setup to fill up your cache drive. Here is a link with more detailed explanation of the Use cache options:

 

https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

 

Also, minor detail I noticed, you are ignoring the warning for multiple keys. The reason this is a warning is because this can cause the wrong key to be replaced if you ever need to go through the key replacement.

8 minutes ago, trurl said:

Most of your disks are very full.

 

Thanks very much for checking,

 

As per your recommendations:

- I've disabled cache for all shares except "system" and "appdata"which are now set to "yes".

- I've added appropriate min-free amounts to all shares.

- I've reduced data on some of the fullest disks using unbalance

 

However I still have a mystery issue causing the docker image to seize up. Is there some way I can identify what's causing the problem?

1 hour ago, enmesh-parisian-latest said:

I've disabled cache for all shares except "system" and "appdata"which are now set to "yes".

 

'Yes' is definitely the wrong setting for those. Did you bother to read the link I gave? 'Yes' is going to move those from cache to the array.

 

Those shares are set to 'prefer' by default so they will try to stay on cache. If you do get some overflow to the array, it will be moved back to cache. That setting also allows people who start without a cache to get those shares moved to cache. 'Prefer' is really the preferred setting for those but 'Only' is also used by people who know they aren't going to fill up cache.

 

You should set those back to 'Prefer' then go to the User Shares page and Compute on each of those to make sure they don't have anything on the array. If they do, you should make room for them on cache, stop Docker and VM services, and run mover to get them back on cache.

 

 

6 minutes ago, trurl said:

 

'Yes' is definitely the wrong setting for those. Did you bother to read the link I gave? 'Yes' is going to move those from cache to the array.

 

Thanks very much for clarifying, I had "read" the link but I incorrectly though I understood the cache setting so somehow skimmed over all the info I needed to read. The "prefer" setting is now set for both shares.

 

I'll re-read the link and try what you've suggested. There's still something wrong, however I expect it can be fixed once shares can write back to the cache.

1 hour ago, trurl said:

'Prefer' is really the preferred setting for those but 'Only' is also used by people who know they aren't going to fill up cache.

Sorry see correction above

Yes, thanks for the link, it's really helpful. I'll slowly add the offending containers and see how I go.

 

On 2/18/2017 at 1:28 PM, RobJ said:

Table of Cache drive usage options and their behaviors

    C=Cache drive

    D=Data drive(s)

 

                                Cache:No    Cache:Yes    Cache:Only  Cache:Prefer

Data should be on:          D              C+D                C             C+D

New files first to:             D                C                   C               C

Files overflow to:             -                 D                    -                D

 Mover moves:                 No           C to D               No            D to C

Orphaned files:                C               -                     D                -

Edited August 16, 20187 yr by enmesh-parisian-latest

Hey so I've made all the suggested changes however docker is still crapping itself after a few hours. There's no appdata or system files outside cache, so I'm assuming there's a corrupted settings file somewhere which is still causing issues, any idea how I can identify the source?

 

The attached diagnostics file was exported a few mins after docker crapped itself. At the time the most intensive action was a beets import of music in the "media" share, I could tell something had happened when the plex music stream cut out. I've found I can temporarily fix the issue by stopping and starting all containers, no need to delete the docker.img.

 

On side note, the Unraid terminal wouldn't load this morning until I restarted the machine, I had the same issue as this guy: 

 

Machine hung for ~45mins during reboot, needed to hold the power button down to get a power cycle. There's something amiss with the system.

 

tobor-server-diagnostics-20180817-1418.zip

Edited August 17, 20187 yr by enmesh-parisian-latest

I'm not seeing anything in the diagnostics. Have you tried deleting docker.img?

3 hours ago, trurl said:

I'm not seeing anything in the diagnostics. Have you tried deleting docker.img?

 

Yes over the last few days I've deleted the docker file about 6-7 times to keep the media server up and running. It's clear now that it's not fixing the issue, so I just stop & start all containers to temporarily fix the issue.

 

Edit: I've noticed that the problem occurs more quickly if beets is importing music from the download folder, is it possible the beets config files are corrupted, or something to do with the media itself? 

Edited August 17, 20187 yr by enmesh-parisian-latest

I don't know what else to suggest except possibly you have filesystem corruption on one or more disks, possibly in the cache pool, due to filling them up. I didn't really see anything in the diagnostics to indicate it though.

 

I guess you could try starting over with the cache pool unless somebody else has another idea. @johnnie.black hasn't chimed in on this thread yet.

Don't see any errors on the diags, please post current ones, grabbed after there are issues with the docker image.

Thanks for the replies, I really appreciate the time and insight.

 

Everything has been running pretty solid now that i've stopped the beets import. I will restart the process once the night time media rush stops, if docker craps out again I'll try deleting all the beets appdata.

 

I've added the latest diagnostics, however there hasn't been an issue since the "tobor-server-diagnostics-20180817-1418.zip" file posted earlier, this was created a few mins after one of the errors.

tobor-server-diagnostics-20180817-2251.zip

Nothing there, and if the docker image was really corrupt it would always show on the syslog, you might be having a different issue.

9 minutes ago, johnnie.black said:

Nothing there, and if the docker image was really corrupt it would always show on the syslog, you might be having a different issue.

Any suggestions on how to isolate the issue? There's definitely something wrong.

If it's a specif docker causing the problems try posting in that docker's support page.

Deleted all beets appdata & started from scratch, seems to be working ok for last few hours. I assume some part of the beets library was corrupted when the cache drive filled but couldn't overflow onto the array (due to incorrect "appdata" cache settings).

 

I still don't understand how a problem in one container can shut down all the other containers, I was under the impression that they operated in a sandbox to prevent issues like this.

 

Thanks for the help, I'll post back if I find any more info.