Mover not Moving

[Macj72x]

I apologize in advance if I'm asking for help on a common problem, I couldn't find the right answer in my searches.  I am struggling with getting my cache pool to move correctly. My cache pool keeps filling up and won't empty even when mover is running.  I have mover set to run every hour if the cache pool gets above 50%, but It will just continue to fill and not empty until a random point in time or if the server is restarted.

 

I understand that "Prefer" will keep all the files from that share on the cache until it fills and "Yes" will write to the cache first and then dump to the array once it is full or mover invoked.  I've checked my share sizes and everything with "Prefer" only adds up to a couple hundred GBs.  After checking the size of all my shares going into the cache, it appears that my media files being downloaded go in and don't move out.  My cache pool is four nvme drives at 2 TBs each, so it is pretty large and quick.

 

What am I missing?  I've attached my logs.  While looking through them I did notice this error "emhttpd: error: get_key_info, 567: Invalid argument (22): stat: /boot/config/Trial.key" I'm not sure why I'm seeing that, but I'm assuming it is unrelated.

Edited February 20, 2023 by Macj72x

[Squid]

What share(s) in particular are you talking about?  Also you have to keep in mind that any open file cannot be moved.  So if (for instance) you are seeding any media etc they will remain on the pool

[trurl]

26 minutes ago, Macj72x said:

I have mover set to run every hour

Running mover more often won't help if you are still writing to cache. Mover is intended for idle time. It is impossible to move from fast cache to slow array as fast as you can write to cache. If you try to move while still writing it only makes things worse.

 

What is the purpose of these 2 cache:prefer shares?

l------e                          shareUseCache="prefer"  # Share exists on cache
m-u                               shareUseCache="prefer"  # Share exists on cache

 

[Macj72x]

I'm using usenet to download files, so I shouldn't be seeding anything.

 

The idea to use the cache for downloading media was that I could get media onto the server with little delay and move to the array as needed.  Then the cache could allow for faster post processing.  It appears the download speed is slower than the array speed.  Am I wrong with this approach?

Edited February 19, 2023 by Macj72x

[Macj72x]

1 hour ago, Squid said:

What share(s) in particular are you talking about?  

It is my media share that seems to be the culprit.

 

51 minutes ago, trurl said:

What is the purpose of these 2 cache:prefer shares?

l------e                          shareUseCache="prefer"  # Share exists on cache
m-u                               shareUseCache="prefer"  # Share exists on cache

 

 The first is a lan cache that I haven't really finished setting up so it is pretty much empty except some folders.

 

The second is where I've stored some m3u files.

[itimpi]

The Prefer setting for Use Cache means you want files moved TO cache by mover.    If you want mover to move files from cache to the array you need the Yes setting.   There is good help built into the GUI for describing how the setting works and how it affects mover.

[kizer]

I personally use the Mover Tuning plugin and have it set to move after a week. That way i can access the files from the Cache and the array, but they are moved automatically after a period of time vs daily. Also the upside is I'm sure the files aren't open or being used unless one of my kids is up watching a video at 5AM. 😃

[Macj72x]

6 minutes ago, itimpi said:

The Prefer setting for Use Cache means you want files moved TO cache by mover.    If you want mover to move files from cache to the array you need the Yes setting.   There is good help built into the GUI for describing how the setting works and how it affects mover.

Yes, I think I understand this properly.  I'm still learning Unraid, so I'm probably doing plenty of things wrong, but I set Prefer to shares that will help the system run faster and Yes to shares that I want data dumped onto the system quickly then eventually moved into the array.

 

I figured I should have my media set to Yes, so I can quickly big data onto my server and be processed in a quicker fashion then moved off onto the array when the cache is near full.  Is this a bad way of looking at this?  Should I just set my downloaded media to go straight to the array?

[Macj72x]

While I still would like to know if I’m using the cache pool in a way that isn’t recommended, I still think something else is wrong with my mover or the way it is configured.  I’ve turned off docker. I now only have my Homeassistant VM running.  I’ve checked my open files and it doesn’t appear that I have anything with my media files open.
 

After this, I’ve invoked mover and it has been running for over and hour and has hardly moved anything. I started with about 250gb space remaining and am now at 267gb. Any guesses where to start?

[trurl]

Your "default shares" have some files on the array.

https://wiki.unraid.net/Manual/Shares#Default_Shares

Usually you want these all on fast pool (cache) so Docker/VM performance isn't impacted by parity, and so array disks can spin down since these files are always open.

 

But you don't have space to move them to cache currently. Set them to cache:only so mover will ignore them for now until you make space on cache for them.

 

I'll leave it up to you if you want to keep those other cache:prefer shares on cache. Currently they are all on cache with nothing on the array.

 

Run mover to get your cache:yes shares moved to the array. Then post new diagnostics and we can work on your default shares.

 

You can see how much of each disk is used by each user share by clicking Compute... for the share on the User Shares page, or the Compute All button.

[Macj72x]

 

 

I have changed the default shares to only.  I have deleted the other two shares listed as prefer as they didn't have any important data on them anyway.

 

I ran the mover and nothing has moved off the cache since you last posted approximately 2 hours ago.  Other than the default shares, all shares are either listed as Yes:Cache or No with nothing on the cache.  Computing all the share size showed that a vast majority of the cache data is from my media download/storage drive (named data).

Edited February 20, 2023 by Macj72x

[trurl]

Feb 19 18:59:00 ChunkyMonkey CA Backup/Restore: Starting binhex-nzbget... (try #1) 
Feb 19 18:59:00 ChunkyMonkey  shfs: cache disk full

and more. Are you still writing to your server? Maybe setting those default shares to cache:only isn't a good idea because they won't overflow if you are still writing data to them.

 

Disable Docker and VM Manager in Settings.

 

What do you get from command line with this?

du -h -d 1 /mnt/cache

 

[Macj72x]

56 minutes ago, trurl said:

 

What do you get from command line with this?

du -h -d 1 /mnt/cache

 

27G     /mnt/cache/appdata
49G     /mnt/cache/domains
71G     /mnt/cache/system
0       /mnt/cache/temp
3.3T    /mnt/cache/data
50G     /mnt/cache/backups
3.5T    /mnt/cache

 

Mover seems to be moving now that both docker and vm is off, but it wasn't earlier in the day when just docker was off.  Like I said earlier, the only vm I'm running right now is Homeassistant which shouldn't be touching my media share.

[Macj72x]

Scratch that. It moved about 100gb then stopped. It says mover is still running but I haven’t seen a change in about 15 minutes and the data rate dropped significantly. 

Edited February 20, 2023 by Macj72x

[Macj72x]

It finally kicked into gear and has almost cleared out all of the media files. It seemed like it started working after about fours hours after that last post. 

Edited February 20, 2023 by Macj72x

[itimpi]

If mover has a lot of very small files to move then they tend to take MUCH longer than the same total size in larger files.

[trurl]

Now set your default shares back to cache:prefer and run mover again to get those moved to cache.

 

Mover won't replace files, so if you have identical files on cache and the array they won't be moved. You can examine these files and clean up the duplicates in Dynamix File Manager.

[Macj72x]

Will do.  Any recommendations going forward? Should I not use cache for media downloads? 

[trurl]

3 minutes ago, Macj72x said:

Any recommendations going forward?

I don't even bother with mover tuning. I have enough cache (only 500G, less than you have) to hold all I write to it in one day, and the cached user shares get moved to the array in the middle of the night.

 

I have a separate pool "fast" (nvme), 256G, where I keep my default shares.

 

Depends on how much you write to cache.

 

When I had simpler needs and simpler hardware, I didn't bother with caching much because most writes were queued downloads or scheduled backups, and nobody was waiting on them to complete anyway.

 

Now, I have downloads that need post-processing, and I have enough hardware that I can download to a cached share, post-process to my "fast" pool, where it gets moved ('arr) to the final destination cached share.

 

Anything you keep permanently on a single disk pool needs backup. CA Backup plugin, for example. In fact, I would say appdata and vdisks need backup even if on a redundant pool.

[Macj72x]

35 minutes ago, trurl said:

Now, I have downloads that need post-processing, and I have enough hardware that I can download to a cached share, post-process to my "fast" pool, where it gets moved ('arr) to the final destination cached share.

Could you break this down a little more? You use two cache pools for your media downloads or just one of the two? What is the path that your media downloads take because that obviously is my issue. Sounds like a great recommendation to break up my cache pool into two.

[trurl]

Typical 'arr dockers finds things to download, passes them to NZBGet, which downloads into a cache:only share on my fast pool, postprocesses to a cache:yes share on my cache pool, 'arr moves the result to destination share based on media type, cache:yes on my cache pool.

[trurl]

Sorry, got that mixed up a bit above when I first posted. I have edited the post.

[Macj72x]

Gotcha.  I like that.  How do you move to a new share for the post processing, a script?  If so, do you mind DMing it to me?

[Macj72x]

Since I'm moving files from cache to array, I also need to remove "use hardlinks instead of copy" option in the 'arr settings, correct?

[trurl]

57 minutes ago, Macj72x said:

How do you move to a new share for the post processing

That is all in NZBGet. Different mapping for InterDir where things get downloaded.

 

I've never used hardlinks, if I understand correctly that is more useful for seeding torrents.