Mover not Moving


Macj72x
Go to solution Solved by trurl,

Recommended Posts

I apologize in advance if I'm asking for help on a common problem, I couldn't find the right answer in my searches.  I am struggling with getting my cache pool to move correctly. My cache pool keeps filling up and won't empty even when mover is running.  I have mover set to run every hour if the cache pool gets above 50%, but It will just continue to fill and not empty until a random point in time or if the server is restarted.

 

I understand that "Prefer" will keep all the files from that share on the cache until it fills and "Yes" will write to the cache first and then dump to the array once it is full or mover invoked.  I've checked my share sizes and everything with "Prefer" only adds up to a couple hundred GBs.  After checking the size of all my shares going into the cache, it appears that my media files being downloaded go in and don't move out.  My cache pool is four nvme drives at 2 TBs each, so it is pretty large and quick.

 

What am I missing?  I've attached my logs.  While looking through them I did notice this error "emhttpd: error: get_key_info, 567: Invalid argument (22): stat: /boot/config/Trial.key" I'm not sure why I'm seeing that, but I'm assuming it is unrelated.

Edited by Macj72x
Link to comment
26 minutes ago, Macj72x said:

I have mover set to run every hour

Running mover more often won't help if you are still writing to cache. Mover is intended for idle time. It is impossible to move from fast cache to slow array as fast as you can write to cache. If you try to move while still writing it only makes things worse.

 

What is the purpose of these 2 cache:prefer shares?

l------e                          shareUseCache="prefer"  # Share exists on cache
m-u                               shareUseCache="prefer"  # Share exists on cache

 

Link to comment

I'm using usenet to download files, so I shouldn't be seeding anything.

 

The idea to use the cache for downloading media was that I could get media onto the server with little delay and move to the array as needed.  Then the cache could allow for faster post processing.  It appears the download speed is slower than the array speed.  Am I wrong with this approach?

Edited by Macj72x
Link to comment
1 hour ago, Squid said:

What share(s) in particular are you talking about?  

It is my media share that seems to be the culprit.

 

51 minutes ago, trurl said:

What is the purpose of these 2 cache:prefer shares?

l------e                          shareUseCache="prefer"  # Share exists on cache
m-u                               shareUseCache="prefer"  # Share exists on cache

 

 The first is a lan cache that I haven't really finished setting up so it is pretty much empty except some folders.

 

The second is where I've stored some m3u files.

Link to comment

I personally use the Mover Tuning plugin and have it set to move after a week. That way i can access the files from the Cache and the array, but they are moved automatically after a period of time vs daily. Also the upside is I'm sure the files aren't open or being used unless one of my kids is up watching a video at 5AM. 😃

Link to comment
6 minutes ago, itimpi said:

The Prefer setting for Use Cache means you want files moved TO cache by mover.    If you want mover to move files from cache to the array you need the Yes setting.   There is good help built into the GUI for describing how the setting works and how it affects mover.

Yes, I think I understand this properly.  I'm still learning Unraid, so I'm probably doing plenty of things wrong, but I set Prefer to shares that will help the system run faster and Yes to shares that I want data dumped onto the system quickly then eventually moved into the array.

 

I figured I should have my media set to Yes, so I can quickly big data onto my server and be processed in a quicker fashion then moved off onto the array when the cache is near full.  Is this a bad way of looking at this?  Should I just set my downloaded media to go straight to the array?

Link to comment

While I still would like to know if I’m using the cache pool in a way that isn’t recommended, I still think something else is wrong with my mover or the way it is configured.  I’ve turned off docker. I now only have my Homeassistant VM running.  I’ve checked my open files and it doesn’t appear that I have anything with my media files open. 65847BFD-F021-4C00-9FF1-87C5529449CE.thumb.jpeg.71616d2137331649f8a8709064a06dad.jpeg
 

After this, I’ve invoked mover and it has been running for over and hour and has hardly moved anything. I started with about 250gb space remaining and am now at 267gb. Any guesses where to start?BF4F52A7-23AE-412E-9207-F9AC2A7315E2.thumb.jpeg.c129851539240abe55ad5e2b9dcef5da.jpeg

Link to comment

Your "default shares" have some files on the array.

https://wiki.unraid.net/Manual/Shares#Default_Shares

Usually you want these all on fast pool (cache) so Docker/VM performance isn't impacted by parity, and so array disks can spin down since these files are always open.

 

But you don't have space to move them to cache currently. Set them to cache:only so mover will ignore them for now until you make space on cache for them.

 

I'll leave it up to you if you want to keep those other cache:prefer shares on cache. Currently they are all on cache with nothing on the array.

 

Run mover to get your cache:yes shares moved to the array. Then post new diagnostics and we can work on your default shares.

 

You can see how much of each disk is used by each user share by clicking Compute... for the share on the User Shares page, or the Compute All button.

Link to comment

 

 

I have changed the default shares to only.  I have deleted the other two shares listed as prefer as they didn't have any important data on them anyway.

 

I ran the mover and nothing has moved off the cache since you last posted approximately 2 hours ago.  Other than the default shares, all shares are either listed as Yes:Cache or No with nothing on the cache.  Computing all the share size showed that a vast majority of the cache data is from my media download/storage drive (named data).

Edited by Macj72x
Link to comment
Feb 19 18:59:00 ChunkyMonkey CA Backup/Restore: Starting binhex-nzbget... (try #1) 
Feb 19 18:59:00 ChunkyMonkey  shfs: cache disk full

and more. Are you still writing to your server? Maybe setting those default shares to cache:only isn't a good idea because they won't overflow if you are still writing data to them.

 

Disable Docker and VM Manager in Settings.

 

What do you get from command line with this?

du -h -d 1 /mnt/cache

 

Link to comment
56 minutes ago, trurl said:

 

What do you get from command line with this?

du -h -d 1 /mnt/cache

 

27G     /mnt/cache/appdata
49G     /mnt/cache/domains
71G     /mnt/cache/system
0       /mnt/cache/temp
3.3T    /mnt/cache/data
50G     /mnt/cache/backups
3.5T    /mnt/cache

 

Mover seems to be moving now that both docker and vm is off, but it wasn't earlier in the day when just docker was off.  Like I said earlier, the only vm I'm running right now is Homeassistant which shouldn't be touching my media share.

Link to comment
  • Solution
3 minutes ago, Macj72x said:

Any recommendations going forward?

I don't even bother with mover tuning. I have enough cache (only 500G, less than you have) to hold all I write to it in one day, and the cached user shares get moved to the array in the middle of the night.

 

I have a separate pool "fast" (nvme), 256G, where I keep my default shares.

 

Depends on how much you write to cache.

 

When I had simpler needs and simpler hardware, I didn't bother with caching much because most writes were queued downloads or scheduled backups, and nobody was waiting on them to complete anyway.

 

Now, I have downloads that need post-processing, and I have enough hardware that I can download to a cached share, post-process to my "fast" pool, where it gets moved ('arr) to the final destination cached share.

 

Anything you keep permanently on a single disk pool needs backup. CA Backup plugin, for example. In fact, I would say appdata and vdisks need backup even if on a redundant pool.

Link to comment
35 minutes ago, trurl said:

Now, I have downloads that need post-processing, and I have enough hardware that I can download to a cached share, post-process to my "fast" pool, where it gets moved ('arr) to the final destination cached share.

Could you break this down a little more? You use two cache pools for your media downloads or just one of the two? What is the path that your media downloads take because that obviously is my issue. Sounds like a great recommendation to break up my cache pool into two.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.