Mover not moving

[rdagitz]

I removed the mover tuning plugin and have run mover 5 times. I have a 500GB ssd for cache. I initially had over 300GB on the cache drive. Mover will not bring the drive to less than 262GB. It moves some of the files to the array but not that part. What is going on?

 

Here is another diagnostic download.

server-diagnostics-20211107-2000.zip

[trurl]

Looks like it isn't moving appdata that is on the array. Those would be moved to cache if they weren't open.

And it isn't moving movie torrents, perhaps they are open seeding.

 

Nothing can move open files.

[rdagitz]

Thank you for help. I really mean that.  I am new to Uraid and I want to know how it works. Where did you look in the diagnostics to see that? I want to be able to self-diagnose these things in the future, if needed. You know, "Teach me how to fish and you'll feed me for a lifetime." 😁

[Frank1940]

2 hours ago, rdagitz said:

I want to be able to self-diagnose these things in the future, if needed. You know, "Teach me how to fish and you'll feed me for a lifetime." 😁

 

You might want to take a deep breath at this point.  Unraid uses a standard Linux OS at its core. First point.  Now realize that both Linux and Windows have certain rules about how files are handled.  Second point. Both of these OS have multi user rules in place to prevent data loss.  Let's look at a few of these basic rules:

• Any file that is opened by another user or process can not be moved.
• Any file that is opened by one user or process with read/write permission can not be modified by another user or process.  (It may be opened in a read-only mode.)
• No file may be replaced by another file without without the requesting  user giving specific permission to do so. 

I think you can see that all of these rules are logical and they are necessary to prevent unintended data loss.  Now, I do believe that the Mover 'program' is actually a script written in BASH.   (BASH is the program that you are actually using when you are using the Command Line in Unraid.  This type of program is called a 'Shell' and BASH is an acronym for Bourne Again Shell.   BASH contains a complete programing language that is often used to perform system level functions without having to write a binary.  (An aside, many of the Linux utilities are actually scripts and not binaries.)  Now, you may ask, "What is a script"?  A script is simply a text file of BASH programing commands to perform a given task.  I have attached a Shell script for you to look at.  (It is the original Preclear script written by @Joe L..  I changed the file extension from .sh to .txt so it is easier to open for your inspection.)  

 

Let's think a bit about Mover.  It is a Shell script that is intended to run in the background without any user interaction.  And it must follow those basic rules listed above to prevent a possible data loss.  With these conditions, what does Mover do if it encounters a file which violates one of those rules?  It simply leaves the file on the cache drive.  Mover has been around for a very long time and thousands of Unraid users use it.  All of the bugs are gone.  So when someone complains that Mover has moved a file off the cache drive, it is because moving it would have violated one of those rules. 

 

So you see that it not great wisdom in analyzing the syslog.  It is simply that there is no entry in the syslog indicating a Mover error.  (If you go to    Settings     >>>  Scheduler  and look at 'Mover Settings', you will see that you can turn on "Mover logging".  It used to be 'on' by default but doing so really floods the syslog with entries...)

 

Hope this helps a bit...

preclear_disk.txt

[rdagitz]

That, my friend, is very well said. I think I follow you pretty well. So there are no mover errors. It is just that I have a "conflict" with the mover script and some process(es) that have the files locked down, thus the files cannot be moved. In light of this line of thought, I shut down a torrent I had seeding and changed a "TV" share that I have (which contains recorded TV) from cache "yes" to cache "no." These are the only processes of which I could think that would be locking down files.

 

On another note, would the mover logs clarify for me which files a locked down?  I have logging enabled on mover. Where are those log files stored?

[Frank1940]

AS I remember they were in the syslog.  That was the reason that the logging of Mover was discontinued as every single file was logged as a separate line.  (As I remember it...)  I know that the backup of my Documents folder has almost 4300 files in it and I do a full backup once a month!  (Plus another 3000 Files from my Wife's computer on the same day.)   And since most people were not having issues with Mover, it was just noise.   (Imagine trying to find some error message about another problem with all that noise...)

[itimpi]

5 hours ago, rdagitz said:

changed a "TV" share that I have (which contains recorded TV) from cache "yes" to cache "no

Mover ignores any shares with Use Cache set to No, so if there are any files for that share on the cache they remain there (although new files will get written to the array).   You need Use Cache=Yes if you want mover to try and move files from cache to array.  You also want the Mimimum Free Space for the cache pool to be set larger than the biggest file you expect to copy as once the free space drops below that value UnRaid will start writing any new files directly to the array.

[rdagitz]

So I went to the Unraid main dashboard and clicked on the "View" icon for my cache drive. Keep in mind, this is the cache drive that is keeping between 300 and 400 GB on it regularly. After some digging I realized that it is the "transcoding-temp" file for Emby server that is taking up all of the space and living on cache drive. Now I just need to do a little extra research to see if I can move it to one of the shares instead. Posting what I found here so that others who may run into this issue will find out.

Edited November 14, 2021 by rdagitz
Updating information

[Frank1940]

@rdagitz, first go to the 'Shares' Tab and look at those shares you mentioned.  Realize that shares that are "Only: cache" or "Prefer: cache" will never be moved by Mover! 

 

To determine the size of each share, click on the "compute..." for that share.  (Realize that it can take a long time to do this for larger shares or shares with thousands of files!) If you have files in the root of the cache drive, double-click on the 'Icon' under the "View" column for that drive to see their size. 

Edited November 14, 2021 by Frank1940