[Guide] How to keep cache drive full of media

[flyize]

Actually, not sure what I've done, but the mover now moves everything off.

[Terebi]

2 hours ago, flyize said:

Actually, not sure what I've done, but the mover now moves everything off.

 

Are you sure you have the second setting set to ignore things named in the file?

[flyize]

Yeah, I knew I had forgotten something.

[flyize]

Just wanted to come back and say thank you for this. It's working perfectly!

[Samsonight]

Have we considered using (60s just a sample timer)

inotifywatch -v -e access -e modify -t 60 -r /mnt/user0

which basically collects statistics of all non-cached (/mnt/user0) files and folders.

 

And then using a script to move the most accessed files to the (unraid) cache (/mnt/user).

 

This would effectively create a "read cache" which could be iterated over and over, for X amount of time, untill the cache is X % full.

 

Please note this records literal disk reads and memory cached content and application cached content is not included.

 

Random example excerpt from the depths of the interwebz...

% inotifywatch -v -e access -e modify -t 60 -r ~/.beagle
Establishing watches...
Setting up watch(es) on /home/rohan/.beagle
OK, /home/rohan/.beagle is now being watched.
Total of 302 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 60 seconds.
total  access  modify  filename
1436   1074    362     /home/rohan/.beagle/Indexes/FileSystemIndex/PrimaryIndex/
1323   1053    270     /home/rohan/.beagle/Indexes/FileSystemIndex/SecondaryIndex/
303    116     187     /home/rohan/.beagle/Indexes/KMailIndex/PrimaryIndex/
261    74      187     /home/rohan/.beagle/TextCache/
206    0       206     /home/rohan/.beagle/Log/
42     0       42      /home/rohan/.beagle/Indexes/FileSystemIndex/Locks/
18     6       12      /home/rohan/.beagle/Indexes/FileSystemIndex/
12     0       12      /home/rohan/.beagle/Indexes/KMailIndex/Locks/
3      0       3       /home/rohan/.beagle/TextCache/54/
3      0       3       /home/rohan/.beagle/TextCache/bc/
3      0       3       /home/rohan/.beagle/TextCache/20/
3      0       3       /home/rohan/.beagle/TextCache/62/
2      2       0       /home/rohan/.beagle/Indexes/KMailIndex/SecondaryIndex/

 

[trurl]

2 hours ago, Samsonight said:

all non-cached (/mnt/user0) files and folders.

 

And then using a script to move the most accessed files to the (unraid) cache (/mnt/user).

/mnt/user is the user shares including all pools. You would have to specify /mnt/poolname, such as /mnt/cache, for the move destination.

 

You definitely don't want to try to move anything from /mnt/user0 to /mnt/user. That would almost certainly result in lost data.

[Samsonight]

1 hour ago, trurl said:

/mnt/user is the user shares including all pools. You would have to specify /mnt/poolname, such as /mnt/cache, for the move destination.

 

You definitely don't want to try to move anything from /mnt/user0 to /mnt/user. That would almost certainly result in lost data.

 

Afaik. If a share has its primary storage on a Pool (cache) and you write to /mnt/user/ShareX it goes to the Pool (instead of the Array).

If you write to the /mnt/user0 it goes to the Array (disk).

 

If /mnt/user0 isn't the (array) disk-only path and /mnt/user is the top layer then what does Mover do when moving to / from array / pool. However it seems Lime is depricating user0, but equally they have not clearly stated before, nor stating now where this functionality would occur. Basically causing threads and confusions like this to fill the forums.

 

Documentation search for user0 https://docs.unraid.net/search/?q=user0

returns only https://docs.unraid.net/unraid-os/manual/shares/#user-shares

while the actual reference is https://docs.unraid.net/unraid-os/manual/shares/user-shares/

 

Please, since you imply factual knowledge of the absolute state of the FUSE layering, could you explain in detail why/how "That would almost certainly result in lost data". Preferrably citing documentation. Just to avoid obscure fud and misunderstandings like this... and how you would suggest a better result achieved. ❤️

Edited May 22 by Samsonight
clarification seems needed

[Terebi]

It certainly might be possible to expand my script to include recently read item into things to preserve on cache, more than just things that were added recently. 

 

For my purposes, recently added and recently viewed overlap a lot though, so it might not be as useful as imagined. 

 

And without more complicated work would not handle promoting things from the array to the cache. 

[Unraidmule]

I deleted the moverignore.txt file, since it was keeping the entries of files I had deleted manually from the cache drive which were no longer present. Instead of recreating the .txt file, it emptied the whole cache drive last night. Is the .txt file always meant to be created before a first run? I don't think the script was able to create it itself.

 

EDIT: I created the moverignore.txt file manually with the command but, seems like it is not being updated any more automatically either via the mover plugin.

Edited May 28 by Unraidmule

[Terebi]

On 5/27/2024 at 2:22 AM, Unraidmule said:

I deleted the moverignore.txt file, since it was keeping the entries of files I had deleted manually from the cache drive which were no longer present. Instead of recreating the .txt file, it emptied the whole cache drive last night. Is the .txt file always meant to be created before a first run? I don't think the script was able to create it itself.

 

EDIT: I created the moverignore.txt file manually with the command but, seems like it is not being updated any more automatically either via the mover plugin.

 

Every time my script runs it should create the file.  Perhaps the script is not in the same location the mover tuner setting is looking for it in? Or the permissions on the script may need to be fixed?

[Unraidmule]

6 hours ago, Terebi said:

 

Every time my script runs it should create the file.  Perhaps the script is not in the same location the mover tuner setting is looking for it in? Or the permissions on the script may need to be fixed?

 

I think it is a permission issue — it lost rights to execute, IDK what the original file permission was I did chmod 777 because I forgot about the +x command. Will let it run tonight and hopefully get the desired results, when it was running it was actually really cool.

[Terebi]

Just now, Unraidmule said:

 

I think it is a permission issue — it lost rights to execute, IDK what the original file permission was I did chmod 777 because I forgot about the +x command. Will let it run tonight and hopefully get the desired results, when it was running it was actually really cool.

 

in a console if you just run mover, you can see if it triggers my script or not

[Unraidmule]

12 hours ago, Terebi said:

 

in a console if you just run mover, you can see if it triggers my script or not

 

I can test it, what do I type in the console? Just “mover”?

 

EDIT: It executed overnight and the .txt file seems to be updating now. IDK what happened, but the executing permissions got removed somehow. If anyone else has issues, that would be the first thing to check.

Edited May 29 by Unraidmule

[moosehunter]

On 5/28/2024 at 5:17 PM, Unraidmule said:

 

I can test it, what do I type in the console? Just “mover”?

 

EDIT: It executed overnight and the .txt file seems to be updating now. IDK what happened, but the executing permissions got removed somehow. If anyone else has issues, that would be the first thing to check.

Are you using user-scripts to make it?  Is your script located on your boot device?

I had same problem, found out that unraid wont let you make anything on the boot drive executable.  I made a copy to cache disk and I believe its working right now.

 

Had another issue with some files not being added to the moverignore.txt file, specifically some files with 3 digit inodes.

If it was currently looking at a file with inode 416, and inode 5416 had already been processed and added to the array, it would count it as a match.  Fixed it by changing the line to:

 

if  [[ "${processed_inodes[*]}" =~ (^|[[:space:]])"$inode"($|[[:space:]]) ]]; then