Hey all,

 

has anyone ever tried to modify the mover script so that specific files are always kept on cache?

 

I'm using Resilio Sync and it rescans all folders every 600s and accesses the .sync/ID File. I managed to reduce drive access by installing cache dir plugin, but as the ID File is actually opened the plugin cant help here. 

 

So my question now is: Is it as simple as editing the /usr/local/sbin/mover script?

 

It looks like the "move" tool is some kind of an rsync wrapper and the last parameter looks like rsync options:

/usr/local/sbin/move /mnt/cache /mnt/user0 '-i -dlDIWR --inplace -pAXogt --numeric-ids'

So can i simply add an "--exclude-from=myfile.txt"  here? 

Thanks in advance

me.so.bad

Edited April 9, 20179 yr by me.so.bad

There is a much better, simpler, and more standard way to do this. You just have to make proper use of the built-in functionality of User Shares - Use Cache Disk setting. I have several shares that stay on cache, and I'm sure most others do too since this is the normal way to setup applications.

 

 

i know that - but i cant store >1TB media files always on cache – also i cant move the .sync folder so i need to store /mnt/user/music/.sync on cache and /mnt/user/music/* on array.

The best solution is to see if you can't have the .sync folder stored outside of the music share (ie: in its own cache only share).  But, I believe that an issue has already been identified in the mover script where it is changing some attributes on some of the files which may ultimately be causing your issue.  LT I believe has stated that this will be fixed, but due to the changes actually required it won't happen until the 6.4 series...

 

Beyond that, nothing is stopping you from actually modifying the system files / scripts and overwriting them at each boot of the server, but its not really recommended. 

.sync outside of share: Afaik not possible

 

mover bug: do you have a link for reference? But i don't think thats the issue here - currenty music doesnt use cache at all

 

editing files: thats what i want to do, but as I cant see inside the move binanry I was hoping someone can confirm my assumption.

Maybe it doesn't apply in your scenario, but it is possible to keep files from a cache-no share on cache. See the description of Cache:No in the FAQ I linked.

 

mover is a script which you can find with

which mover

 

Any change you make will not survive a reboot since it is part of the OS archive unpacked from flash on each boot. You can reapply changes upon boot in config/go script or with User Scripts plugin.

17 minutes ago, me.so.bad said:

mover bug: do you have a link for reference?

 

Not a bug per se, but rather just how things work...

 

 

4 hours ago, me.so.bad said:

It looks like the "move" tool is some kind of an rsync wrapper and the last parameter looks like rsync options:

/usr/local/sbin/move /mnt/cache /mnt/user0 '-i -dlDIWR --inplace -pAXogt --numeric-ids'

So can i simply add an "--exclude-from=myfile.txt"  here? 

 

root@Server_A:/# move --help
usage: move [-d] src-dir tar-dir rsync-options

So yes, if there is no other work around, then you can change the mover script and add in whatever rsync options you require.

All you need to do is mark the user share as cache only, then mover won't get involved. You would then use your own script (or do it manually) to move whatever you want from cache to the array. The user share will consolidate files from both locations regardless of the cache only setting.

Sent from my ONEPLUS A3000 using Tapatalk

Hm, thats maybe a better idea then editing unraid core… thanks!

16 minutes ago, ljm42 said:

All you need to do is mark the user share as cache only, then mover won't get involved. You would then use your own script (or do it manually) to move whatever you want from cache to the array. The user share will consolidate files from both locations regardless of the cache only setting.

Sent from my ONEPLUS A3000 using Tapatalk
 

This is exactly what is described in the FAQ I linked.

 

8 minutes ago, me.so.bad said:

Hm, thats maybe a better idea then editing unraid core… thanks!

 

Sorry, the new forum doesn't show links in Tapatalk, so I couldn't see what you referenced. I just checked from a browser, great FAQ!

Based on that, either cache no or cache only could work, depending on whether you want new files to go to the array or to cache. But the most important thing is that unraid supports this out of the box, no need to modify core files.

Sent from my ONEPLUS A3000 using Tapatalk

I'm running Exactly what your running and I was curious to know why my Disk 8 was on now and then when there really isn't any activity. 

@me.so.bad I'd like to see what your solution is in the end or maybe I'll start tinkering with my own. 

5 hours ago, trurl said:

This is exactly what is described in the FAQ I linked.

havent thought about that possibility - ljm42 pointed it out very nicely.

 

@kizer TBD

 

 

edit: have to take a look at the script tomorrow… 

Edited April 9, 20179 yr by me.so.bad

After some testing and fiddling I've decided to edit the /usr/local/sbin/mover. Everything else ends in moving files too much.

 

I will not post the script here because I cant test it very well (only on 6.3.3) and it can cause too much damage (mover is more or less the most important script of unRAID), but i can tell you I'm simply sed'ing it to replace line 86 from: 

find "./${SHARE%/}" -depth -print \

to

find "./${SHARE%/}" -depth -print | grep -vFf /boot/extra/enhancedmover/.exclude.list \

 

Anyway, if anyone wants to test it - just PN me.

maybe some day I'll create a plugin…

 

Edited April 10, 20179 yr by me.so.bad

Just a warning - that may end up only being a short term solution.   Limetech have said that the mover functionality will be significantly improved for the 6.4 release.    That may mean it is harder to apply your change (and we do not even know if mover is going to remain a script that can be edited).

Yup - Squid mentioned it before. Thats why I dont publish anything at the moment. We'll see what 6.4 brings.

manipulating mover like I did above is a bad idea, move tries to delete the containing folders which leads to errors in logfile:

 

Quote

Apr 17 00:00:16 Datenteich move: rmdir: /mnt/cache/./music/.sync Directory not empty
Apr 17 00:00:16 Datenteich move: rmdir: /mnt/cache/./music Directory not empty

 

So I'll write a script which simply moves the wanted files back to the cache after mover is done… just a quirky workaround but at least as good as having errors in log…

 

lets see what 6.4 brings…

 

 

edit: 

#!/bin/bash

# set to reasonable depth to speed up find cmd…
maxdepth=$1
if [[ -z "$maxdepth" ]]; then
	maxdepth=4
fi

# Is unRAID mover running?
while [ -f "/var/run/mover.pid" ]; do
	echo "mover is running… waiting 5min"
	sleep 5m
done

echo "moving .sync/ID files back to cache"
#now find.
find /mnt/user0/ -name ".sync" -type d -maxdepth $maxdepth -print \
| while read f; do

	user0="$f/ID"
	cache="/mnt/cache/$(echo $user0 | sed 's/\/mnt\/user0\///')"
	SHARE=$(echo $f | cut -d/ -f1)

	# skip if ID file doesnt exist
	[ -f "$user0" ] || continue

	#see /user/local/sbin/mover
	if [[ ! -d "/mnt/cache/$SHARE" ]] && grep -qs 'shareCOW="no"' "/boot/config/shares/${SHARE%/}.cfg" ; then
		rsync -i -d -pAXogt --numeric-ids --exclude="*" "$SHARE" "/mnt/cache/$SHARE"
		chattr +C "/mnt/cache/$SHARE" 2>/dev/null
	fi

	#now move
	mkdir -p "$(dirname "$cache")"
	mv "$user0" "$cache"
	echo "moved $cache"

done

echo "done."

 

Edited April 16, 20179 yr by me.so.bad