DZMM Posted September 8, 2020 Author Share Posted September 8, 2020 (edited) Can I get some help testing please. V1.5.3 of rclone (remember you have to remove and reinstall the plugin to update it) now supports better caching where files can be cached locally. I'll add a variable in for setting the cache location once it's all working, but for now can a few people try these settings in the mount script: # create rclone mount rclone mount \ --allow-other \ --dir-cache-time 720h \ --log-level INFO \ --poll-interval 15s \ --cache-dir=/mnt/user/downloads/rclone/tdrive_vfs/cache \ --vfs-cache-mode full \ --vfs-cache-max-size 500G \ --vfs-cache-max-age 336h \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & set the cache-dir to wherever is convenient. The settings above will keep up to 500GB of files downloaded from gdrive for up to 2 weeks, with the oldest removed first when full. I think this will work well with my kids who keep stopping and starting the same file, or when plex is indexing or doing other operations. However, I don't think it will help majorly with playback for my setup, unless a user tries to open the same file within a few hours. Dunno. There's another new setting --vfs-read-ahead that could potentially help with forward skipping/smoother playback by downloading more data ahead of the current stream position, that we can play with as well. Edit: poll-interval shortens the default 1m, so should hopefully add a bit more butter to updates. Edit 2:. Initial launch times are much faster even before the cache kicks in!! Edited September 8, 2020 by DZMM 1 Quote Link to comment
markrudling Posted September 9, 2020 Share Posted September 9, 2020 18 hours ago, DZMM said: Can I get some help testing please. V1.5.3 of rclone (remember you have to remove and reinstall the plugin to update it) now supports better caching where files can be cached locally. I'll add a variable in for setting the cache location once it's all working, but for now can a few people try these settings in the mount script: # create rclone mount rclone mount \ --allow-other \ --dir-cache-time 720h \ --log-level INFO \ --poll-interval 15s \ --cache-dir=/mnt/user/downloads/rclone/tdrive_vfs/cache \ --vfs-cache-mode full \ --vfs-cache-max-size 500G \ --vfs-cache-max-age 336h \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & set the cache-dir to wherever is convenient. The settings above will keep up to 500GB of files downloaded from gdrive for up to 2 weeks, with the oldest removed first when full. I think this will work well with my kids who keep stopping and starting the same file, or when plex is indexing or doing other operations. However, I don't think it will help majorly with playback for my setup, unless a user tries to open the same file within a few hours. Dunno. There's another new setting --vfs-read-ahead that could potentially help with forward skipping/smoother playback by downloading more data ahead of the current stream position, that we can play with as well. Edit: poll-interval shortens the default 1m, so should hopefully add a bit more butter to updates. Edit 2:. Initial launch times are much faster even before the cache kicks in!! I am testing this now and it seems to work really well. I also have a few users with kids that repeat the same shows all the time. Does it do full read-ahead of the file to cache or only cache the contents that it downloads as per request? IE, if a user watches the first 5 min of a move, will only that 5 min be cached or will the internet connection go wild and pull the full file? Hopefully the former. I had wanted to implemented a smart read-ahead cache process that would monitor the last epp watched and pre-fetch the next epp and store it in local, but now this makes managing that file thereafter so much easier. I can pre"read" the next file and it will then be stored in the cache and will be read from there as required, or removed if not consumed in x days. Anything in particular you want me to test, seems to be working out of the box. 1 Quote Link to comment
Spladge Posted September 9, 2020 Share Posted September 9, 2020 I have been testing the VFS settings fairly extensively for the past few months on the betas. Not with unraid but just my rclone mount scripts on straight ubuntu - what caching will really help with is jumping around - fwds and backwards in a file and as you mentioned with stuff like your kids watching the same file over and over - if you were sharing with a few people then the benefits would be stuff like the game of thrones linux iso final where everyone tunes in to watch on release day. Essentially you are keeping the file local. I keep jumping between full and writes. If there is anything you want tested maybe I can help. 1 Quote Link to comment
DZMM Posted September 9, 2020 Author Share Posted September 9, 2020 20 minutes ago, Spladge said: If there is anything you want tested maybe I can help. Thanks. Can you help me play around with --buffer-size and --vfs-read-ahead please. From the rclone docs: Quote When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk. When using this mode it is recommended that --buffer-size is not set too big and --vfs-read-ahead is set large if required. Previously I had --buffer-size at 256MB, but based on the above I've gone with the smaller default of 16MB - I'm not sure what --vfs-read-ahead defaults to. Whatever the defaults are, my first play times seem about 50% faster (1-2 seconds rather than 2-3 seconds) with the settings above - I want to see if setting a higher --vfs-read-ahead makes skipping forward even smoother. I'm a bit worried though that setting it too high will reduce the hit-rate on the cache if it's filled up with content that isn't used. 32 minutes ago, markrudling said: Does it do full read-ahead of the file to cache or only cache the contents that it downloads as per request? IE, if a user watches the first 5 min of a move, will only that 5 min be cached or will the internet connection go wild and pull the full file? Hopefully the former. It only caches 16MB to the memory buffer (--buffer-size) and to disk whatever the --vfs-read-ahead is set to - as above I don't know what the default is. so it's the former. I'm curious to see if setting --vfs-read-ahead to say 128MB is enough to buffer the next xx seconds so skipping is a bit smoother. 34 minutes ago, markrudling said: I had wanted to implemented a smart read-ahead cache process that would monitor the last epp watched and pre-fetch the next epp and store it in local, but now this makes managing that file thereafter so much easier. I can pre"read" the next file and it will then be stored in the cache and will be read from there as required, or removed if not consumed in x days. I think this might leave you with a lot of unnecessary transfer as you only need the first say 5-10 seconds of each file to get virtually instant playback. I think using --vfs-read-ahead will give you a better result. Looking at my caches, my adult TV cache has already hit 500GB in just over 12 hours - although this might be because I'm building a new test Plex server which is currently scanning that folder. root@Highlander:/mnt/cache/downloads/rclone# du -h --max-depth=1 | sort -hr 756G . 499G ./tdrive_t_adults_vfs 219G ./tdrive_vfs 39G ./tdrive_uhd_vfs 0 ./registry_vfs 0 ./gdrive_media_vfs I've bumped up tdrive_t_adults_vfs to 1TB anyway to see what happens and I'm going to break my kids tv out of tdrive_vfs into tdrive_t_kids_vfs, so I can see how many of their shows I can cache e.g. my son tends to watch the same bits of Adventures of Gumball over and over again. Quote Link to comment
Spladge Posted September 10, 2020 Share Posted September 10, 2020 `--vfs-read-ahead` defaults to zero, no read ahead so if you have not specified a value it will still be 0 I think, or false or off. I am not sure, that is the one thing I have not messed with. As far as improving seek I think whatever you have in `vfs-read-ahead` will be the maximum it will help by, if you want to FFWD to a point 600mb ahead of where you are and the read ahead value is 400mb then I think it will make no difference, but it might also make a difference of 400mb depending on other settings. I think the best thing for me to do is create two different teamdrive mounts with different settings and compare that way. I will try to set up a small VPS, biggest problem though may be that my home internet connection is not that fast and connections from most remote servers are poor to Australia. Quote Link to comment
DZMM Posted September 10, 2020 Author Share Posted September 10, 2020 @Spladge my ff seek experience with the new settings is so good, I can't really see what improvement vfs-read--ahead could make. It feels like local playback as the restart times are so fast - in normal usage I wouldn't be able to tell the difference as I wouldn't be monitoring so closely. I think I'm just going to leave it all alone so as not to rock the boat. Quote Link to comment
markrudling Posted September 10, 2020 Share Posted September 10, 2020 (edited) I think the value --vfs-read-ahead will vary largely based on your use case as well as hardware and internet connection. In my testing, it does not add any extra load time to the start time of videos. Im in SA and it takes between 2-5 seconds to start most videos. Faster than spinning up a sleeping local drive. I have a 1 gig internet connection, but single thread downloads from google drive are limited to about 150-220 megabits. So even if you set the read ahead value very high, it will still take a few min to complete the file and have it in cache. Most users will skip ahead during the first few moments of loading a new video. How many times will you skip when 15 min into your content? If download speeds from google maxed out my line, I would be inclined to pre-read 10-15gig and have 99% of my content pre downloaded. At full speed it would not take long and the forward skipping buffer issue is mitigated. I will leave it my current setting (1gig) because the cache is going to manage its self fairly well, and I have fast enough internet to support many streams at the same time. HDD speed will become an issue if there are a lot of reads and writes to the single cache disk though, so perhaps turning it off completely is best(again, depends on your connection). I think the cache as it is though, is fantastic for the more likely use case. Most users will skip back a few seconds, when they miss something. Cache here is fantastic and the backward skips are instant. Im very happy with the setup at the moment! Edited September 10, 2020 by markrudling Quote Link to comment
DZMM Posted September 10, 2020 Author Share Posted September 10, 2020 1 hour ago, markrudling said: I will leave it my current setting (1gig) because the cache is going to manage its self fairly well, and I have fast enough internet to support many streams at the same time. HDD speed will become an issue if there are a lot of reads and writes to the single cache disk though, so perhaps turning it off completely is best(again, depends on your connection). I think I'm going to try raising my --buffer-size to 256MB or 512MB, rather than using --vfs-read-ahead as I'm nervous about multiple disk writes wiping out any potential benefit. Quote Link to comment
markrudling Posted September 10, 2020 Share Posted September 10, 2020 22 minutes ago, DZMM said: I think I'm going to try raising my --buffer-size to 256MB or 512MB, rather than using --vfs-read-ahead as I'm nervous about multiple disk writes wiping out any potential benefit. --Buffer-size will be how much of the file is stored in ram? The cache will still write whatever was buffered from memory onto disk? Agree though with read ahead removing items from cache prematurely, unless you have the space for large cache. I think where Im landing is that cache is great for rewinds. Bonus for content that is watched more than once. Buffer will help a bit with forward skips, but only a few seconds. 2-5 seconds to wait for a long skip is more than acceptable. Quote Link to comment
DZMM Posted September 10, 2020 Author Share Posted September 10, 2020 (edited) 2 hours ago, markrudling said: The cache will still write whatever was buffered from memory onto disk? That's the bit I'm not sure about - if it's kept once playback stops. Update: Quote The --buffer-size data gets written to the disk when the stream is closed so it isn't wasted. --vfs-read-ahead data is written straight to the disk. Edited September 10, 2020 by DZMM Quote Link to comment
markrudling Posted September 14, 2020 Share Posted September 14, 2020 I have been running with no buffer and no read-ahead. Playback is still pretty much instant on all files, including 4k. Skipping backward is a dream, skipping forward is, well, good enough. The read-ahead when using plex is kinda mitigated if its set fairly small, as plex does its own magic between trans-coding and buffering etc. Im quite happy like this. Less disk read/writes, stable, fast. Me like Quote Link to comment
DZMM Posted September 14, 2020 Author Share Posted September 14, 2020 21 minutes ago, markrudling said: I have been running with no buffer and no read-ahead. Playback is still pretty much instant on all files, including 4k. Skipping backward is a dream, skipping forward is, well, good enough. The read-ahead when using plex is kinda mitigated if its set fairly small, as plex does its own magic between trans-coding and buffering etc. Im quite happy like this. Less disk read/writes, stable, fast. Me like I've come to the same conclusion i.e. the defaults are fine and tinkering is only needed if experiencing problems. I'll try and update the scripts this week - I tried last week, but I had a few problems including a pull from @watchmeexplode5 Quote Link to comment
KeyBoardDabbler Posted September 15, 2020 Share Posted September 15, 2020 (edited) Hi all, My unraid logs are being filled with the below error, every second. The one file i can recall changing last was the rclone config credential. All files in my mergerfs mount are viewable but error when trying to open. Quote Sep 15 22:37:40 HTPC-unRAID emhttpd: error: share_luks_status, 6256: Operation not supported (95): getxattr: /mnt/user/mount_mergerfs Edited September 16, 2020 by KeyBoardDabbler Quote Link to comment
francrouge Posted September 16, 2020 Share Posted September 16, 2020 Hi guys, Anybody knows how to delete file on an crypt google drive. I tried with krusader but i'm seem to delete the file but i can't see it in the trash section of the google drive account. thx all Quote Link to comment
JonathanM Posted September 16, 2020 Share Posted September 16, 2020 8 minutes ago, francrouge said: delete the file but i can't see it in the trash section of the google drive account. Are you sure you have the correct drive selected on the left side of the google page? Mine has separate entries on the left for the main drive and shared drives. Quote Link to comment
francrouge Posted September 16, 2020 Share Posted September 16, 2020 2 hours ago, jonathanm said: Are you sure you have the correct drive selected on the left side of the google page? Mine has separate entries on the left for the main drive and shared drives. Yes but my folders are crypted so i don't know what file i want to delete that why i was wondering if krusder works. because i'm able to delete a file in krusader but i'm not sure if it is really deleted on the drive account thx Quote Link to comment
KeyBoardDabbler Posted September 17, 2020 Share Posted September 17, 2020 (edited) 11 hours ago, francrouge said: Yes but my folders are crypted so i don't know what file i want to delete that why i was wondering if krusder works. because i'm able to delete a file in krusader but i'm not sure if it is really deleted on the drive account thx if i delete a file from my mount_mergerfs folder, i see the change in my google drive within afew seconds. Check the folder activity on the right side. Also you can "rename" a file locally, then see which encrypted name changes in your drive. Then rename it back or delete knowing you have identified the correct file/folder Edited September 17, 2020 by KeyBoardDabbler 1 Quote Link to comment
francrouge Posted September 17, 2020 Share Posted September 17, 2020 6 hours ago, KeyBoardDabbler said: if i delete a file from my mount_mergerfs folder, i see the change in my google drive within afew seconds. Check the folder activity on the right side. Also you can "rename" a file locally, then see which encrypted name changes in your drive. Then rename it back or delete knowing you have identified the correct file/folder thx for the info Quote Link to comment
MowMdown Posted September 20, 2020 Share Posted September 20, 2020 (edited) Is anybody using the built in rclone union for their GDrive and Local mounts with vfs caching? I seem to be struggling with it in a strange way. I have three rclone remotes: 1. Gsuite: 2. Crypt: (this wraps Gsuite:) 3. Union: (this wraps crypt:media and /mnt/disks/media/) (this is also what I am caching w/ vfs shown below) My issue seems to be when I go to issue rclone move /mnt/user/media/movies crypt:media/movies --delete-src-dirs that it moves successfully but then Plex cannot play the file. It can see it just fine but I get an input/output error unless I issue a vfs/refresh rclone rc command. Should I be caching crypt before unionizing it? rclone mount \ --allow-other \ --dir-cache-time 720h \ --poll-interval 15s \ --buffer-size 256M \ --cache-dir=/mnt/disk3/system/rclone/cache \ --vfs-cache-mode writes \ --vfs-cache-max-size 100G \ --vfs-cache-max-age 168h \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --rc \ --rc-addr 192.168.1.200:5572 \ --syslog \ union: /mnt/disks/media & Edited September 20, 2020 by MowMdown Quote Link to comment
DZMM Posted September 20, 2020 Author Share Posted September 20, 2020 4 hours ago, MowMdown said: Is anybody using the built in rclone union for their GDrive and Local mounts with vfs caching? I seem to be struggling with it in a strange way. Nope - I was having different problems (I can't remember what right now) so I'm sticking with mergerfs. Quote Link to comment
lzrdking71 Posted September 20, 2020 Share Posted September 20, 2020 (edited) @DZMM I am having the same issue I think @HonkyKONG22 was having. /local/gdrive_media_vfs /gdrive_media_vfs /tv /mount_mergerfs/gdrive_media_vfs /gdrive_media_vfs /tv /mount_rclone/gdrive_media_vfs /gdrive_media_vfs /tv Any help would be appreciated. I can't figure out why I have the extra gdrive_media_vfs folder. log.txt rclone_config.txt mount script.txt Edited September 20, 2020 by lzrdking71 Quote Link to comment
DZMM Posted September 20, 2020 Author Share Posted September 20, 2020 7 hours ago, lzrdking71 said: @DZMM I am having the same issue I think @HonkyKONG22 was having. /local/gdrive_media_vfs /gdrive_media_vfs /tv /mount_mergerfs/gdrive_media_vfs /gdrive_media_vfs /tv /mount_rclone/gdrive_media_vfs /gdrive_media_vfs /tv Any help would be appreciated. I can't figure out why I have the extra gdrive_media_vfs folder. log.txt 2.54 kB · 0 downloads rclone_config.txt 378 B · 0 downloads mount script.txt 10.27 kB · 0 downloads I think the script doesn't like it it when you have only one folder specified in: MountFolders=\{"tv"\} Try adding a second folder e.g movies MountFolders=\{"movies,tv"\} Quote Link to comment
lzrdking71 Posted September 20, 2020 Share Posted September 20, 2020 (edited) 1 hour ago, DZMM said: I think the script doesn't like it it when you have only one folder specified in: MountFolders=\{"tv"\} Try adding a second folder e.g movies MountFolders=\{"movies,tv"\} I went through and did: fusermount -uz /mnt/user/mount_rclone/gdrive_media_vfs fusermount -uz /mnt/user/mount_mergerfs/gdrive_media_vfs looked at the folders and removed the remaining gdrive_media_vfs folder from /local, /mount_mergerfs, and /mount_rclone made the script modification you suggested and added MountFolders=\{"movies,tv"} re- ran the script and now I again have the extra /gdrive_media_vfs in all of the folders I listed I removed it from above + the added movies folder Edited September 20, 2020 by lzrdking71 Quote Link to comment
DZMM Posted September 20, 2020 Author Share Posted September 20, 2020 The extra folder could be because after being created the upload script added them to gdrive. Delete and they shouldn't come back if my theory is correct. Quote Link to comment
lzrdking71 Posted September 20, 2020 Share Posted September 20, 2020 (edited) 58 minutes ago, DZMM said: The extra folder could be because after being created the upload script added them to gdrive. Delete and they shouldn't come back if my theory is correct. I have not run the upload script? I deleted the extra gdrive_media_vfs folder from the /local location and it disappeared from the other two locations and did not re-appear after running the mount script again. Is this how it should be? It doesn't look like /mount_rclone/gdrive_media_vfs has the /tv or /movie folders now? Edited September 20, 2020 by lzrdking71 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.