Jump to content
DZMM

Guide: How To Use Rclone To Mount Cloud Drives And Play Files

1366 posts in this topic Last Reply

Recommended Posts

23 minutes ago, DZMM said:

unRAID is based on slackware and mergerfs doesn't support slackware.  Luckily the author just built a docker version we can use. 

Do we need to install something from apps to do that? Or you just use the direct repo link? So in the future it could be that the repo gets moved/deleted and it won't work?

Share this post


Link to post
20 minutes ago, Kaizac said:

Do we need to install something from apps to do that? Or you just use the direct repo link? So in the future it could be that the repo gets moved/deleted and it won't work?

Everything is self-contained in the script - no need to touch CA, nerd tools etc except to install the rclone plugin.

 

Re the mergerfs docker - I'm not an expert, but it's building it direct from the mergerfs author's repo, so the script will only need changing if he updates his build options which I think will be unlikely:

 

https://github.com/trapexit/mergerfs#build-options

Share this post


Link to post
1 hour ago, DZMM said:

Everything is self-contained in the script - no need to touch CA, nerd tools etc except to install the rclone plugin.

 

Re the mergerfs docker - I'm not an expert, but it's building it direct from the mergerfs author's repo, so the script will only need changing if he updates his build options which I think will be unlikely:

 

https://github.com/trapexit/mergerfs#build-options

Ok understood.

 

In your mount command you have --dir-cache-time 720h. This used to be 72h. Why the change?

And you also started using --fast-list in the mount command. I thought this didn't work in the mount command only when doing transfers for example. Has that changed?

Share this post


Link to post
1 hour ago, Kaizac said:

Ok understood.

 

In your mount command you have --dir-cache-time 720h. This used to be 72h. Why the change?

And you also started using --fast-list in the mount command. I thought this didn't work in the mount command only when doing transfers for example. Has that changed?

--dir-cache-time can be large as you want - uploads flush the cache.  No real reason, just decided to put a larger number in for the (rare) days when no new content added

--fast-list - yep, that shouldn't be there.  I forgot to delete when I removed the rc command.

 

On 11/6/2018 at 12:44 PM, DZMM said:

 

--fast-list: Improves speed but only in tandem with rclone rc --timeout=1h vfs/refresh recursive=true

 

 

Share this post


Link to post
1 minute ago, DZMM said:

--dir-cache-time can be large as you want - uploads flush the cache.  No real reason, just decided to put a larger number in for the (rare) days when no new content added

--fast-list - yep, that shouldn't be there.  I forgot to delete when I removed the rc command.

 

 

You have a double --fast-list in your upload code. Probably not an issue, but might want to remove it.

 

So far I've just migrated everything over and it seems to be working fine! Don't understand the hardlinking much yet, because I don't use torrents much so don't have the seeding issue. Will have to change parts of my folder structure though to get in line with the new standard.

Share this post


Link to post

I'll stick to unionfs for now. It's not broken for me so not seeing any need to migrate to mergefs.

Share this post


Link to post

@DZMM sorry but in your first post you wrote this:

 

Quote

Either finish any existing uploads from rclone_upload before updating, or move pending downloads from /mnt/user/rclone_upload to the new /mnt/user/local folder, or create a version of the new upload script to upload from /mnt/user/local 

I've tried to understand what you're saying here, but I really can't. What exactly is the difference in user/rclone_upload with user/local? They are both local shares which you include in your union/merge. Maybe I'm missing something in your changes, since my configuration was a bit different because of more local folders.

Share this post


Link to post
4 hours ago, Kaizac said:

@DZMM sorry but in your first post you wrote this:

 

I've tried to understand what you're saying here, but I really can't. What exactly is the difference in user/rclone_upload with user/local? They are both local shares which you include in your union/merge. Maybe I'm missing something in your changes, since my configuration was a bit different because of more local folders.

You can ignore that and just ensure all your dockers, including nzbget etc,  use mappings within the mergerfs folder and not the local folders to fully benefit from mergerfs's better file transfer capabilities.   

 

I decided to ditch rclone_upload to make it easier for users to not mess up and to clean flows up. 

 

E.g. before my flow before was:

 

/mnt/user/downloads/movie -->/mnt/user/import/movie -->/mnt/user/mount_unionfs/movies/movie (via /mnt/user/rclone_upload/movies/movie) I.e. 3 different shares.

 

Now I have /mnt/user/mount_mergerfs/downloads/movie --> /mnt/user/mount_mergerfs/complete/movie --> /mnt/user/mount_mergerfs/movies/movie where any local files are in /mnt/user/local and I exclude mnt/user/local/downloads mnt/user/local/complete and mnt/user/local/seeds in the upload script.

 

Makes my life a hell of a lot easier and by including my 'pre-plex' files in the mount I get the full transfer benefits of mergerfs, and I think it will be easier for anyone new to the scripts.

Share this post


Link to post
17 hours ago, testdasi said:

I'll stick to unionfs for now. It's not broken for me so not seeing any need to migrate to mergefs.

If it's not broken, don't fix it !

Same as you, for the moment I fear to touch anything as it is working flawlessly ... Even if it is burning my finger to try mergerfs :D

Share this post


Link to post
Posted (edited)
1 hour ago, yendi said:

If it's not broken, don't fix it !

Same as you, for the moment I fear to touch anything as it is working flawlessly ... Even if it is burning my finger to try mergerfs :D

@testdasi and @yendi I understand.  But, it's worth it.  This isn't a small improvement - it's a major one with worthwhile performance improvements. 

 

(I) simplest migration: Just change your unionfs mount command for the new mergerfs one is worth doing - you can test it works first by just adding the mergerfs command to a new blank script and mounting in a new location

(II) best migration: do (I) + move all your mappings to within /user/mount_mergerfs i.e. /user/mount_mergerfs/downloads and point your download dockers to this location to get hardlinking, file transfer benefits etc

Edited by DZMM

Share this post


Link to post
1 hour ago, DZMM said:

@testdasi and @yendi I understand.  But, it's worth it.  This isn't a small improvement - it's a major one with worthwhile performance improvements. 

 

(I) simplest migration: Just change your unionfs mount command for the new mergerfs one is worth doing - you can test it works first by just adding the mergerfs command to a new blank script and mounting in a new location

(II) best migration: do (I) + move all your mappings to within /user/mount_mergerfs i.e. /user/mount_mergerfs/downloads and point your download dockers to this location to get hardlinking, file transfer benefits etc

I have a working mergefs mount but it just isn't used by anything.

  • I don't do torrent nor hardlink so no benefit there
  • My mount performance is bottlenecked by rclone and not unionfs so no performance benefit either
  • I consider unionfs COW as a protection layer against cryptovirus (and accidental changes). A silent infection + COW would cause my local UD drive to fill up inexplicably which would alert me that something is amiss.

So unless I'm missing something, there isn't anything that motivates me to fully switch at the moment.

 

Share this post


Link to post

Hmm today i found some errors in radarr:

20-1-11 12:23:02.4|Warn|ImportApprovedMovie|Couldn't import movie /downloads/completed/Filme/movie.1995.German.AC3.BDRip.x264-DHARMA-xpost/Jumanji.1995.German.AC3.BDRip.x264-DHARMA.mkv [v0.2.0.1459] System.UnauthorizedAccessException: Access to the path "/Archiv/Filme/movie (1995)/movie 1995.avi" is denied. at System.IO.File.Delete (System.String path) [0x00073] in <254335e8c4aa42e3923a8ba0d5ce8650>:0 at NzbDrone.Common.Disk.DiskProviderBase.DeleteFile (System.String path) [0x00068] in C:\projects\radarr-usby1\src\NzbDrone.Common\Disk\DiskProviderBase.cs:205 at NzbDrone.Core.MediaFiles.RecycleBinProvider.DeleteFile (System.String path) [0x00054] in C:\projects\radarr-usby1\src\NzbDrone.Core\MediaFiles\RecycleBinProvider.cs:90 at NzbDrone.Core.MediaFiles.UpgradeMediaFileService.UpgradeMovieFile (NzbDrone.Core.MediaFiles.MovieFile movieFile, NzbDrone.Core.Parser.Model.LocalMovie localMovie, System.Boolean copyOnly) [0x0005b] in C:\projects\radarr-usby1\src\NzbDrone.Core\MediaFiles\UpgradeMediaFileService.cs:52 at NzbDrone.Core.MediaFiles.MovieImport.ImportApprovedMovie.Import (System.Collections.Generic.List`1[T] decisions, System.Boolean newDownload, NzbDrone.Core.Download.DownloadClientItem downloadClientItem, NzbDrone.Core.MediaFiles.MovieImport.ImportMode importMode) [0x00258] in C:\projects\radarr-usby1\src\NzbDrone.Core\MediaFiles\MovieImport\ImportApprovedMovie.cs:109

 

Any idea? it cant import some new movies? I can access that file via smb, tho. mount_unionfs\google_vfs\Filme\movie (1995)

 

 

root@Unraid-Server:~# ls -ls /mnt/user/mount_unionfs/google_vfs/Filme/movie\ *
/mnt/user/mount_unionfs/google_vfs/Filme/movie (1995):
total 1493104
1493104 -rw-rw-rw- 1 root root 1528938496 Aug 14  2011 movie\ 1995.avi

Edited by nuhll

Share this post


Link to post
Just now, nuhll said:

Hmm today i found some errors in radarr:

20-1-11 12:23:02.4|Warn|ImportApprovedMovie|Couldn't import movie /downloads/completed/Filme/movie.1995.German.AC3.BDRip.x264-DHARMA-xpost/Jumanji.1995.German.AC3.BDRip.x264-DHARMA.mkv [v0.2.0.1459] System.UnauthorizedAccessException: Access to the path "/Archiv/Filme/movie (1995)/movie 1995.avi" is denied. at System.IO.File.Delete (System.String path) [0x00073] in <254335e8c4aa42e3923a8ba0d5ce8650>:0 at NzbDrone.Common.Disk.DiskProviderBase.DeleteFile (System.String path) [0x00068] uin C:\projects\radarr-usby1\src\NzbDrone.Common\Disk\DiskProviderBase.cs:205 at NzbDrone.Core.MediaFiles.RecycleBinProvider.DeleteFile (System.String path) [0x00054] in C:\projects\radarr-usby1\src\NzbDrone.Core\MediaFiles\RecycleBinProvider.cs:90 at NzbDrone.Core.MediaFiles.UpgradeMediaFileService.UpgradeMovieFile (NzbDrone.Core.MediaFiles.MovieFile movieFile, NzbDrone.Core.Parser.Model.LocalMovie localMovie, System.Boolean copyOnly) [0x0005b] in C:\projects\radarr-usby1\src\NzbDrone.Core\MediaFiles\UpgradeMediaFileService.cs:52 at NzbDrone.Core.MediaFiles.MovieImport.ImportApprovedMovie.Import (System.Collections.Generic.List`1[T] decisions, System.Boolean newDownload, NzbDrone.Core.Download.DownloadClientItem downloadClientItem, NzbDrone.Core.MediaFiles.MovieImport.ImportMode importMode) [0x00258] in C:\projects\radarr-usby1\src\NzbDrone.Core\MediaFiles\MovieImport\ImportApprovedMovie.cs:109

 

Any idea? it cant import some new movies? I can access that file via smb, tho. \\192.168.86.103\mount_unionfs\google_vfs\Filme\movie (1995)

Radarr has some issues lately. Had the same issues last couple of days, but now seem to have fixed it. Change your appdata link from /user/ to /cache/.

Share this post


Link to post
9 minutes ago, Kaizac said:

Radarr has some issues lately. Had the same issues last couple of days, but now seem to have fixed it. Change your appdata link from /user/ to /cache/.

Sorry, what exactly you mean by "appdata link"?

 

My appdata is always on "cache only"

 

edit:

no smethingis wrong with the mount.

 

I cant even delete the file via smb.. i can delete it, it vanishes and after some refreshes its back.. wtf

Edited by nuhll

Share this post


Link to post

I think he's saying to set it as /mnt/cache/appdata instead of /mnt/user/appdata.

Share this post


Link to post
1 minute ago, BRiT said:

I think he's saying to set it as /mnt/cache/appdata instead of /mnt/user/appdata.

where?

 

do you mean the volume mapping?

 

/downloads /mnt/user/downloads/ 

 

 

to 

 

/downloads /mnt/cache/downloads/?

 

But isnt it recommended to use the user shares..?

Edited by nuhll

Share this post


Link to post
1 minute ago, nuhll said:

where?

In your docker settings for radarr

Share this post


Link to post

In the UI settings for Docker? You first need to stop Docker in order to be able to make changes there. Maybe also changes in any particular mappings for RADAR to use /mnt/cache/ for temporary paths etc.

 

image.png.5571cacf1aaff9ad59f3670667b67fdd.png

 

 

Edited by BRiT

Share this post


Link to post
1 minute ago, Kaizac said:

In your docker settings for radarr

Yeah... i thought the problem is the source, but, its not, its the destination... it cant delete the destination... saomething is wrong with the mount

Share this post


Link to post
5 minutes ago, nuhll said:

Yeah... i thought the problem is the source, but, its not, its the destination... it cant delete the destination... saomething is wrong with the mount

Just try it.

Share this post


Link to post

Ok, first i install the new update.

 

Thanks.

 

Thats my mount:

 

fusermount -uz /mnt/user/mount_unionfs/google_vfs

#old unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RO:/mnt/user/Archiv=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs

#new
# removing old binary as a precaution
rm /bin/mergerfs

docker run -v /mnt/user/appdata/other/mergerfs:/build --rm trapexit/mergerfs-static-build
mv /mnt/user/appdata/other/mergerfs/mergerfs /bin

mergerfs /mnt/user/Archiv:/mnt/user/rclone_upload/google_vfs:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true

#new

Edited by nuhll

Share this post


Link to post

Ok ive upgraded unraid and changed it to cache.

 

Still same problem I cant delete \mount_unionfs\google_vfs\Filme\movie (1995)\file.avi

 

root@Unraid-Server:~# rm /mnt/user/mount_unionfs/google_vfs/Filme/movie \ \(1995\)/Jumanji\ 1995.avi 
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/Filme/movie (1995)/movie 1995.avi': Read-only file system

Edited by nuhll

Share this post


Link to post
5 minutes ago, nuhll said:

Ok ive upgraded unraid and changed it to cache.

 

Still same problem I cant delete \mount_unionfs\google_vfs\Filme\movie (1995)\file.avi

 

root@Unraid-Server:~# rm /mnt/user/mount_unionfs/google_vfs/Filme/Jumanji\ \(1995\)/Jumanji\ 1995.avi 
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/Filme/Jumanji (1995)/Jumanji 1995.avi': Read-only file system

Check your r/w settings for your mappings in your docker settings. Rw slave for mount unionfs and rw for the rest

Share this post


Link to post
2 minutes ago, Kaizac said:

Check your r/w settings for your mappings in your docker settings. Rw slave for mount unionfs and rw for the rest

I never changed these, i tried rw slave, but not difference. But its nothing about radarr. I cant even delete it in temerinal from unraid.. 

 

rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/Filme/movie (1995)/movie 1995.avi': Read-only file system

 

theres something wrong with the mergefs mount command

 

mergerfs /mnt/user/Archiv:/mnt/user/rclone_upload/google_vfs:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true

 

 

mnt user archiv is my local storage (should be RW)

Share this post


Link to post

Try fixing your share settings through safe permissions under tools

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.