[Plexdrive]

2 minutes ago, DZMM said:

your mount scripts look fine, although I don't know what the implications are of having two vfs mounts e.g. do they behave properly and don't interfere with each other.

 

Post your move scripts as well - these are what used to cause my unraid server to run out of memory.

 

The 2 vfs mounts are connected to seperate Gdrives (one is my own and the other is a friends share for read only). I found out it was Plex who was filling my memory, every item it was scanning was taking up memory which stacked. Shutting down the Plex docker dropped the memory use.

 

Upload script is currently only used for Series:
 

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/Rclone/rclone_upload" ]]; then
exit
else
touch /mnt/user/Rclone/rclone_upload

fi

# set folders

uploadfolderSeries="/mnt/user/Media/Uploaden/Series_upload"
#uploadfolderFilms="/mnt/user/Media/Uploaden/Films_upload"


# move files

rclone move $uploadfolderSeries Gdrive_Alles_Upload:/Series -vv --drive-chunk-size 512M --delete-empty-src-dirs --checkers 10 --fast-list --transfers 4 --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle/** --exclude *.backup~* --exclude *.partial~* --bwlimit 8500k
#rclone move $uploadfolderFilms Gdrive_Alles_Upload:/Films -vv --drive-chunk-size 512M --delete-empty-src-dirs --checkers 10 --fast-list --transfers 4 --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle/** --exclude *.backup~* --exclude *.partial~* --bwlimit 8500k

rm /mnt/user/Rclone/rclone_upload

exit

 

[Plexdrive]

Does anyone know what this mean? I see it in the log of my mount script for Gdrive. I dont know understand why it's transferring file since I'm not running the upload script.

 

Quote

2018/08/03 11:15:39 INFO :
Transferred: 601.335 MBytes (110.333 kBytes/s)
Errors: 0
Checks: 0
Transferred: 731
Elapsed time: 1h33m0.9s
Transferring:
* ...- S01E03 - The Entire History of You.mp4: 0% /271.934M, 69.993k/s, 1h5m44s

2018/08/03 11:15:41 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: unexpected EOF
2018/08/03 11:15:41 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2018/08/03 11:15:41 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 2/10: unexpected EOF
2018/08/03 11:15:42 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 3/10: unexpected EOF
2018/08/03 11:15:42 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 4/10: unexpected EOF
2018/08/03 11:15:43 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 5/10: unexpected EOF
2018/08/03 11:15:50 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 6/10: unexpected EOF
2018/08/03 11:15:51 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 7/10: unexpected EOF
2018/08/03 11:15:51 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 8/10: unexpected EOF
2018/08/03 11:15:52 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 9/10: unexpected EOF
2018/08/03 11:15:57 ERROR : Series/Black Mirror/Season 1/Black Mirror - S01E03 - The Entire History of You.mp4: ReadFileHandle.Read error: low level retry 10/10: unexpected EOF

 

[Plexdrive]

15 minutes ago, DZMM said:

Your problem is different.  My mounts are fine as I can browse files via putty, SMB etc but my dockers won't see any anything.  I used to run into memory problems with my move jobs - post your rclone lines.

I've tried rw and that didn't help.  Are they supposed to be rw slave even if mounted at mnt/user?  

 

I've been importing from array-->array or ud-->array.  I'm going to try some ud-->cache imports today to see if it's been an io problem on my array.  I'm pretty sure it's not, but that will confirm whether or not.

 

You mount unionfs on /mnt/disks right? Isn't that on the array directly, so that would mean your writing to it would be slowed. I've mounted on mnt/user/Media and put the Media share also on Cache (with mover invoked at night) and those writes are fast now.

 

Edit: I'm talking bullshit, your /mnt/disks are not on your array. So maybe it just has to do with not mounting a /mnt/user/XX folder from the docker.

 

My rclone lines. I've just added -attr-timeout 60s because apparantly the out of memory issue crashes the mount (probably because of plex and emby scanning the libraries causing too much strain on the system). This line should mitigate it. I've also disabled thumbnail creation and such in both dockers. Waiting for my server to reboot and then hopefully the mounts are correct.

I'm writing to GdriveStream which is also the mount that keeps disconnecting.

rclone mount --allow-other --dir-cache-time 48h --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit 1G --buffer-size 1G --log-level INFO --stats 1m --attr-timeout 60s Gdrive_Alles_Crypt: /mnt/disks/GdriveStream &
rclone mount --max-read-ahead 512M --allow-non-empty --allow-other OnedriveBen: /mnt/disks/Onedrive &
rclone mount --max-read-ahead 512M --allow-non-empty --allow-other OnedriveGroot: /mnt/disks/OnedriveGroot &
rclone mount --allow-other --dir-cache-time 48h --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit 1G --buffer-size 1G --log-level INFO --stats 1m --attr-timeout 60s Gdrive_Alles_Media: /mnt/disks/MediaLibGdrive &

 

[Plexdrive]

9 hours ago, DZMM said:

I've just added a 2TB unassigned for my usenet and torrent downloads - hopefully this will speedup my import to mu unionfs folders as I'm only getting around 5MB/s.  It's going to take a while for me to know as I need to wait for old torrents to finish downloading before moving them.

 

I wonder sometimes if my unionfs mount at /mnt/disks rather than /mnt/user/somewhere is the culprit.  If the UD doesn't work, I think I'll give /mnt/user/somewhere another crack. 

 

I aborted my last attempt as for some weird reason sonarr/radarr/plex etc sometimes wouldn't see the mount files - anyone else come across this?  I could see the files via putty/SMB etc, but the dockers would be temperamental.

 

 

 

I have the same problem now I think. My Gdrive gets disconnected after a while. I also get out of memory notifications which is weird since I have 28GB of ram and I don't see anything using memory. So apparently it's spiking and then just disconnecting the Gdrive. This causes Sonarr to stop importing, since it gets an access denied (even though the local upload folder is still up).

Very frustrating.

[Plexdrive]

2 minutes ago, slimshizn said:

Would turning on privileged do it?

 

Is the unassigned cache mounted as RW or RO?

[Plexdrive]

 

Just now, slimshizn said:

That's already where I have it pointed actually. Still getting the perm error.

 

Remove the -unmask part from your rclone mount line. I think that caused my previous errors aswell. Might be worth trying.

[Plexdrive]

Sonarr should point to /mnt/user/Media/Cloud/Series in your case. Then it will automatically add it to /mnt/user/Media/Cloud/Seriestmp for the upload cron job DZMM uses.

 

Does that change things for you?

[Plexdrive]

4 minutes ago, slimshizn said:

Is this the culprit?

 

 

Shouldn't be. But I've seperated the Upload and Gdrive folders (in your case the unionfs would be in /Media/Series and upload in /Media/Uploads/Seriestmp and the gdrive in /Media/Cloud/Series. Maybe it makes a difference? I just check my logs and didn't have these errors. My config is this:

 

unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Uploaden/Films_upload=RW:/mnt/disks/GdriveStream/Films=RO /mnt/user/Media/Films
unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Uploaden/Series_upload=RW:/mnt/disks/GdriveStream/Series=RO /mnt/user/Media/Series

 

[Plexdrive]

5 hours ago, slimshizn said:

I use a ssd cache and the unassigned drive is ssd. If you use a single no raid config xfs is said to have the best performance. 

 

Thanks, just upgraded to a 1TB SSD cache using XFS and it seems a lot faster now!

 

@DZMM: how does Sonarr/Radarr handle "upgrades" with the unionfs setup? I think it can only upload, but not delete the worse quality versions right? Or does that work as usual?

[Plexdrive]

2 hours ago, slimshizn said:

Try moving your download folder to an unassigned drive, as well as appdata to the unassigned drive. After I did my io issues are almost non existent.

 

Are you using a regular spinner for this or an SSD? I think a big SSD cache should work the same right?

[Plexdrive]

9 minutes ago, DZMM said:

 

Nope.  My family are going away next week so I'm going to experiment with mounting unionfs at /mnt/user/rclone_vfs/movies rather than /mnt/disks/rclone_vfs/movies to see if file write speeds improve/hardlinking is supported etc

 

if you just mount gdrive does it work?  Try putting a pause in - I've added a 5s delay between mounting gdrive and unionfs as I had similar problems.

 

Not sure what the permission errror are

 

I'm mounting the unionfs on the mnt/user/Media folder (one for Movies one for Series) and that works perfectly, no errors. I think my slow speed has to do with my array being overloaded by io, since I'm also writing a lot of files to an Rclone cache. Add Sabnzbd extraction and Sonarr moving to the mix and it's too much for the regular HDD array.

 

Thanks for the tip about the delay, I saw that in your config but didn't know why you put it in. Will try that later on.

 

[Plexdrive]

19 hours ago, DZMM said:

 

I've:

• increased chunk-size-limit and buffer-size to 2G as I'm more confident now that I've got enough memory after running for a few weeks now
• deleted umask 002 as being honest I don't know what it does
• removed --bind as I don't think it's needed for my setup
• removed --cache-dir and --vfs-cache-mode as this is only needed if you write files direct to the vfs mount, which I don't - I use a scheduled rclone move job 

rclone mount --allow-other --dir-cache-time 48h --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit 2G --buffer-size 2G --log-level INFO gdrive_media_vfs: /mnt/disks/rclone_vfs --stats 1m

I'm tempted one day to try --vfs-read-chunk-size 64M as sometimes I have to start a file twice, which I think is because it doesn't have enough info to start playing the first time.  The trade-off is my start times might be a little longer

 

Thanks, I've changed those in my configs as well.

Are you still getting slow writes from Radarr/Sonarr to your unionfs folders? To me it seems slower indeed, so maybe putting it on an SSD cache will help. Will try then when I get a bigger SSD.

 

Another thing which is strange is that I have to run the mount script twice for it to mount the unionfs folders from within my Gdrive. First it mounts my Gdrive but not the folders within and then I have to mount again to also get the subfolders. Maybe the mount is too slow the first time so it needs some time to load the folders within.

 

Sonarr is also giving me permission errors in the logs, are you getting these aswell?

 

18-7-31 17:05:02.2|Warn|MediaFileAttributeService|Unable to apply permissions to: /tv/The Blue Planet/Season 1

[v2.0.0.5228] NzbDrone.Mono.Disk.LinuxPermissionsException: Error setting file owner and/or group: EPERM
  at NzbDrone.Mono.Disk.DiskProvider.SetOwner (System.String path, System.String user, System.String group) [0x00057] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Mono\Disk\DiskProvider.cs:223 
  at NzbDrone.Mono.Disk.DiskProvider.SetPermissions (System.String path, System.String mask, System.String user, System.String group) [0x00008] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Mono\Disk\DiskProvider.cs:75 
  at NzbDrone.Core.MediaFiles.MediaFileAttributeService.SetMonoPermissions (System.String path, System.String permissions) [0x0000f] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Core\MediaFiles\MediaFileAttributeService.cs:88 

[Plexdrive]

35 minutes ago, DZMM said:

Unionfs merges the upload folder and the Google folder.  When sonarr tries to add a new file, it is actually written to the upload folder not the Google folder - when you move it to the Google folder, to sonarr it looks like it hasn't moved as sonarr is looking at the unionfs folder.

 

Radarr/sonarr etc should be pulling files in from sab/nzbget etc, not sab writing directly to your media folders

 

Thanks for the clarification, I'm amazed it works like that! Did you change anything in your mounting settings (buffers and such) since you last tried this:

 

rclone mount --allow-other --dir-cache-time 48h --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit 1G --buffer-size 1G --umask 002 --bind 172.30.12.2 --cache-dir=/mnt/software/rclone/vfs --vfs-cache-mode writes --log-level INFO --stats 1m gdrive_media_vfs: /mnt/disks/rclone_vfs

[Plexdrive]

1 minute ago, DZMM said:

You point them to your unionfs folders, that merge what's been uploaded with what's local

 

Thanks for the quick reply!

 

But how does Radarr/Sonarr then transfer the files to the upload folder? Did you set up Sabnzbd to your upload folder?

[Plexdrive]

On 7/14/2018 at 8:20 PM, DZMM said:

 

So far I've moved all of my tv shows and a good chunk of my movies.  At first I selected the less important ones by moving to a new share that rclone watches and uploads from and is part of the unionfs folder, but now I'm going to focus on the smaller, less important ones so there's less chance of something going wrong.  I'm not too worried about uploading more important ones though as I'm sure if Google scupper the setup there'll be time to download my content if I want to.

 

Thanks for all your scripts, info and invested time in getting everything right. I've been "following" you here and on the Rclone forums. It's been quite a challenge to get everything working for my set-up based on your scripts and so far I'm at 80% I think. However there is something I don't get. If you point Sonarr/Radarr to your _upload folder (which is empty when there is nothing to be uploaded) how do you make Sonarr/Radarr know what is already in your library and which files still need to be downloaded or can be upgraded?

[[Support] Linuxserver.io - SWAG - Secure Web Application Gateway (Nginx/PHP/Certbot/Fail2ban)]

53 minutes ago, aptalca said:

 

Plex has a built in setting for that

 

That only works with a Plex Pass AFAIK, which I do not have. And since I also have Nextcloud I would like to be able to limit it from within NGINX (or maybe there is an other and better solution).

[[Support] Linuxserver.io - SWAG - Secure Web Application Gateway (Nginx/PHP/Certbot/Fail2ban)]

I would like to limit the upload speed of my server to the outside world for specific dockers. Is this possible to do? I'm currently running Nextcloud and Plex for which I would like to limit the upload speed. I've found some articles about rate limiting in NGINX, but I can't make much sense of it to apply it.

 

Would be nice to make sure other people are not collapsing my full bandwidth.