axeman
-
Posts
538 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by axeman
-
-
Anyone else getting dropouts on the MergerFS mount? My mount seemed to have gone down last night. When I try to re-run the mount script, all I see in the logs is:
28.12.2020 13:54:08 INFO: Creating local folders. 28.12.2020 13:54:08 INFO: Creating MergerFS folders.
Rebooting helps, but it goes down again fairly quickly. Started happening last night .
-
4 minutes ago, DZMM said:
Probably the same weird ? problem
hmm the only thing that I did before it happened was move some files around. my local share had a /videos/TV_Classics and a /videos/cloud_Videos/TV_Classics I moved the entire folder (using a Windows machine) from /local/Videos/ to /local/cloud_Videos
That was basically the last thing I did (re-parent the directory).
a quick reboot fixed it - just curious if the reparenting did it.
-
Strangely - my array went down after an upload finished. Tried to run the upload script again and am getting this now:
Quotetouch: cannot touch '/mnt/user/appdata/other/rclone/remotes/cloud_Videos/upload_running_daily_upload': Transport endpoint is not connected
-
2 hours ago, DZMM said:
Are you sure this is 'live' ? I think the log is showing what happened when your upload script kicked in
Yeah, it's live. I just tried watching a show, and Emby must've updated the .nfo file, because log shows similar (upload script hasn't been run since a reboot).
Maybe smaller files that are already in the rclone cache get upload directly?
When I copy a large (new) file to the mergerFs mount, it does exactly as expected. The file goes to the corresponding folder on the "local" share.
I'll test some more in a few hours.
-
Interstingly - it looks like mergerFS files aren't going to the "local" but are being uploaded directly...
Quote2020/12/20 23:31:56 INFO : TV_Classics/MyShow/tvshow.nfo: Copied (replaced existing)
2020/12/20 23:31:56 INFO : TV_Classics/MyShow/tvshow.nfo: vfs cache: upload succeeded try #1Seeing lots of these from a Sonarr refresh.
-
Hi there!
Is there a way to control the order of execution on the "at Array First Start?" I have three scripts that I'd like two to run BEFORE the third one runs. Does it go by the script name or by what's in the #name variable? or something else?
-
2 minutes ago, DZMM said:
I don't think the issue is at Google's end, as the problem is with the MergerFS mount - the rclone mounts seem to be behaving
Yes - excellent observation. I see the mergerfs github shows some activity from 5 days ago on fuse.c - maybe there was a breaking change? MergerFS gets installed at each array reboot right?
-
something is definitely going on ... even after a reboot, it came back. I'm in the middle of an upload on my paltry 400Mbit down/40Mbit up connection.... so definitely not hitting the quota - even though I do have service accounts.
wonder if this is related to the stuff that happened last week w. google... maybe they are putting in some other sort of throttling?
-
I got the same "Transport endpoint is not connected" error last night as I was uploading. Rebooting fixed it.
Funny thing was, I saw a lot of symmetrical read/writes on the flash drive.
-
20 minutes ago, DZMM said:
Is anyone else getting a lot of buffering tonight? I'm wondering if it's linked to Google's outage yesterday?
My entire array went and borked itself last night/this morning. one of my ARRs was doing a massive update at the same time as Emby's library scan. Not sure if this was all related, but I stopped everything, manually deleted the rclone_cache, and restarted UnRaid. Everything seems OK again.
-
16 hours ago, DZMM said:
what's your objective?
- Main tdrive getting too big? Create a new rclone mount with a new tdrive and disable the new mergerfs share in that new script. In your original script add the new rclone mount as an extra local folder
- you want another mergerfs mount for plex or similar with just your TV shows? I would manually create a 2nd mergerfs mount that merges the rclone remote that's already mounted on your PC and the corresponding local folder. That way you've got only got 1 rclone mount of the remote
None of the above - more so that I can have a protected share that the kiddos can't see.
-
On 12/5/2020 at 2:37 PM, DZMM said:
It's in the script settings:
MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disableOkay - final dumb question - I want to completely separate out my adult tv shows so that it's not on the same mount. Can I have another merger fs mount, and corresponding upload script?
Or will the conflict with each other?
-
1 hour ago, DZMM said:
It's in the script settings:
MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disableDOH! SORRY. i thought it was a whole different script.
-
On 6/22/2020 at 6:14 PM, DZMM said:
Multiple mounts, one upload and one tidy-up script.
@watchmeexplode5 did some testing and performance gets worse as you get closer to the 400k mark, so you'll need to do something like below soon:
1. My folder structure looks something like this:
mount_mergerfs/tdrive_vfs/movies
mount_mergerfs/tdrive_vfs/music
mount_mergerfs/tdrive_vfs/uhd
mount_mergerfs/tdrive_vfs/tv_adults
mount_mergerfs/tdrive_vfs/tv_kids
2. I created separate tdrives / rclone mounts for some of the bigger folders e.g.
mount_rclone/tdrive_vfs/movies
mount_rclone/tdrive_vfs/music
mount_rclone/tdrive_vfs/uhd
mount_rclone/tdrive_vfs/adults_tv
for each of those I created a mount script instance where I do NOT create a mergerfs mount
3. I mount each in turn and for the final main mount add the extra tdrive rclone mounts as extra mergerfs folders:
############################################################### ###################### mount tdrive ######################### ############################################################### # REQUIRED SETTINGS RcloneRemoteName="tdrive_vfs" RcloneMountShare="/mnt/user/mount_rclone" LocalFilesShare="/mnt/user/local" MergerfsMountShare="/mnt/user/mount_mergerfs" # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="/mnt/user/mount_rclone/music" LocalFilesShare3="/mnt/user/mount_rclone/uhd" LocalFilesShare4="/mnt/user/mount_rclone/adults_tv"4. Run the single upload script - everything initially gets moved from /mnt/user/local/tdrive_vfs to the tdrive_vfs teamdrive
5. Overnight I run another script to move files from the folders that are in tdrive_vfs: to the correct teamdrive. You have to work out the encrypted folder names for this to work. Because rclone is moving the files, the mergerfs mount gets updated i.e. it looks to plex etc like they haven't moved
#!/bin/bash rclone move tdrive:crypt/music_tdrive_encrypted_folder_name gdrive:crypt/music_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs rclone move tdrive:crypt/tv_tdrive_encrypted_folder_name tdrive_t_adults:crypt/tv_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs rclone move tdrive:crypt/uhd_tdrive_encrypted_folder_name tdrive_uhd:crypt/uhd_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs exitHi - can you share that mount script with the merger FS parts striped out?
Quotefor each of those I created a mount script instance where I do NOT create a mergerfs mount
is it just this part?
# create rclone mount rclone mount \ $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ --allow-other \ --dir-cache-time $RcloneMountDirCacheTime \ --log-level INFO \ --poll-interval 15s \ --cache-dir=$RcloneCacheShare/cache/$RcloneRemoteName \ --vfs-cache-mode full \ --vfs-cache-max-size $RcloneCacheMaxSize \ --vfs-cache-max-age $RcloneCacheMaxAge \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation &
-
1 hour ago, chefmoisas said:
There is a pid that i can't stop
That's just from your command - you should see that your array has now stopped.
-
9 minutes ago, chefmoisas said:
Hi guys
I use this guide to install rclone and after i start the mount plugin i can't stop it anyway, even if I start the unmount plugin. If i try to stop the array I get this message "Array Stopping•Retry unmounting user share(s)...".
Anyone with a solution ?
Many thanks
This is a bit crude - but i run this in terminal:
ps -ef | grep /mnt/user
for each pid that comes up, i do this:
kill <pid>
Someone can probably script a better way to do it and run at array stop, but for now that's what i've been doing.
-
On 11/18/2020 at 10:50 AM, MowMdown said:
That is strange, Im not sure why it would write to the _vfs upstream if you have your local path listed first in the union when writing to the union... for the record you can export /mnt/disks/some_dir as a share through SMB example:
#unassigned_devices_start #Unassigned devices share includes include = /tmp/unassigned.devices/smb-settings.conf #unassigned_devices_end [some_dir] path = /mnt/disks/some_dir comment = browsable = yes # Public public = yes writeable = yes vfs object =You simply add this to the SMB extras under Settings > SMB
So even with this - Windows cannot seem to write to the share via SMB. Googled it, and seems like sharing an rclone vfs mount can work with some modifications. Don't know that I have the know how to do that.
I'm grateful for your time - going through it like this with me. I may have to go back to merger fs.
-
On 11/18/2020 at 10:50 AM, MowMdown said:
That is strange, Im not sure why it would write to the _vfs upstream if you have your local path listed first in the union when writing to the union... for the record you can export /mnt/disks/some_dir as a share through SMB example:
#unassigned_devices_start #Unassigned devices share includes include = /tmp/unassigned.devices/smb-settings.conf #unassigned_devices_end [some_dir] path = /mnt/disks/some_dir comment = browsable = yes # Public public = yes writeable = yes vfs object =You simply add this to the SMB extras under Settings > SMB
Thanks - I will try that. Didn't know it was possible. Incidentally, do you know where /mnt/disks/some_dir is physically located? like does it go on Cache drive? or is it in memory/ram?
-
On 11/14/2020 at 10:12 AM, MowMdown said:
when writing to the union mount directory “media” the non “vfs” one, shouldn’t be touching the cache because it should only write to the local drives your first upstream in the Union setup. Sounds like maybe you should check the spelling/case of that first path.
you might need to add the flag
-vvto the mount command so you can verbosely debug the issue further.
This might be a me thing - since I need the rclone mounts available to my Windows machines, I have it in /mnt/user .
If I try to copy a file into unioned mount from inside of UnRaid via MC, it works exactly as you'd want... the file goes right to the local share.
However, if I do the same from a windows machine - it fails.
Interestingly this doesn't seem to be problem on an Android device (using SolidExplorer). Nor does it happen with the @DZMM mergerfs based scripts.
-
On 11/9/2020 at 9:38 AM, MowMdown said:
@axeman, no, if you pay close attention to the path for that move command, im using "/mnt/user/media" not "/mnt/disks/media" (I don't mount to /user/)
My rclone mount is under "/mnt/disks/media" so it does not interfere with the move. I'm essentiall moving the files from /mnt/user/media to the "crypt:media" mount but as far as the unraid is concerned the file isn't actually moving since no matter where I put the file it always shows up in /mnt/disks/media.
So I'm having trouble creating files on the union location. It's strange because I if I go directly to the media_vfs mount, I can create files. But can't on the media one. I even tried installing the unassigned devices and updated the paths to /disks/ instead of /user/ ..
What could I be doing wrong?
Thanks for your time.
Edit : this shows up on the script log:
/test2.txt: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
/test2.txt: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
an rclone forum post said to set the cache mode ... but I see we are already doing that on the mount script. It seems to happen regardless of the :nc modifier.
-
2 hours ago, Hoopster said:
The permanent fix for this issue is already in unRAID. It's called WS Discovery (In SMB settings) and is Microsoft's replacement for network discovery that does not rely on SMB 1.0. Microsoft has been trying to get rid of SMB 1.0 for years due to security issues and they will likely continue to disable it by default before eventually dumping it all together.
I have completely disabled SMB 1.0 support on all my WIndows machines and unRAID ever since unRAID began to support WSD many months ago in version 6.8.0. It seems to work great and no more problems with SMB 1.0.
Here are the unRAID settings I currently have for WSD (disabling NetBIOS turns off SMB 1.0 support on unRAID):
Thanks! where'd you get the info on what to put for the WSD options?
Strangely my array was fine - but started doing this after I started using rclone - i thought it was related to my NIC. but seems to be OK when I use IP address.
-
10 hours ago, MowMdown said:
@axeman, no, if you pay close attention to the path for that move command, im using "/mnt/user/media" not "/mnt/disks/media" (I don't mount to /user/)
My rclone mount is under "/mnt/disks/media" so it does not interfere with the move. I'm essentiall moving the files from /mnt/user/media to the "crypt:media" mount but as far as the unraid is concerned the file isn't actually moving since no matter where I put the file it always shows up in /mnt/disks/media.
Thanks I noticed that - but don't really understand the difference. I thought perhaps you just typed it as an example of your script.
-
On 11/4/2020 at 3:11 PM, MowMdown said:
I just run this nightly at 3am using user scripts, super simple. (obviously I don't have a folder named "files" but you can use your imagination)
rclone move /mnt/user/media/files crypt:files -v --delete-empty-src-dirs --fast-list --drive-stop-on-upload-limit --order-by size,descI have a single 500GB drive that I fill up with whatever I want to be moved to the cloud and that small script does it.
sorry if this is a dumb question (maybe my imagination isn't working) ... but isn't that /media the mount where your union is? So would rclone know where the "local" version is? Or does it basically traverse the entire folder to see if there's differences between crypt:files and /mnt/user/media/files ?
-
11 minutes ago, MowMdown said:
I just run this nightly at 3am using user scripts, super simple. (obviously I don't have a folder named "files" but you can use your imagination)
rclone move /mnt/user/media/files crypt:files -v --delete-empty-src-dirs --fast-list --drive-stop-on-upload-limit --order-by size,descI have a single 500GB drive that I fill up with whatever I want to be moved to the cloud and that small script does it.
whoa - that's nice indeed. I like the service account rotation - but maybe not needed given my upload bandwidth is 40mbit anyway.
ESXi 5.x - pre-built VMDK for unRAID
in Virtualizing Unraid
Posted
Not sure if it's relevant, but the latest version of ESXi allow booting from flash. You can remove a layer of complexity.