Issue with Cache and Krusader, copying from Unassigned Devices

[CameraRick]

Hi there,

 

I ran into an issue earlier and can't quite figure it out.

 

For initial ingesting of a project I connected a USB Harddrive, HFS+ filesystem. I mount it with the Unassigned Devices plugin.

Then I use Krusader to copy the files over, as the Share I copy to is set to "Cache: Yes", it copies to cache first.

However, seemingly this just floods the cache, claims that I can't copy to this share anymore because it's full. So, it doesn't "dynamically" switch to the array once cache is full, like if I would copy via SMB. When I set the share to "Cache: No", I can copy further just fine.

Now with the Cache deactivated some data still sits on the Cache, but I can't use the Mover till I enable it again, which is kind of a bummer.

 

It's some terabytes of data and the cache isn't that big, also because the copy takes some time I don't want to block a 2nd machine (+unRAID ressources) and copy through SMB over many hours. Also, ideally only one machine adds to the power bill, not two, haha

 

Any way I can get around that?

 

Best,

Rick

[JorgeB]

31 minutes ago, CameraRick said:

Any way I can get around that?

Use cache="prefer", if you turn the GUI help on you can see all the options.

[itimpi]

54 minutes ago, CameraRick said:

However, seemingly this just floods the cache, claims that I can't copy to this share anymore because it's full. So, it doesn't "dynamically" switch to the array once cache is full,

What is the actual path you are using under Krusader and what is this mapped to at the Unraid level.    Reason I am asking is that if you are explicitly using the cache as the target you are by-passing the User Share system

i believe that if you are copying lots of files in one go then Krusader creates the folders first which might affect which drive is actually used.

[CameraRick]

54 minutes ago, JorgeB said:

Use cache="prefer", if you turn the GUI help on you can see all the options.

I'm aware, but "prefer" would be the exact opposite of what I want? It's "the same" as "Yes", but when Mover is invoked it copies from Array to Cache, not Cache to Array

 

31 minutes ago, itimpi said:

What is the actual path you are using under Krusader and what is this mapped to at the Unraid level.    Reason I am asking is that if you are explicitly using the cache as the target you are by-passing the User Share system

i believe that if you are copying lots of files in one go then Krusader creates the folders first which might affect which drive is actually used.

Hi itimpi,

 

I'm not so Linux-structure-savvy, so I hope I answer correct.

I go to

/mnt/user/disks/HDD_NAME

for the HDD mounted in Unassigned Devices and then copy it to

/mnt/user/user/SHARE_NAME

 

So from my understanding, I do not use Cache specifically? When I changed the Setting from "yes" to "no", I didn't change anything inside Krusader and it worked right away (not sure if that is an indication)

 

Best,

Rick

[JorgeB]

2 minutes ago, CameraRick said:

I'm aware, but "prefer" would be the exact opposite of what I want?

Yes, sorry, blonde moment, cache=yes is the correct option, but note that split level overrides that, and IIRC Krusader creates all the folders before starting the copy, minimum free space for the share also needs to be correctly set, if Krusader still doesn't work try with Windows explorer, it will still do the copy/move locally.

[CameraRick]

On 2/8/2022 at 6:16 PM, JorgeB said:

Yes, sorry, blonde moment, cache=yes is the correct option, but note that split level overrides that, and IIRC Krusader creates all the folders before starting the copy, minimum free space for the share also needs to be correctly set, if Krusader still doesn't work try with Windows explorer, it will still do the copy/move locally.

Haha no worries thanks for getting back to me!

 

The Minimum Free Space for the share is at default(?) of 0KB, and according to the tooltip should be equal or greater than the biggest file I think of using. As we do work with video data, files can easily be 70GB or more at a piece, should I use that then?

As far as I understand the Cache-usage, it "dynamically" shifts from Cache to array when cache is full, while (over SMB) making it seem like it's all in one place. So, Krusader making the folders first somehow circumvents that?

 

Over SMB, inside Explorer (or rather FInder, we are on macOS) it works just fine, but I really don't want to use that because it's seemingly slower (could be wrong here) and more importantly blocks the machine. I can't fully utilize it while copying data, no restart etc. Ideally, I really want to keep it all on THAT machine, so no other machine has to run (and "waste" network traffic or whatever).

 

Just a thought as I have no experience with VMs yet - could I run a VM and do the copy there? Then it would be "self-contained" on the server, no?

 

Best,

Rick

[JorgeB]

39 minutes ago, CameraRick said:

files can easily be 70GB or more at a piece, should I use that then?

Value needs to be larger than the largest file you expect to copy there, we usually suggest using twice the value of the largest file you expect to copy, to have some margin.

 

40 minutes ago, CameraRick said:

or rather FInder, we are on macOS

Don't know is OSX supports server side copy with Samba, like Windows explorer does, if it doesn't data will transverse the network and will be much slower.

 

41 minutes ago, CameraRick said:

Just a thought as I have no experience with VMs yet - could I run a VM and do the copy there? Then it would be "self-contained" on the server, no?

Yes, and if a Windows VM you can use explorer to make a local copy/move like mentioned.

[CameraRick]

Hi Jorge,

 

it's really hard to tell because if there's a Master-File for a 90min film, which is not often but can happen, it's easily 400GB. Setting that to 800GB/1TB, which is larger than the Cache actually is atm, I'm not sure if that's helpful? Or do I get this function wrong?

 

I'm also not sure if macOS lets you do this, but as I don't want a 2nd machine for this anyway and a VM is a viable solution, I will totally use just that benefit is that other collegues which are even less server-savvy than I am can copy data much easier than using Krusader (which quickly becomes intimidating with all the filesystem-folder-structure).

Probably a small Linux distro will work just fine for this endevour, or something lightweight like Windows 7

 

Best,

Rick

[JorgeB]

11 minutes ago, CameraRick said:

it's really hard to tell because if there's a Master-File for a 90min film, which is not often but can happen, it's easily 400GB. Setting that to 800GB/1TB, which is larger than the Cache actually is atm, I'm not sure if that's helpful? Or do I get this function wrong?

 

You'd need to get a bigger cache device or set the minimum free space to a lower value, like 100GB, but if a file larger than that comes up copy directly to the array, or check manually that cache has enough free space before starting the copy.

 

13 minutes ago, CameraRick said:

or something lightweight like Windows 7

Only Windows 8 or newer supports Samba's server side copy using windows explorer.

[CameraRick]

Hi there Jorge,

 

hm I see. Well, when I copy a project with multiple Terabytes I'm well aware that the Cache won't hold it, haha however the issue I encountered doesn't seem to root from one large file flooding the cache, but the cache not switching to array like it would when I would do the copy over SMB.

 

21 hours ago, JorgeB said:

Only Windows 8 or newer supports Samba's server side copy using windows explorer.

Ok I see - but if the VM runs on the unRAID, there's no network inbetween to slow it down greatly, no?

Also, do you have an idea if "Explorer" in this case is interchangable with any software that copies files? I'm not a big fan to move large amounts of data with Explorer or Finder, I rather use e.g. FreeFileSync or so

 

Best,

Rick

[itimpi]

13 minutes ago, CameraRick said:

but the cache not switching to array like it would when I would do the copy over SMB.

Have you set the Minimum Free Space setting for the cache pool?    It is the free space dropping below this value that triggers Unraid to start bypassing the cache.

[JorgeB]

12 minutes ago, CameraRick said:

but the cache not switching to array like it would when I would do the copy over SMB.

Minimum space incorrectly set will cause that, as well as split level.

 

13 minutes ago, CameraRick said:

but if the VM runs on the unRAID, there's no network inbetween to slow it down greatly, no?

There's still a virtual network that will be used if the OS doesn't support server side copy, making it much slower.

 

 

14 minutes ago, CameraRick said:

Explorer

This is Windows explorer, robocopy also works, others might not.

 

https://wiki.samba.org/index.php/Server-Side_Copy

[JonathanM]

1 hour ago, JorgeB said:

robocopy also works,

There is a tickle in the back of my brain about someone having massive issues with robocopy and Unraid, but I can't remember what it was about. Permissions and extended attributes perhaps?

[JorgeB]

41 minutes ago, JonathanM said:

Permissions and extended attributes perhaps?

Permissions should be fine, not sure about extended attributes, but never used it much.

 

 

 

[JonathanM]

https://forums.unraid.net/topic/54717-robocopy-windows-to-unraid-ruining-all-permissions-of-destination-folder/

https://forums.unraid.net/topic/79957-robocopy-does-not-sync-correctly/

@Squid? Any idea if robocopy is still problematic?

 

I've never personally used it, so don't have anything to add other than hearsay.

[Squid]

Haven't used it in years, but as I posted in that one thread, it did consistently trash all of my permissions when I was using it.

[CameraRick]

Hey guys,

 

thank you so much for all the infos, sorry I get back so late. I will check all of this back in the office. You guys rock!