Empty folders when migrating large collection to share

[stev067]

Hey everyone, I hope this is an easy one since I'm new...

I'm trying to dump a 30TB collection of 600+ folders each containing files, onto a share using Krusader. It will span multiple disks, and I set the share to split only on top level directory. When I initiate the transfer, it immediately creates all 600+ folders onto the first disk, and then begins to add the files. Since I'm using highwater, once the drive reaches 50% it will switch to writing to the next drive, but I don't want all of these empty folders being left behind. Will it delete those? Am I doing something wrong, or do I just need to clean up empty folders afterwards?

Thanks!

[JorgeB]

8 minutes ago, stev067 said:

When I initiate the transfer, it immediately creates all 600+ folders onto the first disk

That's normal with Krusader (and rsync), depending on the source you may be able to use Windows explorer, it won't do that.

[stev067]

23 minutes ago, JorgeB said:

That's normal with Krusader (and rsync), depending on the source you may be able to use Windows explorer, it won't do that.

So I have a windows 10 VM running on this machine. I could pass through this external drive which is the source, and then transfer to the share via windows. And that would solve my problem?

[FreeMan]

Unless you're accessing the DISKxx shares directly (and generally you shouldn't be), you'll never notice the empty directories. The UNRAID OS will aggregate the contents of the splits and it'll look seamless to you. I get that simply knowing that there are empty directories there can be annoying, but it won't really impact anything.

 

Krusader will, most likely, copy the files far faster than even Win10 running in a VM, if speed is of any consideration.

[stev067]

3 hours ago, FreeMan said:

Unless you're accessing the DISKxx shares directly (and generally you shouldn't be), you'll never notice the empty directories. The UNRAID OS will aggregate the contents of the splits and it'll look seamless to you. I get that simply knowing that there are empty directories there can be annoying, but it won't really impact anything.

 

Krusader will, most likely, copy the files far faster than even Win10 running in a VM, if speed is of any consideration.

Thanks. Another thing I realized though is that after the first drive reaches capacity during this initial write, because all of the folders are on that drive and because of the way my split is set at top level only, it will just stop and say the drive is full. And the reason I want to use the top level only split is to minimize multiple drive spin-ups. I'm just writing in chunks now to try and manually balance the drives for this initial write. But I'm a little disappointed this hasn't been baked into the programming yet!

Edited May 13, 2021 by stev067

[JonathanM]

46 minutes ago, stev067 said:

I'm a little disappointed this hasn't been baked into the programming yet!

What would you change?

[stev067]

40 minutes ago, jonathanm said:

What would you change?

I'm not sure if this is a Krusader or Unraid thing, but when writing anything to a share, the entire folder structure of the source shouldn't just be dumped onto the first destination drive at the start. I imagine the folders can be added on a file-by-file basis, mirroring the source's folder structure as the write progresses. That way, when you reach write capacity of the first drive, (or 50% in the case of highwater), the split level rule won't prevent the next-in-line folders from being written to the next-in-line drive, because those empty folders won't have been pre-written to the first drive.

Edited May 14, 2021 by stev067
a word

[JonathanM]

Unfortunately Unraid has no control over what gets written first. It's totally controlled by the application doing the writing. Some applications work like you want, but unfortunately many don't.

[stev067]

1 minute ago, jonathanm said:

Unfortunately Unraid has no control over what gets written first. It's totally controlled by the application doing the writing. Some applications work like you want, but unfortunately many don't.

Yeah I kinda figured it was a Krusader thing. I'm very new so that is the one I've heard recommended. Can you recommend another that works more like the way I'm describing?

[JonathanM]

6 hours ago, JorgeB said:

Windows explorer, it won't do that.

 

[stev067]

5 hours ago, FreeMan said:

Krusader will, most likely, copy the files far faster than even Win10 running in a VM, if speed is of any consideration.

Ok and is this true? @jonathanm

Edited May 14, 2021 by stev067

[JonathanM]

Depends on the source, destination, and transfer media speed. If you have valid active parity during the transfer that's pretty much the limiting factor.

[JorgeB]

If the external drive is mounted with UD you can even use Windows explorer on a networked computer and all the data will still be copied locally on the server, without using the network, at the same speed it would be with Krusader and without creating the empty folders.