Copying files from disk to user share


Recommended Posts

Using unRAID 6.0-beta14b, I copied all of my data (~1.4TB) from my existing NAS onto drive 1 of my unRAID server using rsync.  Now i would like to copy those files to the appropriate user shares; following the individual share settings of directory splitting, high-water, included and excluded disks, etc... is there a way to do this using the CLI (Midnight Commander), a script, or a plugin?  If possible, I would rather not mount the shares on an external workstation and copy all of the data across the network in order to properly fill the drives.  Thanks!

Link to comment

Sureguy: Are you talking about not moving from disk to user share with midnight commander? Does the bug occur if i were to mount two user shares and move the data over the network? (Even though it would be moving it from drive 1 to the user share)

 

Yes the same error will happen.

 

The problem is related to an issue with the way the file system is addressed, and is not specific to Midnight Commander or SMB or anything.

 

The issue is that the two file systems User and disk can address the same file differently, thus the copy can fail cause data loss when it attempts to copy and write to the same spot on the disk. Something that most copy commands like cp or rm normally prevent from happening.

 

But don't fret you can still spread your data to user shares, you just do it manually.

 

Say for example you create a share named "SuperShare" you can then create folders manually on disks and copy data into that folder, and the share will be aggregated.

 

You can address this location as /mnt/user/SuperShare (But that's really just the aggregate of all the /SuperShare folders of all of the disks with /SuperShare on them)

 

For example if you have two disks and files on both in that folder you could look at it like this.

 

/mnt/user/SuperShare (Aggregate of files on Disk 1 and Disk 2)
/mnt/disk1/SuperShare (Files in SuperShare that are actually on disk1
/mnt/disk2/SuperShare (Files in SuperShare that are actually on disk2

 

So what you should do is copy from /mnt/disk1/SuperShare (or whatever it's called now) to /mnt/disk2/SuperShare, and ignore /mnt/user/SuperShare for now. Mixing /mnt/user/ and /mnt/disk/ is what causes problems.

 

This needs to be stated more explicitly somewhere, so that you see it before you make a huge mistake....

Link to comment

Using unRAID 6.0-beta14b, I copied all of my data (~1.4TB) from my existing NAS onto drive 1 of my unRAID server using rsync.  Now i would like to copy those files to the appropriate user shares; following the individual share settings of directory splitting, high-water, included and excluded disks, etc... is there a way to do this using the CLI (Midnight Commander), a script, or a plugin?  If possible, I would rather not mount the shares on an external workstation and copy all of the data across the network in order to properly fill the drives.  Thanks!

Not sure if there is a good way to do this.

 

What MIGHT work is to copy file from /mnt/disk?? to /mnt/cache and then invoke mover to move the files back off the cache drive onto the array data disks.    However this is seems inefficient; the size of the cache drive is a constraint; and the existing files/folders on disk1 might confuse mover  into not using your split level/allocation method settings anyway.

 

I would instead recommend that you simply move files directly to where you intend them to end up (e.g. from /mnt/disk1 to /mnt/disk?.  This is the most efficient way to do things.

 

As was mentioned what you must NOT do is move files from /mnt/disk1 to anything under /mnt/user as this could easily lead to data loss.

Link to comment

If you're not copying from a component folder of the target user share folder then it's okay.

 

The user share copy bug is when you're copying from a component folder of a share to the share.

 

e.g. if you copied from \\Tower\disk3\MyShare  to  \\Tower\MyShare  then all the data you copied this way would be lost.

 

But if you're copying from, for example, \\Tower\disk1\StuffFromMyOldNAS  to a DIFFERENT share ... e.g. to \\Tower\MyNewShare  then it's fine.

 

The way most folks have encountered this problem is they want to remove a disk from a share;  so they change the share settings to some disk is no longer "Included" in the share ... say disk5 ... and then try to copy the data from that old disk to the share ==> e.g. from \\Tower\disk5\MyShare  to  \\Tower\MyShare    You DO NOT want to do this -- you'll lose all of the data you try to copy like that.

 

Link to comment

Thanks everyone. What you've said makes a lot of sense. Luckily I am not copying from disk to user share with the same path, as garycase mentions; as I've already copied quite a bit. I have been copying from /mnt/disk1/backup to /mnt/usr/movies. One question... if I did experience those issues during the move, would the copy error out, or would the files copy successfully, but become corrupt on the destination user share? I've copied a lot of movies in-between those two folders by mounting them via SMB on a windows box. I didn't receive errors, but i don't want to have to check every movie for corruption. Thanks!

Link to comment

I discovered this bug quite a while back and have tried to raise awareness so users did not loose data. This one can wipe out a folder of large files VERY QUICKLY!

 

The simple rule to not copy from disk to shares or vice versa is easy to remember and the forum knows. Explaining all the nuances and exceptions is not productive. It does not constrain users doing what they need to do.

 

If you copied your files to a disk in the FromOldNAS folder, a share called "FromOldNAS" would be created which would only be on one disk. Why then not just copy from the FromOldNAS user share to the movies user share? That is the best way to do it and avoid the bug. It avoids manually balancing the load and breaking the easy to remember rule.

 

Gary - would prefer you delete your post.

 

Thanks!

Link to comment

... The simple rule to not copy from disk to shares or vice versa is easy to remember and the forum knows

 

That is indeed a simple workaround that avoids the problem -- but it is NOT the actual problem.

 

 

... Explaining all the nuances and exceptions is not productive.

 

There aren't a lot of "nuances and exceptions" ==> there's ONE thing that causes this problem -- and nothing else.  The problem is, as I noted above, "... when you're copying from a component folder of a share to the share."

 

Anything else works fine -- copying from one share to another (as you just suggested); copying from a disk to a share (as long as the source isn't a component of that share);  or doing moves instead of copies with the same parameters.

 

I think UnRAID users are smart enough to understand the difference  :)

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.