jsn0327 Posted April 2, 2015 Share Posted April 2, 2015 Using unRAID 6.0-beta14b, I copied all of my data (~1.4TB) from my existing NAS onto drive 1 of my unRAID server using rsync. Now i would like to copy those files to the appropriate user shares; following the individual share settings of directory splitting, high-water, included and excluded disks, etc... is there a way to do this using the CLI (Midnight Commander), a script, or a plugin? If possible, I would rather not mount the shares on an external workstation and copy all of the data across the network in order to properly fill the drives. Thanks! Quote Link to comment
sureguy Posted April 2, 2015 Share Posted April 2, 2015 Do NOT move from disks to user shares, there is a known bug where data can be deleted. Only copy from disk to disk. Quote Link to comment
jsn0327 Posted April 2, 2015 Author Share Posted April 2, 2015 Sureguy: Are you talking about not moving from disk to user share with midnight commander? Does the bug occur if i were to mount two user shares and move the data over the network? (Even though it would be moving it from drive 1 to the user share) Quote Link to comment
gundamguy Posted April 2, 2015 Share Posted April 2, 2015 Sureguy: Are you talking about not moving from disk to user share with midnight commander? Does the bug occur if i were to mount two user shares and move the data over the network? (Even though it would be moving it from drive 1 to the user share) Yes the same error will happen. The problem is related to an issue with the way the file system is addressed, and is not specific to Midnight Commander or SMB or anything. The issue is that the two file systems User and disk can address the same file differently, thus the copy can fail cause data loss when it attempts to copy and write to the same spot on the disk. Something that most copy commands like cp or rm normally prevent from happening. But don't fret you can still spread your data to user shares, you just do it manually. Say for example you create a share named "SuperShare" you can then create folders manually on disks and copy data into that folder, and the share will be aggregated. You can address this location as /mnt/user/SuperShare (But that's really just the aggregate of all the /SuperShare folders of all of the disks with /SuperShare on them) For example if you have two disks and files on both in that folder you could look at it like this. /mnt/user/SuperShare (Aggregate of files on Disk 1 and Disk 2) /mnt/disk1/SuperShare (Files in SuperShare that are actually on disk1 /mnt/disk2/SuperShare (Files in SuperShare that are actually on disk2 So what you should do is copy from /mnt/disk1/SuperShare (or whatever it's called now) to /mnt/disk2/SuperShare, and ignore /mnt/user/SuperShare for now. Mixing /mnt/user/ and /mnt/disk/ is what causes problems. This needs to be stated more explicitly somewhere, so that you see it before you make a huge mistake.... Quote Link to comment
itimpi Posted April 2, 2015 Share Posted April 2, 2015 Using unRAID 6.0-beta14b, I copied all of my data (~1.4TB) from my existing NAS onto drive 1 of my unRAID server using rsync. Now i would like to copy those files to the appropriate user shares; following the individual share settings of directory splitting, high-water, included and excluded disks, etc... is there a way to do this using the CLI (Midnight Commander), a script, or a plugin? If possible, I would rather not mount the shares on an external workstation and copy all of the data across the network in order to properly fill the drives. Thanks! Not sure if there is a good way to do this. What MIGHT work is to copy file from /mnt/disk?? to /mnt/cache and then invoke mover to move the files back off the cache drive onto the array data disks. However this is seems inefficient; the size of the cache drive is a constraint; and the existing files/folders on disk1 might confuse mover into not using your split level/allocation method settings anyway. I would instead recommend that you simply move files directly to where you intend them to end up (e.g. from /mnt/disk1 to /mnt/disk?. This is the most efficient way to do things. As was mentioned what you must NOT do is move files from /mnt/disk1 to anything under /mnt/user as this could easily lead to data loss. Quote Link to comment
garycase Posted April 2, 2015 Share Posted April 2, 2015 If you're not copying from a component folder of the target user share folder then it's okay. The user share copy bug is when you're copying from a component folder of a share to the share. e.g. if you copied from \\Tower\disk3\MyShare to \\Tower\MyShare then all the data you copied this way would be lost. But if you're copying from, for example, \\Tower\disk1\StuffFromMyOldNAS to a DIFFERENT share ... e.g. to \\Tower\MyNewShare then it's fine. The way most folks have encountered this problem is they want to remove a disk from a share; so they change the share settings to some disk is no longer "Included" in the share ... say disk5 ... and then try to copy the data from that old disk to the share ==> e.g. from \\Tower\disk5\MyShare to \\Tower\MyShare You DO NOT want to do this -- you'll lose all of the data you try to copy like that. Quote Link to comment
jsn0327 Posted April 2, 2015 Author Share Posted April 2, 2015 Thanks everyone. What you've said makes a lot of sense. Luckily I am not copying from disk to user share with the same path, as garycase mentions; as I've already copied quite a bit. I have been copying from /mnt/disk1/backup to /mnt/usr/movies. One question... if I did experience those issues during the move, would the copy error out, or would the files copy successfully, but become corrupt on the destination user share? I've copied a lot of movies in-between those two folders by mounting them via SMB on a windows box. I didn't receive errors, but i don't want to have to check every movie for corruption. Thanks! Quote Link to comment
garycase Posted April 2, 2015 Share Posted April 2, 2015 If you make a mistake and do what I noted you should not, the resulting file will have NO data ... i.e. it will be a zero-length file. That data doesn't get "corrupted" -- it's simply not there. You won't, however, get any error messages. Quote Link to comment
jsn0327 Posted April 2, 2015 Author Share Posted April 2, 2015 Ok thanks garycase. Quote Link to comment
SSD Posted April 2, 2015 Share Posted April 2, 2015 I discovered this bug quite a while back and have tried to raise awareness so users did not loose data. This one can wipe out a folder of large files VERY QUICKLY! The simple rule to not copy from disk to shares or vice versa is easy to remember and the forum knows. Explaining all the nuances and exceptions is not productive. It does not constrain users doing what they need to do. If you copied your files to a disk in the FromOldNAS folder, a share called "FromOldNAS" would be created which would only be on one disk. Why then not just copy from the FromOldNAS user share to the movies user share? That is the best way to do it and avoid the bug. It avoids manually balancing the load and breaking the easy to remember rule. Gary - would prefer you delete your post. Thanks! Quote Link to comment
jsn0327 Posted April 2, 2015 Author Share Posted April 2, 2015 bjp999: I will do that going forward. I was not aware of the bug when i spent days rsync'ing all of my data. Luckily, i don't think that i will be affected by the bug with my current folder structure. Thanks again everyone! Quote Link to comment
garycase Posted April 2, 2015 Share Posted April 2, 2015 ... The simple rule to not copy from disk to shares or vice versa is easy to remember and the forum knows That is indeed a simple workaround that avoids the problem -- but it is NOT the actual problem. ... Explaining all the nuances and exceptions is not productive. There aren't a lot of "nuances and exceptions" ==> there's ONE thing that causes this problem -- and nothing else. The problem is, as I noted above, "... when you're copying from a component folder of a share to the share." Anything else works fine -- copying from one share to another (as you just suggested); copying from a disk to a share (as long as the source isn't a component of that share); or doing moves instead of copies with the same parameters. I think UnRAID users are smart enough to understand the difference Quote Link to comment
jsn0327 Posted April 2, 2015 Author Share Posted April 2, 2015 I agree with garycase. I would keep this thread the way that it is. If someone has the same issue in the future, they need to know the different scenarios and that their data may not be lost. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.