[6.0-beta6] User Share Copy Bug

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

 

That is, on the Share Settings page, there are "global" Include/Exclude masks.  Say for example, you want to empty all the files from "disk2" to the user share file system.  What you would do is set "Excluded disks(s)" to "disk2" and click Apply.  Now it should work fine to copy everything off disk2 to a user share, letting shfs decide where to write things.

 

There's really no other way to prevent user from shooting himself in foot without giving up some other functionality.

 

I don't want to change the meaning of the share-level Include/Exclude masks, again proper documentation is required here as well.

Tom =>  Does changing the global Exclude setting for a disk (e.g. disk2 in your example) work for this even if that disk (e.g. disk2) was previously part of the share you're copying to?    i.e. it has a top-level folder with the share's name?    I know that if you exclude the disk from the share (not at the global level) the problem still manifests itself.

 

I'd think the "safe" thing to do would be to not only add the disk to the global Exclude list; but to also rename the top-level folder on it ... but just curious if that last step is necessary.

 

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

Tom - I would encourage you to change the default behavior so that individual disks are not exported as Disk1, Disk2, etc. when the server is first set up.  Expert users who want those disks exported individually can still export them, but I would encourage newbies to use User shares exclusively. 

 

You could also add an option to the GUI when setting up User shares that automatically removes the individual disk shares.  Have this option enabled by default, and give the user a checkbox to clear this option and keep individual disk shares.  You can add warning text or a link to documentation on the pros and cons of having both shares active on the server.

 

I understand that none of this will protect a user who regularly works in a terminal window on the server, but it should protect 90+ percent of new, inexperienced users from ever being at risk.

Just thought I'd bump this to mention that someone had some data lost to this issue today. See here.

I also ran into this issue.

Luckily shortly after setting up my first server.

I still had the original files.

 

Since then I'm using an intermediate directory name which will

be renamed when all is done.

Just thought I'd bump this to mention that someone had some data lost to this issue today. See here.

 

And now again today...me. At least I think that is what has happened.

 

Is this for fucking real? Like seriously, I lost some <i>irreplaceable</i> files because of this known issue? Where is this mentioned in the wiki when discussing moving files?

 

So pissed off right now.

 

Small chunk of stuff that wasn't in the cloud yet...gone gone gone.

Just thought I'd bump this to mention that someone had some data lost to this issue today. See here.

 

And now again today...me. At least I think that is what has happened.

 

Is this for fucking real? Like seriously, I lost some <i>irreplaceable</i> files because of this known issue? Where is this mentioned in the wiki when discussing moving files?

 

So pissed off right now.

 

Small chunk of stuff that wasn't in the cloud yet...gone gone gone.

Exactly what did you do that makes you think this happened?

 

Assuming reiserfs, click the Search link at the top left of the forum. Paste this into Search for:

reiserfsck –scan-whole-partition –rebuild-tree 

Maybe something for you in the search results.

Is this still a problem in version 6.1? If the bug itself hasn't been fixed, were the promised warnings ever implemented?

 

Is this still a problem in version 6.1? If the bug itself hasn't been fixed, were the promised warnings ever implemented?

The default behavior in 6.1 is to not export any disks that are part of user shares. This prevents someone from accidentally encountering this bug over the network.

Thanks trurl. I've only ever exported user shares - a private one for each user plus a public one.

 

I just changed my settings to allow disk shares..  Let me see if I understand it...

 

Say I have a share called DVD.

Say I have a folder called MyMovie on Disk 1.

 

I want to move it to disk 2 (or some other disk) to free up disk 1.

 

So

cp /mnt/disk1/DVD/MyMovie    /mnt/disk2/DVD would work perfectly fine?

 

But if I did

 

cp /mnt/disk1/DVD/MyMovie    /mnt/user/DVD    would lose the whole MyMovie folder?

 

Now what if I exclude disk1 from the share DVD clicked apply  and then did

 

cp /mnt/disk1/DVD/MyMovie    /mnt/user/DVD

 

Would it then work fine and unraid would move the file to some other disk of it's choosing?

 

I routinly used to copy from one disk share to another disk share to free up some space and such.  I don't think I ever tried a disk to user share move/copy.

But I'd like to understand the limitation.

 

 

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

 

That is, on the Share Settings page, there are "global" Include/Exclude masks.  Say for example, you want to empty all the files from "disk2" to the user share file system.  What you would do is set "Excluded disks(s)" to "disk2" and click Apply.  Now it should work fine to copy everything off disk2 to a user share, letting shfs decide where to write things.

 

There's really no other way to prevent user from shooting himself in foot without giving up some other functionality.

 

I don't want to change the meaning of the share-level Include/Exclude masks, again proper documentation is required here as well.

 

I just tried this and it did NOT work.  Maybe an array restart is required.. but this didn't work by just hitting apply. The file went into limbo as if I never excluded the disk.

It does work if you rename the source directory to something different first (and there is at least one disk that already has the share directory).

Note this was tried from the command line..  not from windows...  if that makes a difference.

 

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

 

That is, on the Share Settings page, there are "global" Include/Exclude masks.  Say for example, you want to empty all the files from "disk2" to the user share file system.  What you would do is set "Excluded disks(s)" to "disk2" and click Apply.  Now it should work fine to copy everything off disk2 to a user share, letting shfs decide where to write things.

 

There's really no other way to prevent user from shooting himself in foot without giving up some other functionality.

 

I don't want to change the meaning of the share-level Include/Exclude masks, again proper documentation is required here as well.

 

I just tried this and it did NOT work.  Maybe an array restart is required.. but this didn't work by just hitting apply. The file went into limbo as if I never excluded the disk.

It does work if you rename the source directory to something different first (and there is at least one disk that already has the share directory).

Note this was tried from the command line..  not from windows...  if that makes a difference.

 

Nevermind...  I misread Tom's instructions.  I didn't use the global "don't use for shares"  I did the local exclude in the share settings.

I had another idea to partially mitigate this issue.

 

When a drive is removed from a user share, unRAID could automatically rename its user share folder (e.g., append "_X", so "Movies" would become "Movies_X"). unRAID will then create a user share for that new name, making it easy to then copy files from the removed share to the real share. This would mitigate a common (probably the most common) condition giving rise to the bug.

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

Since the drive is not part of the array it cannot be affected by this. In fact, even an array drive that is excluded from user shares (Global Share Settings) cannot be affected by this.

 

If you read some of the explanation in the thread the problem occurs because Linux move/copy operations cannot tell that a path on an array disk and a path on a user share might ultimately be referring to the same drive.

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

 

As trurl noted above, yes, you can do this without an issue, since the external drive is clearly NOT part of the share.

 

Note, however, that doing this over the network wouldn't necessarily be any slower => it's unlikely your drive can be written at a speed that exceeds a Gb network's transfer rate.

 

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

Doing the entire process locally on unraid means you don't need another PC running while the copy is in progress, but keep in mind that it may actually go quicker over the network. I'm not sure if the USB speeds have improved on the latest builds, but in the past a locally attached USB drive was way slower than going over the network to a client PC with a good USB3 connection. I'll be interested to see if your experience is different.

I'll bump this thread back to life after almost two years cause I think I might have triggered a corner case of it... I have a unRAID server, with all sharing protocols (SMB, NFS, etc) DISABLED. The UserShares I defined have share minimum set to 100GB free space. Split level set to top or top-two. 

 

I'm using UnAssigned Disk to remote mount a SMB share of a different unRAID server and to a copy using mc to my user share. It looks like this...

 

Source: /mnt/disks/192.168.1.XXX_SourceShare

Target: /mnt/user/TargetShare

 

The copy keeps running out of space on the first disk in the UserShare. I exclude the disk and it makes no difference, still keeps trying to copy to it. I delete >100GB of files, still keeps trying to copy to it. It will not go on to the next disk in the UserShare until I set SpiltLevel back to default (disabled) 'split any directory'.

 

Using 6.4.0-rc9f

 

 

 

 

11 minutes ago, Lev said:

I'll bump this thread back to life after almost two years cause I think I might have triggered a corner case of it... I have a unRAID server, with all sharing protocols (SMB, NFS, etc) DISABLED. The UserShares I defined have share minimum set to 100GB free space. Split level set to top or top-two. 

 

I'm using UnAssigned Disk to remote mount a SMB share of a different unRAID server and to a copy using mc to my user share. It looks like this...

 

Source: /mnt/disks/192.168.1.XXX_SourceShare

Target: /mnt/user/TargetShare

 

The copy keeps running out of space on the first disk in the UserShare. I exclude the disk and it makes no difference, still keeps trying to copy to it. I delete >100GB of files, still keeps trying to copy to it. It will not go on to the next disk in the UserShare until I set SpiltLevel back to default (disabled) 'split any directory'.

 

Using 6.4.0-rc9f

 

 

 

 

 

I'm struggling to see how your issue is connected to the user share copy bug. 

3 hours ago, Lev said:

The copy keeps running out of space on the first disk in the UserShare. I exclude the disk and it makes no difference, still keeps trying to copy to it. I delete >100GB of files, still keeps trying to copy to it. It will not go on to the next disk in the UserShare until I set SpiltLevel back to default (disabled) 'split any directory'.

This is probably because the Split Level wins any contention for which disk to use.

3 hours ago, itimpi said:

This is probably because the Split Level wins any contention for which disk to use.

 

This would definitely cause the problem you are seeing. Double check this setting. If you are deeper than the split level, unRAID knows which disk to pick and it will ignore the user share settings like min free even if disk has little or no space available.

Can someone confirm if this defect is still unresolved? 

 

Many thanks

2 minutes ago, jaybee said:

Can someone confirm if this defect is still unresolved?

Semantics aside, this is not a solvable issue. I wouldn't call it a defect, it's a byproduct of how the user share system works at a very low level.

 

Unless you thoroughly understand what is going on under the hood, don't mix user share and disk locations when moving data. Use either one or the other.

5 minutes ago, jaybee said:

Can someone confirm if this defect is still unresolved? 

Yes. Easy to avoid though. Don't mix user shares and disks when moving or copying. I recommend not sharing disks on the network at all. If you do need to work with disks, do so directly on the server. And when working with disks, only work with disks