User Share Copy Bug


111 posts in this topic Last Reply

Recommended Posts

Tom's symlink approach could resolve this nicely. Since the original file will be referenced directly to the original.

A caveat may be slower access to large user shares with lots of files.

 

Since a symlink is actually a type of file that references the original file, it has to be stated, opened, readlink, closed before any other filesystem operation.

 

There's a code example here.

http://linux.die.net/man/2/readlink

 

On the pro side, when traversing a user share with ftw or find as in cache_dirs, the original file and the usershare file(symlink) will be in the caches.

 

On top of that, if this reduces the NFS stale handle issue, that's two issued handled.

Link to post
  • Replies 110
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

The problem only occurs when you accidentally copy the file to itself.  (e.g something like /mnt/disk1/dir1/file1 to /mnt/user/dir1/file1).   In your example that is not happening so you do not encoun

The question is whether it is even a 'bug' in the strict sense of the word!   The word 'bug' is used in this context to mean that it is not the behaviour the user expects even though the system is beh

Posted Images

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

 

That is, on the Share Settings page, there are "global" Include/Exclude masks.  Say for example, you want to empty all the files from "disk2" to the user share file system.  What you would do is set "Excluded disks(s)" to "disk2" and click Apply.  Now it should work fine to copy everything off disk2 to a user share, letting shfs decide where to write things.

 

There's really no other way to prevent user from shooting himself in foot without giving up some other functionality.

 

I don't want to change the meaning of the share-level Include/Exclude masks, again proper documentation is required here as well.

Link to post

Tom =>  Does changing the global Exclude setting for a disk (e.g. disk2 in your example) work for this even if that disk (e.g. disk2) was previously part of the share you're copying to?    i.e. it has a top-level folder with the share's name?    I know that if you exclude the disk from the share (not at the global level) the problem still manifests itself.

 

I'd think the "safe" thing to do would be to not only add the disk to the global Exclude list; but to also rename the top-level folder on it ... but just curious if that last step is necessary.

 

Link to post
  • 2 weeks later...

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

Tom - I would encourage you to change the default behavior so that individual disks are not exported as Disk1, Disk2, etc. when the server is first set up.  Expert users who want those disks exported individually can still export them, but I would encourage newbies to use User shares exclusively. 

 

You could also add an option to the GUI when setting up User shares that automatically removes the individual disk shares.  Have this option enabled by default, and give the user a checkbox to clear this option and keep individual disk shares.  You can add warning text or a link to documentation on the pros and cons of having both shares active on the server.

 

I understand that none of this will protect a user who regularly works in a terminal window on the server, but it should protect 90+ percent of new, inexperienced users from ever being at risk.

Link to post
  • 5 months later...
  • 6 months later...

Just thought I'd bump this to mention that someone had some data lost to this issue today. See here.

 

And now again today...me. At least I think that is what has happened.

 

Is this for fucking real? Like seriously, I lost some <i>irreplaceable</i> files because of this known issue? Where is this mentioned in the wiki when discussing moving files?

 

So pissed off right now.

 

Small chunk of stuff that wasn't in the cloud yet...gone gone gone.

Link to post

Just thought I'd bump this to mention that someone had some data lost to this issue today. See here.

 

And now again today...me. At least I think that is what has happened.

 

Is this for fucking real? Like seriously, I lost some <i>irreplaceable</i> files because of this known issue? Where is this mentioned in the wiki when discussing moving files?

 

So pissed off right now.

 

Small chunk of stuff that wasn't in the cloud yet...gone gone gone.

Exactly what did you do that makes you think this happened?

 

Assuming reiserfs, click the Search link at the top left of the forum. Paste this into Search for:

reiserfsck –scan-whole-partition –rebuild-tree 

Maybe something for you in the search results.

Link to post
  • 2 weeks later...

Is this still a problem in version 6.1? If the bug itself hasn't been fixed, were the promised warnings ever implemented?

The default behavior in 6.1 is to not export any disks that are part of user shares. This prevents someone from accidentally encountering this bug over the network.
Link to post
  • 4 weeks later...

I just changed my settings to allow disk shares..  Let me see if I understand it...

 

Say I have a share called DVD.

Say I have a folder called MyMovie on Disk 1.

 

I want to move it to disk 2 (or some other disk) to free up disk 1.

 

So

cp /mnt/disk1/DVD/MyMovie    /mnt/disk2/DVD would work perfectly fine?

 

But if I did

 

cp /mnt/disk1/DVD/MyMovie    /mnt/user/DVD    would lose the whole MyMovie folder?

 

Now what if I exclude disk1 from the share DVD clicked apply  and then did

 

cp /mnt/disk1/DVD/MyMovie    /mnt/user/DVD

 

Would it then work fine and unraid would move the file to some other disk of it's choosing?

 

I routinly used to copy from one disk share to another disk share to free up some space and such.  I don't think I ever tried a disk to user share move/copy.

But I'd like to understand the limitation.

 

 

Link to post

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

 

That is, on the Share Settings page, there are "global" Include/Exclude masks.  Say for example, you want to empty all the files from "disk2" to the user share file system.  What you would do is set "Excluded disks(s)" to "disk2" and click Apply.  Now it should work fine to copy everything off disk2 to a user share, letting shfs decide where to write things.

 

There's really no other way to prevent user from shooting himself in foot without giving up some other functionality.

 

I don't want to change the meaning of the share-level Include/Exclude masks, again proper documentation is required here as well.

 

I just tried this and it did NOT work.  Maybe an array restart is required.. but this didn't work by just hitting apply. The file went into limbo as if I never excluded the disk.

It does work if you rename the source directory to something different first (and there is at least one disk that already has the share directory).

Note this was tried from the command line..  not from windows...  if that makes a difference.

 

Link to post

The symlink idea is not something that's going to get into -beta7 and probably not in 6.0-final.  Just too many other critical features to address.

 

What we're going to do is create some warning notice on the Share Settings page that talks about this issue: that it's not recommend to copy files directly from a disk share to user share UNLESS: the source disk is not configured to be part of user shares.

 

That is, on the Share Settings page, there are "global" Include/Exclude masks.  Say for example, you want to empty all the files from "disk2" to the user share file system.  What you would do is set "Excluded disks(s)" to "disk2" and click Apply.  Now it should work fine to copy everything off disk2 to a user share, letting shfs decide where to write things.

 

There's really no other way to prevent user from shooting himself in foot without giving up some other functionality.

 

I don't want to change the meaning of the share-level Include/Exclude masks, again proper documentation is required here as well.

 

I just tried this and it did NOT work.  Maybe an array restart is required.. but this didn't work by just hitting apply. The file went into limbo as if I never excluded the disk.

It does work if you rename the source directory to something different first (and there is at least one disk that already has the share directory).

Note this was tried from the command line..  not from windows...  if that makes a difference.

 

Nevermind...  I misread Tom's instructions.  I didn't use the global "don't use for shares"  I did the local exclude in the share settings.

Link to post
  • 1 month later...

I had another idea to partially mitigate this issue.

 

When a drive is removed from a user share, unRAID could automatically rename its user share folder (e.g., append "_X", so "Movies" would become "Movies_X"). unRAID will then create a user share for that new name, making it easy to then copy files from the removed share to the real share. This would mitigate a common (probably the most common) condition giving rise to the bug.

Link to post
  • 2 weeks later...

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

Link to post

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

Since the drive is not part of the array it cannot be affected by this. In fact, even an array drive that is excluded from user shares (Global Share Settings) cannot be affected by this.

 

If you read some of the explanation in the thread the problem occurs because Linux move/copy operations cannot tell that a path on an array disk and a path on a user share might ultimately be referring to the same drive.

Link to post

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

 

As trurl noted above, yes, you can do this without an issue, since the external drive is clearly NOT part of the share.

 

Note, however, that doing this over the network wouldn't necessarily be any slower => it's unlikely your drive can be written at a speed that exceeds a Gb network's transfer rate.

 

Link to post

Can I get a little clarification on this bug? I am looking to backup one of my user shares to an external USB drive I mount with the "unassigned devices" plugin via MC. Since the usb drive can not be part of the array (and therefore not part of the share) will I be safe to copy over a user share to the mounted USB disk? Its 3TB of data so I'd prefer to use MC rather than network to save time.

 

Thanks

Doing the entire process locally on unraid means you don't need another PC running while the copy is in progress, but keep in mind that it may actually go quicker over the network. I'm not sure if the USB speeds have improved on the latest builds, but in the past a locally attached USB drive was way slower than going over the network to a client PC with a good USB3 connection. I'll be interested to see if your experience is different.
Link to post
  • 1 year later...

I'll bump this thread back to life after almost two years cause I think I might have triggered a corner case of it... I have a unRAID server, with all sharing protocols (SMB, NFS, etc) DISABLED. The UserShares I defined have share minimum set to 100GB free space. Split level set to top or top-two. 

 

I'm using UnAssigned Disk to remote mount a SMB share of a different unRAID server and to a copy using mc to my user share. It looks like this...

 

Source: /mnt/disks/192.168.1.XXX_SourceShare

Target: /mnt/user/TargetShare

 

The copy keeps running out of space on the first disk in the UserShare. I exclude the disk and it makes no difference, still keeps trying to copy to it. I delete >100GB of files, still keeps trying to copy to it. It will not go on to the next disk in the UserShare until I set SpiltLevel back to default (disabled) 'split any directory'.

 

Using 6.4.0-rc9f

 

 

 

 

Link to post
11 minutes ago, Lev said:

I'll bump this thread back to life after almost two years cause I think I might have triggered a corner case of it... I have a unRAID server, with all sharing protocols (SMB, NFS, etc) DISABLED. The UserShares I defined have share minimum set to 100GB free space. Split level set to top or top-two. 

 

I'm using UnAssigned Disk to remote mount a SMB share of a different unRAID server and to a copy using mc to my user share. It looks like this...

 

Source: /mnt/disks/192.168.1.XXX_SourceShare

Target: /mnt/user/TargetShare

 

The copy keeps running out of space on the first disk in the UserShare. I exclude the disk and it makes no difference, still keeps trying to copy to it. I delete >100GB of files, still keeps trying to copy to it. It will not go on to the next disk in the UserShare until I set SpiltLevel back to default (disabled) 'split any directory'.

 

Using 6.4.0-rc9f

 

 

 

 

 

I'm struggling to see how your issue is connected to the user share copy bug. 

Link to post
3 hours ago, Lev said:

The copy keeps running out of space on the first disk in the UserShare. I exclude the disk and it makes no difference, still keeps trying to copy to it. I delete >100GB of files, still keeps trying to copy to it. It will not go on to the next disk in the UserShare until I set SpiltLevel back to default (disabled) 'split any directory'.

This is probably because the Split Level wins any contention for which disk to use.

Link to post
3 hours ago, itimpi said:

This is probably because the Split Level wins any contention for which disk to use.

 

This would definitely cause the problem you are seeing. Double check this setting. If you are deeper than the split level, unRAID knows which disk to pick and it will ignore the user share settings like min free even if disk has little or no space available.

Link to post
  • 5 months later...
2 minutes ago, jaybee said:

Can someone confirm if this defect is still unresolved?

Semantics aside, this is not a solvable issue. I wouldn't call it a defect, it's a byproduct of how the user share system works at a very low level.

 

Unless you thoroughly understand what is going on under the hood, don't mix user share and disk locations when moving data. Use either one or the other.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.