[6.0-beta6] User Share Copy Bug

Here is what I want to do about this:  "If a disk share is being exported via any protocol, then that disk does not participate at all in the shfs union."

 

For example suppose we have this:

 

disk1/movies/A

disk2/movies/B

disk3/movies/C

 

Under Network in windows, currently you would see:

 

disk1

A

disk2

B

disk3

C

movies

A

B

C

 

ie, everything visible.

 

Assuming no config changes, after I make this change we have this:

 

disk1

A

disk2

B

disk3

C

movies

 

ie, all the shares are there but all the files in the movies share disappears because all the disks are still exported.  To fix this user would set 'export' to No on each disk, resulting in this:

 

movies

A

B

C

 

ie, disk shares not there of course, but their contents now show up under movies user share.  Now suppose user wanted to copy files off disk2.  First they would "export" disk2, resulting in this:

 

disk2

B

movies

A

C

 

Now disk2 is visible again, but its contents are no longer visible in the user share.  At this point user can use windows explorer and drag B to 'movies'.  shfs will put B either on disk1 or disk3.

 

This change prevents casual user from clobbering data.  User could still use console to clobber data

 

The other way to "fix" this is to change meaning of include/exclude masks to do the same thing.  At present these masks only affect which disks are eligible for new object creation.  I could change them so that they completely define (if set) what disks are used at all in shfs.  The problem with this though, is that it still doesn't prevent the casual user from clobbering data - they would have to know what these masks do, and carefully set them to accomplish what the want do do (move data off a disk and onto a user share, letting shfs decide where to put the files).

 

Thoughts?

Here is what I want to do about this:  "If a disk share is being exported via any protocol, then that disk does not participate at all in the shfs union."

...

Thoughts?

 

So this simply excludes the source disk from being part of the share, right?    Certainly seems like that would resolve this issue.

 

BTW, I put together an old system last night with 3 disks running v4.7 to test this, and will do so this weekend  (There was some discussion about this on a forum thread, and I said I'd provide a definitive answer about it when I had a chance) -- but if you happen to already know whether or not this is also a problem on 4.7, it would save me a bit of time.

 

I don't like this solution. Many users that use user shares use them only for reading, and manage their writes to the array by manually copying data to their disk of choice rather than using the various allocation methods and split levels.

 

This is what I do.

Actually, I do not believe that this fixes the problem. A user can still copy files using midnight commander or mv/cp on the server itself. in fact, I believe this is the the scenario in which data has been lost.

Actually, I do not believe that this fixes the problem. A user can still copy files using midnight commander or mv/cp on the server itself. in fact, I believe this is the the scenario in which data has been lost.

 

Yes this is why I say "casual user".  For someone transferring files via command line or MC it will be possible to do the wrong thing.  One idea to mitigate that is to put a dot in front of the mount point name if the share is not exported, eg:

 

/mnt/disk1

/mnt/.disk2

/mnt/disk3

/mnt/user

 

In this case if you type a 'ls /mnt' you would not "see" .disk2 mount point.  MC can be configured to now show hidden files.

 

But if someone is using the command line or MC they are not a casual user and they should be expected to "know" what they are doing.

I don't like this solution. Many users that use user shares use them only for reading, and manage their writes to the array by manually copying data to their disk of choice rather than using the various allocation methods and split levels.

 

This is what I do.

 

Right, there's that.  Two ways to address that:

1) if you want to write to a disk share, temporarily export it, write, then un-export it.  I can see issues with this.

2) define a 'read-only' mode for the user shares.  If in read-only then permit all disk shares to be exported.  Starting to get complex.

Your method would always cause all of the disk mounts to be unshared, right? Because every directory is a user share, every disk (except an empty one maybe) would be in at least one user share.

 

There are lots of other reasons to want to access the disk shares even if user shares are turned on. Some users backup their servers to a backup server a disk at a time. I am working on some scripts to maintain MD5s on a disk by disk basis so I could do an integrity check if evern a disk needed to be repaired. I would be very against trying to prohibit export of disk shares if user shares are turned on. I think it would cause some of the more advanced users to be upset.

 

"Know what they are doing" requires knowledge of the exposure. It would take some very good warnings to make people, even very strong Linux people, understand the risk here. And I believe they would expect it to be fixed, not just warning them. And mv/cp are not exactly complex Linux commands. I use them all of the time.

 

Here are a couple of options that come to mind ...

 

1 - Compare the inode of the source file with the inode of the selected destination file, and if they match fail the copy.

 

2 - If the copy/move is coming from a server disk, exclude that disk from the possible destination list. Sort of like a one-time exclude. So if copying from /mnt/disk2/Movies/batman.iso to /mnt/user/Movies, and the Movies user share included disk1, disk2, and disk3, exclude disk2 from the possible destinations. So the file would be copied to either disk1 or disk3 - duplicating the file. When you think about it, this is what the user wants to have happen in this situation.

The proposed new logic totally breaks how I use my server.

I organize my files read/write on the disk shares for speed.

I read some folders via the user shares for organizational consolidation.

 

Example I have to capture the streamed music to the fastest disk share, then reorganize it after tagging to the user share.

 

If someone was moving files from a disk share to a movie share, making the matching disk directories on the user share readonly would prevent them from being clobbered.

 

I.E. disabling writes to a specific disk that is participating in the user share.

 

The other option is to use rsync, which will copy to a .tmpname then move it into place. 

 

Is there some way the user share layer can know a file is being opened read and read/write and prevent the clobbering?

Maybe there's a way to turn on a no clobber switch temporarily if the file being read is the file that is going to be opened for writing.

 

Perhaps the user share, versions the file during the open for write (renames) and allows the open for write on a new file then removes the old file when it is closed.  This means for the purpose of the write, extra space is used. (as in rsync).

The issue I see is that a disk file and a user share file do not share the same device and inode, which is why cp allows clobbering the file.

 

root@unRAID:/mnt/disk1/Music/DJMUSIC# ls -l /mnt/user/Music/DJMUSIC/VirtualDJ\ Local\ Database\ v6.xml /mnt/disk1/Music/DJMUSIC/VirtualDJ\ Local\ Database\ v6.xml 
-rw-rw---- 1 rcotrone users 22487065 2012-09-01 16:42 /mnt/disk1/Music/DJMUSIC/VirtualDJ\ Local\ Database\ v6.xml
-rw-rw---- 1 rcotrone users 22487065 2012-09-01 16:42 /mnt/user/Music/DJMUSIC/VirtualDJ\ Local\ Database\ v6.xml

root@unRAID:/mnt/disk1/Music/DJMUSIC# stat /mnt/user/Music/DJMUSIC/VirtualDJ\ Local\ Database\ v6.xml /mnt/disk1/Music/DJMUSIC/VirtualDJ\ Local\ Database\ v6.xml       
  File: `/mnt/user/Music/DJMUSIC/VirtualDJ Local Database v6.xml'
  Size: 22487065        Blocks: 43970      IO Block: 131072 regular file
Device: 10h/16d Inode: 164905      Links: 1
Access: (0660/-rw-rw----)  Uid: ( 1000/rcotrone)   Gid: (  100/   users)
Access: 2013-03-20 12:19:03.000000000 -0400
Modify: 2012-09-01 16:42:03.000000000 -0400
Change: 2013-03-20 08:20:47.000000000 -0400

  File: `/mnt/disk1/Music/DJMUSIC/VirtualDJ Local Database v6.xml'
  Size: 22487065        Blocks: 43970      IO Block: 4096   regular file
Device: 901h/2305d      Inode: 147702      Links: 1
Access: (0660/-rw-rw----)  Uid: ( 1000/rcotrone)   Gid: (  100/   users)
Access: 2013-03-20 12:19:03.000000000 -0400
Modify: 2012-09-01 16:42:03.000000000 -0400
Change: 2013-03-20 08:20:47.000000000 -0400

I think it would cause some of the more advanced users to be upset.

 

WeeboTech, for example

 

2 - If the copy/move is coming from a server disk, exclude that disk from the possible destination list. Sort of like a one-time exclude. So if copying from /mnt/disk2/Movies/batman.iso to /mnt/user/Movies, and the Movies user share included disk1, disk2, and disk3, exclude disk2 from the possible destinations. So the file would be copied to either disk1 or disk3 - duplicating the file. When you think about it, this is what the user wants to have happen in this situation.

 

If the inode approach won't work, how about this one? Not quite as simple as what I laid out, though. Copying over top of an existing file bypasses the "select a disk" logic. Logic would have to be added to bypass this "always overwrite" model.

I think it would cause some of the more advanced users to be upset.

 

WeeboTech, for example

 

I too would be upset.  :'(

 

I use the disk shares to put files specifically where I want, and use the "user-shares" in a read-only mode for the media players.  Both are "shared" on the LAN, but the disk shares are not known to the media players.

 

There must be a better solution than excluding one if the other is being shared.  Besides, it does nothing to fix command line aw-shit's.

 

Joe L.

Clearly I misunderstood what Tom was suggesting.  I thought he was going to exclude a disk share from the user share IF the disk share was the source for a copy or move operation to the share.    Looking more closely at what he suggested (and reading the subsequent comments) it's clear that's not the case (and, on reflection, couldn't be, since UnRAID would have no way to knowing that a read of the disk share was being done by the same program that was writing to the user share).

 

The problem, in a nutshell, is that the user is copying a file from what is logically the same "drive" onto itself ... i.e. a copy from \\Tower\disk1\MyDVDs  to  \\Tower\MyDVDs  is effectively the same thing as copying a file from \\Tower\MyDVDs  to itself.    Most OS's will not allow this, or will automatically append a suffix to the filename to differentiate the two files.    But since the two references are different, the "copier" (whether it's a Windows, Mac, or Linux utility) doesn't realize that's what's happening.    ... and the user simply isn't thinking about that.

 

Clearly the issue is trivial to avoid -- if you want to move a file that's in a user share from one disk to another just do it using the disk share references.      But how to force that is an interesting problem, given that UnRAID has no control over what utility is being used to copy or move the files.    Logically, asking the question, "Is this filename currently open for reading on any disk belonging to this share?", whenever a file write attempt is made to a User share, would let UnRAID deny the write ... which would resolve the problem [A write to a disk share would still succeed].    ... or UnRAID could specifically exclude the disk that the file being read is on from the share when determining where to write the new file with the same name [Kind of an automated implementation of what Tom suggested above].

 

Clearly it gets complex -- but perhaps a variation of what Tom suggested would be a bit less so:  Don't exclude exported disks from a share;  but if any disk in a share is exported, change the share to read only.    But don't do this for disks set to "hidden".    And, to keep configuration simple for casual users, default the disks to "No" for export.    So a knowledgeable user who wants to be able to move his files around could set the disk shares to "Yes - Hidden", which would NOT trigger a read-only state on the share;  and could then move files from \\Tower\diskX\ShareName  to \\Tower\diskY\Sharename.  Since these are "knowledgeable users", they would KNOW not to copy from the disk share to the user share    [A WARNING could be posted on the User Share tab about this potential issue]

 

Casual users -- or even knowledgeable ones who don't care about writing to their User shares ... which many of us don't -- could export their disk shares and write to the individual disks as desired, while still having read-only access to the shares that works exactly as it does now.

 

As for this issue ...

 

Besides, it does nothing to fix command line aw-shit's.

 

Casual users have no business using command line parameters they don't understand !!    There will ALWAYS be plenty of ways to mess up if you use commands you don't understand.  [Or just mistype a command because you're in a hurry and it's 3:00AM ... MANY years ago I once typed "Del *.*" instead of "Dir *.*" -- needless to say the results weren't pretty.    Fortunately I had backups    ]

 

It seems we would be considering this as a fix and issue resolved?

 

This would not be ideal as leave it hanging about for all other access methods will shoot someone in the foot eventually. Midnight Commander has been mentioned but what about ajax files type tools or even XBMC. Docker opens up easy access to many tools and all of these have the potential of seeing this bug and we cant hope to educate every users to prevent this happening again.

 

Is there no possibility of a more fundamental fix that can be put in place?

 

FYI if we implemented this solution I would also need to change the logic of how I work in a very similar fashion to Joe L

FYI if we implemented this solution I would also need to change the logic of how I work in a very similar fashion to Joe L

 

Joe noted that he ...

 

I use the disk shares to put files specifically where I want, and use the "user-shares" in a read-only mode for the media players.  Both are "shared" on the LAN, but the disk shares are not known to the media players.

 

If the logic to make a share read-only if a component disk was exported was implemented, this would have NO impact on working like that.  (I work the same way -- writing virtually everything to the disk shares)    In fact, if it was implemented as I suggested, you could set the disk shares to "Hidden" and still work the same way, while also being able to write directly to the user share (although this would not keep you from doing the "bad" thing that could cause the problem).

 

Not sure how else this could be resolved, as the fundamental issue that's causing the problem is effectively copying/moving a file where the source and destination are the SAME place.    If you're using a tool that doesn't recognize that, it's hard to imagine how UnRAID could be "prescient" about what is going on.    If the User Share isn't allowed to be the target of the write (i.e. is made read-only), then this resolves this for virtually any copy tool.

 

 

 

I think we should keep these defect report discussions narrowly focused on the issue at hand.

 

To summarize so far ...

1. Tom's initial suggestion of removing disk shares for disks involved in user shares (which is every disk) was not acceptable because:

- it addresses the issue only for users accessing the user over the network, not for users accessing the server directly

- many users, including most of the moderators, use their servers in ways that this change would interfere with their normal usage patterns. variations (e.g., making user shares read only) may be viable although restrictive (for example, my media players write resume files to the disks, and making it read only would nullify that feature). But overall, this whole line of solution reasoning really fails as it doesn't address the non-network copying.

 

2. Detecting an overwriting of the same inode may not be technically possible due to the obfuscation that happens within the user shares

 

3. An option of detecting an attempt to copy a file from a disk share to a user share and omitting the disk share from the possible destination has been suggested and hopefully Tom will comment on whether that or something similar is possible.

I think if the issue is resolved for all over-the-network access, that's a reasonable fix.    Folks who access the server directly via Linux commands should be knowledgeable users ... and if not, we can't protect against everyone who wants to "tinker" under-the-hood.

 

The option noted in #3 should, as I noted earlier, resolve this and would, in fact, also work for local copies.    But it clearly requires some fairly extensive logic, which would be involved in EVERY write to a share ... Tom needs to evaluate whether this is a technically viable solution.

 

 

2. Detecting an overwriting of the same inode may not be technically possible due to the obfuscation that happens within the user shares

 

 

This 'may' be possible with file locks.

 

3. An option of detecting an attempt to copy a file from a disk share to a user share and omitting the disk share from the possible destination has been suggested and hopefully Tom will comment on whether that or something similar is possible.

 

 

If there was an option that could be turned on temporarily in that every new open for write would cause the user share to re-balance the destination, I.E. Write to a new location avoiding the current disk. Write to the new name and remove the old one. 

 

 

Frankly, the safest way to move files locally via console is with rsync as it creates a .tmpname before removing the old file.

...Folks who access the server directly via Linux commands should be knowledgeable users ... and if not, we can't protect against everyone who wants to "tinker" under-the-hood....

 

This completely ignores the whole docker development push. From a unRAID user standpoint there is no longer a difference between "over the network" and "local" so any real solution cant differentiate either.

 

I suppose that's true ... and we can't necessarily assume that Docker apps are "intelligent users" who will avoid the issue.

 

I think if the issue is resolved for all over-the-network access, that's a reasonable fix.    Folks who access the server directly via Linux commands should be knowledgeable users ... and if not, we can't protect against everyone who wants to "tinker" under-the-hood.

 

The option noted in #3 should, as I noted earlier, resolve this and would, in fact, also work for local copies.    But it clearly requires some fairly extensive logic, which would be involved in EVERY write to a share ... Tom needs to evaluate whether this is a technically viable solution.

 

I disagree with this from so many dimensions.

 

1 - Fixing the issue for over the network access comes with negatives for existing users. And it doesn't solve the problem. It is useless. Writes occur to user shares in my environment from the media boxes, but they provide a useful function to track resume positions. Having my user share be read only when my disk shares are available would not work for me.

 

2 - This is a bug. The comment "it clearly requires some fairly extensive logic" is something you could not possibly know and I think is BS. I have programmed for most of my career. This is a special case that needs to be coded for. My guess is that it would require Tom crawl into code he hasn't touched in a while. It is a PITA. I get it. But it is a bug and needs to be fixed the right way, and not with a poorly fitting band-aid that only covers part of the wound and leaves user data exposed ON PURPOSE.

 

If its not fixed completely then don't do any code changes. Clearly document and present the risk to all users as they upgrade or enable user shares, along with options and workarounds (like disabling disk shares). I would be disappointed if this is the solution, but less disappointed than a half-assed fix.

 

I suppose that's true ... and we can't necessarily assume that Docker apps are "intelligent users" who will avoid the issue.

 

I was about to post when this popped up. What is an "intelligent user"? There is only ONE way that a user would know to avoid this issue - reading MY posts on the forum. An unknowing "intelligent user" would go into user shares, remove a disk from the share, and then copy data from that disk share to the user share. Perfectly logical. Its what any of us would reasonably do. And we'd lose all our copied data. (I could easily imagine a customer doing this, losing his kids pictures, finding out LimeTech knew this and didn't warn him, and going after LimeTech for the data loss.) It is not intelligence that is needed to avoid the issue!

 

I suggest we let Tom chime in. Our banter is useless without his involvement.

 

 

There is only ONE way that a user would know to avoid this issue - reading MY posts on the forum. An unknowing "intelligent user" would go into user shares, remove a disk from the share, and then copy data from that disk share to the user share. Perfectly logical. Its what any of us would reasonably do. And we'd lose all our copied data. (I could easily imagine a customer doing this, losing his kids pictures, finding out LimeTech knew this and didn't warn him, and going after LimeTech for the data loss.) It is not intelligence that is needed to avoid the issue!

 

 

This could be resolved with not allowing implicit usershares to be created at top level directories. (by some option).

I.E. usershare configurations must be specifically set to the disks for inclusion into the usershare.

Thus when you take a disk out of a usershare.  FUSE/usershare no longer sees it.

 

 

This could also be resolved with a user renaming the directory first, then doing the move.

Either way a user needs to be educated.

 

 

With locks there may be a way to grab the pid of each process holding a file open, thus determining if an open read/open write with truncate is being done by the same process and catching the condition.

I would assume FUSE intercepts the open, and at that point can do the lookup if the open is a write that will truncate.

WeeboTech's suggestion to require specific inclusion and not automatically include disks based on top level folder names would work as long as the user specifically removed a disk from the share before doing any copies from it to the share.  But could still cause the same issue if a user simply copied from a disk share to the user share without removing the disk from the share first.

 

Perhaps the simplest "fix" would be to take a cue from Windows (and other OS's that handle it the same way) ... if a write (create) to a user share specifies a filename that already exists in the share, just append a suffix to the filename.    i.e. if I copy a file, say Test.Doc,  from a folder to the same folder in Windows, the name is automatically changed to Test - Copy.Doc.    If I copy it again, it will automatically be called Test - Copy (2).doc;  etc.    This would eliminate the data loss ... at the "cost" of renaming a few files after you had copied them and then deleted the originals.    And if you simply renamed the top level folder so it was no longer part of the share then the file wouldn't be renamed.

 

WeeboTech's suggestion to require specific inclusion and not automatically include disks based on top level folder names would work as long as the user specifically removed a disk from the share before doing any copies from it to the share.  But could still cause the same issue if a user simply copied from a disk share to the user share without removing the disk from the share first.

 

Perhaps the simplest "fix" would be to take a cue from Windows (and other OS's that handle it the same way) ... if a write (create) to a user share specifies a filename that already exists in the share, just append a suffix to the filename.    i.e. if I copy a file, say Test.Doc,  from a folder to the same folder in Windows, the name is automatically changed to Test - Copy.Doc.    If I copy it again, it will automatically be called Test - Copy (2).doc;  etc.    This would eliminate the data loss ... at the "cost" of renaming a few files after you had copied them and then deleted the originals.    And if you simply renamed the top level folder so it was no longer part of the share then the file wouldn't be renamed.

 

If I remember correctly, there was the topic of version'ing.  So the file could be versioned as the example defines.

But then how do you handle files that are updated incrementally. There's going to be continual versions.

Which means every write to update will require copying the file and writing to the new file.

 

It all depends on how version'ing is used. on VMS new files are made with filename;verison

however you can also set a limit on version, so anything over 5 versions will start removing the earlier versions so only the latest 5 versions are available.

 

I'm not educated on FUSE, so I can only speculate, that the intercepted open call can handle all the renaming and pruning.

 

WeeboTech's suggestion to require specific inclusion and not automatically include disks based on top level folder names would work as long as the user specifically removed a disk from the share before doing any copies from it to the share.  But could still cause the same issue if a user simply copied from a disk share to the user share without removing the disk from the share first.

 

Perhaps the simplest "fix" would be to take a cue from Windows (and other OS's that handle it the same way) ... if a write (create) to a user share specifies a filename that already exists in the share, just append a suffix to the filename.    i.e. if I copy a file, say Test.Doc,  from a folder to the same folder in Windows, the name is automatically changed to Test - Copy.Doc.    If I copy it again, it will automatically be called Test - Copy (2).doc;  etc.    This would eliminate the data loss ... at the "cost" of renaming a few files after you had copied them and then deleted the originals.    And if you simply renamed the top level folder so it was no longer part of the share then the file wouldn't be renamed.

 

Agree with point #1.

 

Point #2 - Knowing it is coming from a disk share and is going back to the same directory as the source would make this about the same level of difficulty as some of the options I listed. Not opposed to this though. But I am curious of the interplay with Windows. If you copied a file from Windows to a user share, it would prompt you whether to overwrite it or rename it. If you selected overwrite, but unRAID always renamed it, it would be confusing.