(SOLVED) Copy/Move files from /mnt/user/SHARE - AND/OR - /mnt/diskX to /mnt/disks/UNASSIGNED_DISK

[FlorinB]

I know that withing the array you should not mix copying/moving files from disk to share. It should always be used disk to disk or share to share, due to a bug which is leading to data corruption.

 

Do we have the same issue with data corruption in the below copy/move situations?

1. /mnt/user/SHARE to /mnt/disks/UNASSIGNED_DISK

2. /mnt/diskX to /mnt/disks/UNASSIGNED_DISK

3. /mnt/disks/UNASSIGNED_DISK mount to /mnt/user/SHARE

4. /mnt/disks/UNASSIGNED_DISK /mnt/diskX

 

Note: Using the Plugin Unassigned Devices with Destructive Mode:Enabled it is possible to format and use the hot spare disks outside of the unRaid Array.

 

 

[JonathanM]

16 minutes ago, FlorinB said:

Do we have the same issue with data corruption in the below copy/move situations?

1. /mnt/user/SHARE to /mnt/disks/UNASSIGNED_DISK

2. /mnt/diskX to /mnt/disks/UNASSIGNED_DISK

3. /mnt/disks/UNASSIGNED_DISK mount to /mnt/user/SHARE

4. /mnt/disks/UNASSIGNED_DISK /mnt/diskX

No, because UD shares can not participate in user shares.

 

The root of this issue is that /mnt/user/share/folder/file.txt is the SAME FILE as /mnt/disk1/share/folder/file.txt, but it appears in two different paths. If you were to copy the contents of /mnt/user/share to /mnt/disk1/share then file.txt would be corrupted. /mnt/user is just a combined view of all the root folders in all the array /mnt/diskX(s) and /mnt/cache. /mnt/user0 is the same without /mnt/cache

[JorgeB]

There was already a thread about this today, but to answer your question, no problem since /mnt/disks is outside the user shares, user shares are on /mnt/user

[FlorinB]

mea culpa...i was searching yesterday and did not found too much.

 

Thank you johnnie.black.

Edited June 26, 2018 by FlorinB

[FlorinB]

3 hours ago, jonathanm said:

No, because UD shares can not participate in user shares.

Those are good news.

At this moment on my 4TB Array I have a lot of 500GB disks with 2 x 1.5TB for parity.

I am planning replace gradualy in time the 500GB with 4TB WD RED disks, but to add a 4TB disk into array i will actually need 3x4TB disks (2 for parity), which is impossible for me at this moment to buy at once. Since the Unassigned Devices Plugin is giving us the possibility to mount out of array disks, I will use the first and the second 4TB disks outside of the array, to backup the array or to save additional data. 

 

3 hours ago, jonathanm said:

The root of this issue is that /mnt/user/share/folder/file.txt is the SAME FILE as /mnt/disk1/share/folder/file.txt, but it appears in two different paths.

Clear enough and somehow expected behaviour. I would not name it a bug.

 

Thank you jonathanm ?

[SSD]

5 hours ago, jonathanm said:

The root of this issue is that /mnt/user/share/folder/file.txt is the SAME FILE as /mnt/disk1/share/folder/file.txt, but it appears in two different paths.

 

I don't agree with this. You could use symlinks to cause a file to be referred to by two different paths and the OS will see through it and refuse to copy it over top of itself.

 

The root of this issue is a bug in SHFS that subverts the OS protection mechanism against overwriting files.

[pwm]

1 minute ago, SSD said:

I don't agree with this. You could use symlinks to cause a file to be referred to by two different paths and the OS will see through it and refuse to copy it over top of itself.

 

When the application opens a file through a symlink, the file open command will directly follow the symlink and go to the final file. So while symlinks can be seen as aliases, they are fully transparent.

 

shfs, samba, nfs, direct access are four different mechanisms that introduces real aliases. And you have similar issues on Windows machines. You had similar issues on OS/2.

[SSD]

2 minutes ago, pwm said:

 

When the application opens a file through a symlink, the file open command will directly follow the symlink and go to the final file. So while symlinks can be seen as aliases, they are fully transparent.

 

shfs, samba, nfs, direct access are four different mechanisms that introduces real aliases. And you have similar issues on Windows machines. You had similar issues on OS/2.

 

Provide a reproducible set of commands in Windows that results in a file trying to overwrite itself when the copy command is used on a single machine.

 

The only reason you say symlinks are "transparent" is because the OS is aware of them and has the necessary logic to detect the situation.

[FlorinB]

37 minutes ago, SSD said:

Provide a reproducible set of commands in Windows that results in a file trying to overwrite itself when the copy command is used on a single machine.

Test 1 - try to copy file to itself

Quote

c:\test1>copy test_copy.txt c:\test1\test_copy.txt
The file cannot be copied onto itself.
        0 file(s) copied.

Test 2 - try to copy file to a symbolic link pointing to itself

Quote

c:\test1>copy test_copy.txt C:\test1\test_copy.txt.lnk
Overwrite C:\test1\test_copy.txt.lnk? (Yes/No/All): yes
        1 file(s) copied. -> result the symbolic link is replaced with the content of test_copy.txt, not pointing anymore as a symlink to the original file

Is this a real example when on Windows OS a symbolic link is altered? Yes.

What is happening in this situation? When the original file is changed, the symbolic link remains with the content of the file previously copied.

Is this a Windows filesystem bug?

Edited June 26, 2018 by FlorinB

[SSD]

3 hours ago, FlorinB said:

Test 1 - try to copy file to itself

 

Yes - that worked as I expected.

 

3 hours ago, FlorinB said:

Test 2 - try to copy file to a symbolic link pointing to itself

Is this a real example when on Windows OS a symbolic link is altered? Yes.

What is happening in this situation? When the original file is changed, the symbolic link remains with the content of the file previously copied.

 

I created a link to a file on my C drive in a different directory on my C drive.

 

The file was called "test.docx", and the shortcut was called "test.docx - Shortcut" in the GUI, and "test.docx - Shortcut.lnk" from the cmd prompt.

 

If I double click the shortcut in the GUI, it opens the file in Word.  Any changes I make are saved back to the original file.

 

If I type the shortcut at the cmd prompt, I get some gobblety gook but see the actual file name and folder name in there.

 

If I delete the shortcut in Windows, the shortcut goes away but the original file is unaffected.

 

If I delete the shortcut at cmd prompt, same thing. The shortcut is gone but the original file is unaffected.

 

If I drag and drop a different file over top of the shortcut in Windows GUI, Windows gives a null symbol and won't do it.

 

If I drag and drop the original file over top of the shortcut in Windows GUI, Windows gives a null symbol and won't do it.

 

(So Windows does not allow any drag and drop on top of a file shortcut. If it has been a directory shortcut, the drag and drop would have caused a file to get added to the linked subdirectory).

 

If I copy a file from the command prompt over top of the .lnk file, Windows asks if I want to overwrite the file. If I say yes, it overwrites the .lnk file. I can type the .lnk file and it is the content of the file I copied over it. But the shortcut disappears in Windows since it is not a valid .lnk file format.

 

So was I ever able to coerce a file to overwrite itself using links? No

Was I able to delete the real file using links? No

Was I able to confuse windows by overwriting the .lnk file? Yes. Did it cause any loss of data? No.

 

All in all, I can't say we have any sort of smoking gun like we are seeing in Linux with SHFS causing source files to be deleted.

 

3 hours ago, FlorinB said:

Is this a Windows filesystem bug?

I think not. The Windows filesystem is unaware of any significance of the .lnk files.

 

The Windows GUI attached special meaning to them, and overwriting them with junk is going to cause them to stop working as Shortcuts in Windows, but at the file system level, all is fine.

 

If we did the equivalent in Linux, you would never be able to overwrite the equivalent of a link file? Why? Because in Linux symlinks ARE maintained at the filesystem level. They are not files at all. I may do the equivalent of what we did here and post the results.

 

I didn't try manipulating the shortcuts in Samba, but pretty sure the results would be very similar to what we see in Windows. I may play with this also.

 

@FlorinB - thanks for trying this out and posting your results. I think you highlighted some odd behavior of the Windows shortcut feature. Not sure why the OS allows the .lnk file to be updated. Those .lnk files could simply be marked read-only and stopped that. But no real opportunity for data loss, so that is good news and what I expected.

[FlorinB]

Thank you for your answers. It was an interesting debate.

 

Coming back to the initial question: Copy/Move files from /mnt/user/SHARE - AND/OR - /mnt/diskX to /mnt/disks/UNASSIGNED_DISK it will work and I want to use this to:

- copy data from an external USB drive (it is faster than over the network in a home environment)

- make backups of the array (if the unassigned device disk is big enough)

 

 

[pwm]

On 6/27/2018 at 5:38 AM, SSD said:

If we did the equivalent in Linux, you would never be able to overwrite the equivalent of a link file? Why? Because in Linux symlinks ARE maintained at the filesystem level. They are not files at all. I may do the equivalent of what we did here and post the results.

 

Windows supports real symlinks too. The .lnk files are just ancient toys that M$ introduced because FAT file systems doesn't support real symlinks. However, NTFS does. It's one of multiple types of reparse points that Windows supports.