Hellomynameisleo Posted January 23, 2023 Share Posted January 23, 2023 If I was to use BTRFS on UNRAID and mount the usershare to my Windows PC as a drive letter, when using the Windows PC to copy any file/folder from the UNRAID mounted usershare to the Windows PC using a simple copy and paste or robocopy mirror command does UNRAID use BTRFS checksum verification to read the source file before copying over? Quote Link to comment
Hellomynameisleo Posted January 24, 2023 Author Share Posted January 24, 2023 18 minutes ago, JorgeB said: Yes. Thanks, will it also give an error and abord automatically if detected so it doesn't overide the good copy with the corrupted file? Quote Link to comment
JorgeB Posted January 24, 2023 Share Posted January 24, 2023 Correct, though sometimes Samba still allows the copy to go through, less likely to happen with the current release. Quote Link to comment
Hellomynameisleo Posted January 24, 2023 Author Share Posted January 24, 2023 9 hours ago, JorgeB said: Correct, though sometimes Samba still allows the copy to go through, less likely to happen with the current release. That doesn't sound good, is there a solution around this as I don't want my data to be sometimes corrupted. I know there's a data integrity plugin but have heard mostly bad things about it being very unstable/unreliable. Quote Link to comment
primeval_god Posted January 24, 2023 Share Posted January 24, 2023 10 hours ago, JorgeB said: Correct, though sometimes Samba still allows the copy to go through, less likely to happen with the current release. Has there been any change since the last time this issue was updated? Quote Link to comment
JorgeB Posted January 24, 2023 Share Posted January 24, 2023 30 minutes ago, Hellomynameisleo said: That doesn't sound good, is there a solution around this as I don't want my data to be sometimes corrupted You can monitor the btrfs stats, an error will be always logged there, so if it's detected but the file still goes though you'll be warned. 23 minutes ago, primeval_god said: Has there been any change since the last time this issue was updated? It's not as bad now, before a corrupt file would always go though, in the last tests I did last time with v6.11.5 most times you get an i/o error, but if you retry many times it can still go through. Quote Link to comment
Hellomynameisleo Posted January 24, 2023 Author Share Posted January 24, 2023 44 minutes ago, JorgeB said: You can monitor the btrfs stats, an error will be always logged there, so if it's detected but the file still goes though you'll be warned. It's not as bad now, before a corrupt file would always go though, in the last tests I did last time with v6.11.5 most times you get an i/o error, but if you retry many times it can still go through. Thats the problem, I want it to be robust and something I can comfortably rely on. What use if it overides the backup with the corrupted file and notifies me, I can't do anything then. Thanks for the information and help, hope this will change to be 100% reliable in the future. Quote Link to comment
JorgeB Posted January 25, 2023 Share Posted January 25, 2023 When I have some time I'm going to test with zfs on Unraid v6.12-beta to see if it's the same or not. Quote Link to comment
Hellomynameisleo Posted January 26, 2023 Author Share Posted January 26, 2023 On 1/25/2023 at 8:56 AM, JorgeB said: When I have some time I'm going to test with zfs on Unraid v6.12-beta to see if it's the same or not. can this be tested also with BTRFS please? Quote Link to comment
JorgeB Posted January 26, 2023 Share Posted January 26, 2023 With btrfs the file can go through, even when not using FUSE, so likely a Samba issue, you get an i/o when copying locally. Quote Link to comment
Hellomynameisleo Posted January 26, 2023 Author Share Posted January 26, 2023 (edited) 1 hour ago, JorgeB said: With btrfs the file can go through, even when not using FUSE, so likely a Samba issue, you get an i/o when copying locally. thanks but is there a work around this? The only thing I can think if is doing a full system scrub on BTRFS UNRAID before transfering but I've got lots of 50 TB of data so this wouldn't be ideal. Also does this issue also effect ZFS or BTRFS not on UNRAID when using SMB? Edited January 26, 2023 by Hellomynameisleo Quote Link to comment
JorgeB Posted January 27, 2023 Share Posted January 27, 2023 So this is the current status: Btrfs using a disk share, transfer stops for a few seconds when an error is detected, sometimes you get an i/o error, sometimes not, if you get an error and click retry it might copy after one or more tries, this is logged: Jan 27 16:47:53 Tower15 smbd[28944]: smb2_sendfile_send_data: sendfile failed for file test/1.iso (Input/output error) for client ptr=0x559485d0f570,id=0,addr=ipv4:192.168.1.109:53238. Terminating Jan 27 16:47:53 Tower15 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 426, gen 0 Jan 27 16:47:53 Tower15 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 257 off 476086272 csum 0xc8687774 expected csum 0x1e2ce107 mirror 1 Note the smb error with "Terminating" but it still goes through. Btrfs using a user share, transfer is normal without any slowdown, only the btrfs checksum error is logged: Jan 27 16:50:04 Tower15 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 430, gen 0 Jan 27 16:50:04 Tower15 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 257 off 476086272 csum 0xc8687774 expected csum 0x1e2ce107 mirror 1 Zfs using disk share or user share, transfer always aborts, at least it did the multiple times I've tried, only the SMB error is logged (but you can see the checksum errors and the affected file name if you run zpool stats -v) Jan 27 16:42:23 Tower15 smbd[25007]: smb2_sendfile_send_data: sendfile failed for file test/10_x64.iso (Input/output error) for client ptr=0x55d86e6f6d40,id=0,addr=ipv4:192.168.1.109:52866. Terminating Quote Link to comment
JorgeB Posted January 27, 2023 Share Posted January 27, 2023 20 hours ago, Hellomynameisleo said: Also does this issue also effect ZFS or BTRFS not on UNRAID when using SMB? Don't know, never tried, not easy for me to do, but if I have time someday I will test. Quote Link to comment
Hellomynameisleo Posted January 27, 2023 Author Share Posted January 27, 2023 2 hours ago, JorgeB said: So this is the current status: Btrfs using a disk share, transfer stops for a few seconds when an error is detected, sometimes you get an i/o error, sometimes not, if you get an error and click retry it might copy after one or more tries, this is logged: Jan 27 16:47:53 Tower15 smbd[28944]: smb2_sendfile_send_data: sendfile failed for file test/1.iso (Input/output error) for client ptr=0x559485d0f570,id=0,addr=ipv4:192.168.1.109:53238. Terminating Jan 27 16:47:53 Tower15 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 426, gen 0 Jan 27 16:47:53 Tower15 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 257 off 476086272 csum 0xc8687774 expected csum 0x1e2ce107 mirror 1 Note the smb error with "Terminating" but it still goes through. Btrfs using a user share, transfer is normal without any slowdown, only the btrfs checksum error is logged: Jan 27 16:50:04 Tower15 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 430, gen 0 Jan 27 16:50:04 Tower15 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 257 off 476086272 csum 0xc8687774 expected csum 0x1e2ce107 mirror 1 Zfs using disk share or user share, transfer always aborts, at least it did the multiple times I've tried, only the SMB error is logged (but you can see the checksum errors and the affected file name if you run zpool stats -v) Jan 27 16:42:23 Tower15 smbd[25007]: smb2_sendfile_send_data: sendfile failed for file test/10_x64.iso (Input/output error) for client ptr=0x55d86e6f6d40,id=0,addr=ipv4:192.168.1.109:52866. Terminating thanks for the test, what if you was to copy a file from the windows PC side instead of operating from unraid instead such as copying and pasting a file from the NAS user/disk share to the windows pc? Would this come out the same result? Quote Link to comment
JorgeB Posted January 27, 2023 Share Posted January 27, 2023 23 minutes ago, Hellomynameisleo said: what if you was to copy a file from the windows PC side This is what I did. Also note regarding your scenario On 1/24/2023 at 8:20 AM, Hellomynameisleo said: will it also give an error and abord automatically if detected so it doesn't overide the good copy with the corrupted file? This will still be a problem even if it was working correctly (or you start using zfs), i.e., it will ask to overwrite the existing file, if you say yes and then a checksum error is detected the copy will abort but the good existing file on Windows would be gone, though don't see a reason to replace existing files with the same ones. Quote Link to comment
Hellomynameisleo Posted January 27, 2023 Author Share Posted January 27, 2023 31 minutes ago, JorgeB said: This is what I did. Also note regarding your scenario This will still be a problem even if it was working correctly (or you start using zfs), i.e., it will ask to overwrite the existing file, if you say yes and then a checksum error is detected the copy will abort but the good existing file on Windows would be gone, though don't see a reason to replace existing files with the same ones. Will rclone be a good solution to this and when a corrupted file that stills go through? It has a --backup-dir option when syncing where any file that is going to to be ovewritten or deleted will simply get to put into a seperate backup directory instead. Seems like a good solution for all this Quote Link to comment
JorgeB Posted January 28, 2023 Share Posted January 28, 2023 That should work, I believe rclone is based on rsync? By default rsync and robocopy won't overwrite existing files with the same size/modification date. Quote Link to comment
Hellomynameisleo Posted January 28, 2023 Author Share Posted January 28, 2023 50 minutes ago, JorgeB said: That should work, I believe rclone is based on rsync? By default rsync and robocopy won't overwrite existing files with the same size/modification date. It seems similar but also can do cloud based storage syncing. rsync and rclone can have checksum checks option so it checks the hash of the source and destination when syncing Quote Link to comment
JorgeB Posted January 28, 2023 Share Posted January 28, 2023 17 minutes ago, Hellomynameisleo said: rsync and rclone can have checksum checks option so it checks the hash of the source and destination when syncing Yes, but that off by default, it will take a long time to transfer data with checksums enabled. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.