When data is moved from the cache drive to the parity-protected array we currently have no way of knowing that the transfer was successful and that the data arrived without corruption.  I use TeraCopy to run CRC checks on every bit of data I transfer to my server, but any data that first goes to the cache drive is then at risk of corruption again.  Could a CRC or MD5 check be added to the mover script as a last step?  So the data would first be copied from the cache drive to the array, then CRC/MD5 checksums calculated and compared, and finally (if no errors are found) the data would be deleted from the cache drive.  If there is a checksum mismatch, then the user should be notified somehow (via the unRAID GUI I guess, though something a bit more proactive would be nice too) and the data should not be deleted from the cache drive.

I'd like to add my vote for that idea!

If you cannot trust your data when moving from one drive to another, There are bigger issues at play.

One beauty of rsync is that it only deletes the source file after successful copy.

If the prior file exists, it will do a checksum test to see what has changed and act accordingly.

 

Where teracopy works best is when there is a windows machine and network in the middle.

Even then you should not have to be worried about things like this.

 

In any case, it's possible to do an md5, then do the rsync, then do the md5 again and compare.

The mover would have to call a shell script instead of the rsync command.

 

All the effort of crc comparsion would be done in the shell script.

I use TeraCopy to run CRC checks on every bit of data I transfer to my server, but any data that first goes to the cache drive is then at risk of corruption again.

 

I am curious if you are seeing periodic corruption occur.

No, I've never seen corruption occur when transferring data from the cache drive to the array.  Then again, I've never checked for it.  It should show up in parity checks, right?

 

This just seemed to me to be something that would be fairly easy to implement.  However, if it isn't needed, that's fine too.  It just occurred to me that I take steps to make sure that my data reaches the server safely (by using TeraCopy) but that I take no steps to verify data integrity later on (besides the standard monthly parity check).

+1 for this.

 

Data corruption can occur regardless of whatever the underlying cause.  Although it's no comparison, it's a good example.  Recently at my office we discovered during the initial DR testing of an application that the backups being exported out of the database, while appearing to have no problems, were somehow being modified on the storage controller that we were using.  When we tried to restore the dump files, the database would error out.  sum would return different output for the same file on this one controller compared to a different controller.

 

Raj is right.  We take steps to make sure what we write to unraid is intact, it would be nice, and I think "inexpensive" for the array to make sure any movement of data within the grid was intact also.  Is it likely that a problem will happen?  No.  Is it impossible?  Also, no.

 

Just my 0.02c.

 

edited for typo

No, I've never seen corruption occur when transferring data from the cache drive to the array.  Then again, I've never checked for it.  It should show up in parity checks, right?

 

But how would you know that the data you have on the cache is valid to begin with? I think any problems that show up from copying from the cache to the array, would be when the mover script reads the data on the cache (hours after your did you teracopy crc check), and that is somehow corrupted. So the mover script would already calculate a bad checksum anyway... The most "dangerous" time is when the data is just sitting there on the cache... between both copies.

 

I think the safest way would be to generate a SFV file or something, right after you copy your data to the cache. That would be instantly verified against your local copy.

Later, the mover script would move the data to the array, and re-check the SFV for matches.

 

But that's just too paranoid...

That's a very good point Longinus, and that also seems like a reasonable addition to the mover script.

 

There's no such thing as too paranoid

No, I've never seen corruption occur when transferring data from the cache drive to the array.  Then again, I've never checked for it.  It should show up in parity checks, right?

 

This just seemed to me to be something that would be fairly easy to implement.  However, if it isn't needed, that's fine too.  It just occurred to me that I take steps to make sure that my data reaches the server safely (by using TeraCopy) but that I take no steps to verify data integrity later on (besides the standard monthly parity check).

 

You said that you are continuously verifying copies to your unRAID server.  I was wondering if those verifications had ever found a data verification error.  I realize you are not monitoring moving files from cache to array, so not asking about that.  If you have had verfiication errors, were you able to isolate the cause?

I'm curious too. How often does Teracopy detect a transfer error? Both Ethernet and TCP calculate checksums on the data as its transferred. Does adding a third layer of error detection make a difference?

I had a recurring problem with CRC errors using Teracopy, usually with large (>1GB) files. This was reproducible using Explorer to copy and then CRC'ing both sides. Neither demonstrated any error during the copy otherwise. In fact I would sometimes get different CRCs on different attempts (for the second copy of course). SMART checks passed so I did what I usually do when I get weird errors, check the RAM. Turns out the second stick of 2x1GB was bad. RMA'd it (yeah lifetime warranty!). After a few months the new memory began doing the same thing. Several runs with memtest showed non reproducible errors. This time I swapped the RAM with some other 1GB chips which memtested fine. No problems in the months since.

 

Interestingly I put that "bad" RAM in another computer where it tested OK after 8 hours on memtest. I know the timings were right in unRaid so I guess they just didn't like each other. 

RAM issues can be very tricky.

Just by helping friends/family with computer problems, I've seen a lot of crazy issues that could be traced back to bad RAM.

 

I wonder why RAM sticks don't have some form of SMART system. Some way that makes the system able to tell if they are bad.

Though Windows 7 has a memory test option now, so that helps...  since it's not so easy to ask your grandma to run a memtest86.

I have actually had Teracopy detect data validation errors from time to time, though I've never really been able to track down the cause of it.  There's a few I've come to expect:

 

1) When making backups of Macs and Windows computers, I generally just grab the entire user folder (contains desktop, docs, music, etc.).  The issue is that this folder also contains applications and some system files (generally hidden).  I've found that a lot of these application and system files will return CRC errors after a transfer.  I've chalked it up to the Windows computer not understanding the Mac files, or the Linux file system being incompatible with the Windows system files, or something like that.  As these files are never important from the backup standpoint, so I just delete them and move on.

 

2) I often fix computers for others, and this often involves backing up their data.  Sometimes I'll see a random mp3 file or some other non-critical thing throw a CRC error.  When looking deeper, they almost always are in a folder called 'Limewire' .  I delete the file and tell the user to stop using Limewire and learn to obtain their media from torrents like a civilized person.  Problem solved.

 

3) Very rarely (maybe once or twice) I've seen some of my data throw CRC errors when transferring it from my computer to my unRAID server.  I just delete the data on the server and copy it again, and on the second try it will be fine.  I haven't bothered to track down the source of that issue as it is very infrequent.

 

I certainly don't see CRC errors on a day-to-day basis, but I do see them probably once every month or two.  I probably see them more often than most as I often back up others' data to my server.

... since it's not so easy to ask your grandma to run a memtest86.

 

I recently had to ask my 77 year old mother-in-law over the phone to type CMD; then IPCONFIG to tell me information to help diagnose a router glitch.  She did it perfectly.  So I guess it would be possible....