February 25, 201115 yr Hi there I was just wondering if anybody can tell me what the best way or program is to transfer like 6TB of data from my windows 7 computer to my unraid server. Right now im just copying threw windows 7 network, but thought there might be a faster way. Thanks
February 25, 201115 yr Author Yes i do have the parity drive unassigned. Its just im doing multiple transfers and thought there might be a better way than using windows. Thanks
February 25, 201115 yr Do the initial transfer with the parity drive unassigned. Peter But ONLY if you are using a copy program that uses checksums, otherwise, a bad sector on a disk and you'll create parity with a part of a file potentially missing. At least with parity enabled, when you go to read/play the movie, if a read-error on a data disk occurs unRAID can re-create the missing sector from parity in combination with the other data drives and re-write it to the data disk so the SMART firmware can re-allocate it. If you do not have parity enabled, that automatic correction cannot occur and the first time you'll learn of the error is after you initially create parity based on the un-readable sector. (The data is gone at that point if you did not use checksums when initially transferring it.) Yes, enabling parity initially makes the initial load of data slower... but it has its own disadvantage. self-correction of read-errors on data disks via SMART is not possible. Joe L.
February 25, 201115 yr Teracopy is a great free program that can be used to copy files and you can turn on checksum verification. it will make it slower. but better safe than sorry. i've been using it happily to move my files over
February 25, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes.
February 25, 201115 yr Yes i do have the parity drive unassigned. Its just im doing multiple transfers and thought there might be a better way than using windows. Thanks Multiple transfers from or to a single disk is not a good idea. It makes the disk thrash as it jumps between reading or writing the 2 different files. Do one transfer at a time. Peter
February 25, 201115 yr Is there a similar option on a Mac to do the copy with checksum? Or can you force a copy with checksum if you add the drive (not in the array) to the unRAID box, and copy from there?
February 25, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? I know it is slower, but how much slower? once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity?
February 25, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? Correct. I know it is slower, but how much slower? Typical write rates to the array with parity enabled are between 25 and 35 MB/s. once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity? Yes. Once parity is enabled your data is protected as soon as you copy it to the array... as long as you are NOT using a cache drive. If you are using a cache drive the data is only parity protected once it is moved from the cache drive to the protected array. The cache drive is almost useless when initially loading the array with large amounts of data as it will fill and then not get used until emptied. Joe L
February 26, 201115 yr Is there a similar option on a Mac to do the copy with checksum? Or can you force a copy with checksum if you add the drive (not in the array) to the unRAID box, and copy from there? Google "Mac Teracopy" I believe there is another program for mac.
February 26, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? Correct. I know it is slower, but how much slower? Typical write rates to the array with parity enabled are between 25 and 35 MB/s. once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity? Yes. Once parity is enabled your data is protected as soon as you copy it to the array... as long as you are NOT using a cache drive. If you are using a cache drive the data is only parity protected once it is moved from the cache drive to the protected array. The cache drive is almost useless when initially loading the array with large amounts of data as it will fill and then not get used until emptied. Joe L 25 to 35 MB/s? hmm when I was copying data using teracopy I was only getting like 8-10mb/s? is the 25+ only for MOVING data not copying? If you just move data to your server with parity enabled and using teracopy with CRC what happens to a file if it finds errors just doesn't get moved?
February 26, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? Correct. I know it is slower, but how much slower? Typical write rates to the array with parity enabled are between 25 and 35 MB/s. once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity? Yes. Once parity is enabled your data is protected as soon as you copy it to the array... as long as you are NOT using a cache drive. If you are using a cache drive the data is only parity protected once it is moved from the cache drive to the protected array. The cache drive is almost useless when initially loading the array with large amounts of data as it will fill and then not get used until emptied. Joe L 25 to 35 MB/s? hmm when I was copying data using teracopy I was only getting like 8-10mb/s? is the 25+ only for MOVING data not copying? If you just move data to your server with parity enabled and using teracopy with CRC what happens to a file if it finds errors just doesn't get moved? Do you mean 8-10MBps. Big B is bytes, little b is bits. If your getting 8-10mb/s there is definitely a problem. 8-10MB/s is normal for 100Mbps Ethernet and possibly WIFI.
February 26, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? Correct. I know it is slower, but how much slower? Typical write rates to the array with parity enabled are between 25 and 35 MB/s. once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity? Yes. Once parity is enabled your data is protected as soon as you copy it to the array... as long as you are NOT using a cache drive. If you are using a cache drive the data is only parity protected once it is moved from the cache drive to the protected array. The cache drive is almost useless when initially loading the array with large amounts of data as it will fill and then not get used until emptied. Joe L 25 to 35 MB/s? hmm when I was copying data using teracopy I was only getting like 8-10mb/s? is the 25+ only for MOVING data not copying? If you just move data to your server with parity enabled and using teracopy with CRC what happens to a file if it finds errors just doesn't get moved? Do you mean 8-10MBps. Big B is bytes, little b is bits. If your getting 8-10mb/s there is definitely a problem. 8-10MB/s is normal for 100Mbps Ethernet and possibly WIFI. it is 1000Mbps ethernet and not wifi. and yes I did mean 8-10MB/s
February 26, 201115 yr Check the connection speed of unRAID and your PC. It appears that one of them is connecting at 100MBps. Enter ifconfig eth0 to check unRAIDs speed.
February 26, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? Correct. I know it is slower, but how much slower? Typical write rates to the array with parity enabled are between 25 and 35 MB/s. once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity? Yes. Once parity is enabled your data is protected as soon as you copy it to the array... as long as you are NOT using a cache drive. If you are using a cache drive the data is only parity protected once it is moved from the cache drive to the protected array. The cache drive is almost useless when initially loading the array with large amounts of data as it will fill and then not get used until emptied. Joe L 25 to 35 MB/s? hmm when I was copying data using teracopy I was only getting like 8-10mb/s? is the 25+ only for MOVING data not copying? If you just move data to your server with parity enabled and using teracopy with CRC what happens to a file if it finds errors just doesn't get moved? Do you mean 8-10MBps. Big B is bytes, little b is bits. If your getting 8-10mb/s there is definitely a problem. 8-10MB/s is normal for 100Mbps Ethernet and possibly WIFI. it is 1000Mbps ethernet and not wifi. and yes I did mean 8-10MB/s Does the machine you are copying from have a gigbyte LAN adapter? going through a gigabyte router/switch. if you cannot copy off of your existing drive at a rate higher than that, enabling parity will not slow you down. I copied from my older IDE drive/PCI bus based server to my newer one at roughly 35MB/s, with parity enabled.
February 26, 201115 yr The answer is faster is not necessarily better. Enable parity. Use teracopy with checksums enabled. Slower, yes, safer, yes. If you do it this way do you not have to do a parity check after then? Correct. I know it is slower, but how much slower? Typical write rates to the array with parity enabled are between 25 and 35 MB/s. once you have parity enabled and all your data on your drive when you start transferring new data does it automatically use parity? Yes. Once parity is enabled your data is protected as soon as you copy it to the array... as long as you are NOT using a cache drive. If you are using a cache drive the data is only parity protected once it is moved from the cache drive to the protected array. The cache drive is almost useless when initially loading the array with large amounts of data as it will fill and then not get used until emptied. Joe L Now that you've gotten the conservative and safe way to transfer files, I can detail a faster method. CRC check requires that all written data be read back in order to calculate CRC. It will double the time if your using a cache drive. If not, then the read will be a lot faster than the write. I spent many days ripping DVDs to unRAID and it was would have been unbearable for me to double the transfer time. I used teracopy to calculate CRC for about 10% of my rips. They all checked out OK and I'm comfortable that the remaining 90% are good. Besides if I find a bad files later I can always re-rip it. If your data is really important you should maintain DVD copies of the originals and backup off-site. I use a cache drive but I don't delete the original files for at least 24 hours, so I'm certain the data has been moved into the array. I am still able to survive a single disk failure, including the cache drive, and my data is safe.
February 26, 201115 yr Jake, please start a new topic and give this one back to spencer. Removed my posts will start a new topic. Sorry spencer!
February 26, 201115 yr I have a slightly different transfer question relating to the initial transfer of tons of data to a new UnRAID server. I understand that the initial transfer goes a lot faster without parity calculation, at the expense of potential data corruption which could potentially then be incorporated into parity, when calculated - as has been stressed in previous posts in this thread. If, however, you are able to mount the drives (with the data to be transferred) in the UnRAID box (but not added to the array), would rsync be the safest way to make sure the transfer is uncorrupted? It seems like rsync automatically does a crc check after transferring files, no? Or, is there a better option?
February 26, 201115 yr I have a slightly different transfer question relating to the initial transfer of tons of data to a new UnRAID server. I understand that the initial transfer goes a lot faster without parity calculation, at the expense of potential data corruption which could potentially then be incorporated into parity, when calculated - as has been stressed in previous posts in this thread. If, however, you are able to mount the drives (with the data to be transferred) in the UnRAID box (but not added to the array), would rsync be the safest way to make sure the transfer is uncorrupted? It seems like rsync automatically does a crc check after transferring files, no? Or, is there a better option? That is what I would use in that situation.
February 26, 201115 yr And a final question (at least for this topic). You got me thinking about file corruption, probably too much, so what do cp and mv do? Do we have to worry about corruption when moving files around on disks when using those commands, or are there integrity checks built-in. I would hope the latter. I did move a few files over to the server using cp, and want to be sure I shouldn't repeat it with rsync.
February 27, 201115 yr And a final question (at least for this topic). You got me thinking about file corruption, probably too much, so what do cp and mv do? Do we have to worry about corruption when moving files around on disks when using those commands, or are there integrity checks built-in. I would hope the latter. I did move a few files over to the server using cp, and want to be sure I shouldn't repeat it with rsync. In all my years on unix/linux, I've never seen "cp" or "mv" mangle a file. They do not perform any checksum. You can mangle a file if they run out of space on the target file-system. Now, even with a checksum via terracopy or rsync you are not safe from corruption. They are great at detecting errors in transmission... They suffer from the same flaw. the disk buffer cache will supply the content read back unless they specifically disable it.. I suspect they write to the disk, it sits in the buffer cache. eventually gets written to the disk. If it is read back for a crc, and still in the buffer cache, the disk is not accessed at all. You really almost never read back what was written to the disk by reading back the physical disk.
February 27, 201115 yr Joe, as always, your answers contain a wealth of information. I really appreciate your taking time to comment in these threads. I knew I was waaay over-thinking this corruption thing, I mean logically you can't really have a copy command that mangles data, can you?
February 27, 201115 yr Joe, as always, your answers contain a wealth of information. I really appreciate your taking time to comment in these threads. I knew I was waaay over-thinking this corruption thing, I mean logically you can't really have a copy command that mangles data, can you? You sure can, if you have a flaky bit of memory, or a marginal power supply. It can happen, but you are likely to be unstable in many other ways too.
Archived
This topic is now archived and is closed to further replies.