Data transfer workflow

[jeff.lebowski]

Hi everyone. An unRAID user has been telling me about how great unRAID has been for him, but I resisted, thinking that the power usage from that many drives always spinning made no sense. So I implemented an eSATA dock and some internal hard drives. It's been pretty slick. Easy to use, but there's no protection. Bad news if something gets funky, and it has. I lost about 800gb of data, but managed to get it all back. Turns out the drives in an unRAID server are not always spinning.

 

So, I've got my parts on order (due on 10 Jan). The parts will build a 5 Drive Budget Box with an Icy Cool cage. The server will be placed on a shelf at about 6' from the ground, so minimizing taking it down is high on the list. Hot swap (server not running!) will be great.

 

My first issue I want to ask about is the process of actually moving data. Here are the drives I already have:

 

empty:

 

a) 2tb ears (JUMPER!!!)

 

in use (unfortunately, all are near full):

 

b) 2tb eads

c) 2tb eads

d) 2tb eads

e) 2tb ears (JUMPER!!!)

f) 1tb fals

g) 1tb fals

h) 1tb external

 

Between my four Windows machines, I have almost 2tb of free space. I'd like to setup a workflow for moving data to the new server.

 

1) Install drive a as parity - preclear and format

2) Empty drive b to the Windows machines and install as data a - preclear, parity sync, parity check, format

3) Copy data from Windows machines to data drive a over the gigabit network

4) Somehow confirm data is not corrupted (How exactly?)

5) Clear data from Windows machines (super scary)

6) Repeat steps 2, 3, 4, and 5 with drives c, d, e (JUMPER!!!), f, g, and h, NOT removing and installing the external drive into the server. I'm keeping that for future use.

 

How does that sound? I'd REALLY like to avoid buying a new drive, if at all possible.

 

Is the parity data automatically created as the data is transferred to a data drive, or is that a process I need to begin manually?

 

I assume the parity sync and check are easy to do from the server's page?

 

Regarding transfers, is the server a mapped drive, or is just a network location (IP address)?

 

I use Total Commander instead of Windows Explorer.

 

Looking forward to having my data protected, and sorry for the novel. >_>

 

[dgaschk]

You can install the parity drive after you have transferred data to the data drives. This will speed up writing to the server. You will be without parity protection but this is no worse than you are now. Install the parity drive and calculate parity once the initial data transfer is done.

 

Use teracopy (free) to copy your files. It can verify that the data is written correctly but verification will double your transfer time because it has to read all of the written data.

 

For audio and video files a bit error is not going to effect playback. Transfer errors are highly unlikely due to the checksums that TCP/IP uses. I used the verify option to spot check a few files during my transfer and never found an error. I did not verify everything because doubling the total time would have been unbearable.

 

 

[jeff.lebowski]

You can install the parity drive after you have transferred data to the data drives. This will speed up writing to the server. You will be without parity protection but this is no worse than you are now. Install the parity drive and calculate parity once the initial data transfer is done.

 

Use teracopy (free) to copy your files. It can verify that the data is written correctly but verification will double your transfer time because it has to read all of the written data.

 

For audio and video files a bit error is not going to effect playback. Transfer errors are highly unlikely due to the checksums that TCP/IP uses. I used the verify option to spot check a few files during my transfer and never found an error. I did not verify everything because doubling the total time would have been unbearable.

 

 

 

I think this is only possible for the first data drive. Once the data is transferred from the Windows machines to the server, I need to delete it from the Windows machines to make room for the next drive's data. Unless I'm misunderstanding?

[burtjr]

Since you are using re-purposed drives I will ask the question if any have been in a unit with a Gigabyte board. There is an HPA issue with some of the older boards that unRAID will give erroneous size of the disk. If not your in luck if so search for HPA and you should find plenty of posts on the subject.

[jeff.lebowski]

Since you are using re-purposed drives I will ask the question if any have been in a unit with a Gigabyte board. There is an HPA issue with some of the older boards that unRAID will give erroneous size of the disk. If not your in luck if so search for HPA and you should find plenty of posts on the subject.

 

This drive has only been installed in an ASUS board.