October 4, 201015 yr I'm an unRAID newbie. Currently, I have one 1.5TB drive (OS + non-media files) and four 1TB drives in a 2.8TB RAID5 array. Both the drive and array are ~90% full--I have about 130GB free on the OS drive and about 150GB free on the array. I just got a new unRAID server (thanks Jomp!) and a new 2TB drive. I'd like to move everything in the array over to the unRAID server. Call it ~2.6TB of data. Once the array is empty, I'll move the four 1TB drives over the unRAID, and the larger drive will be the parity drive. I could buy another WD20EARS, but I'm really trying to keep expenses down. If I can figure out a sequence that works (I keep thinking of Apollo 13 ), a 4TB unRAID array will tide me over for at least half a year or so, and HD prices will drop in that time. Obviously my first problem is that I have an excess of 600GB or so of data preventing me from simply moving everything over to the 2TB drive, installing the 4 drives into unRAID, and firing up the unRAID array. I do have 2x120GB and 1x160GB in old SATA drives laying around. That plus the 130GB on the 1.5TB OS drive might get me close enough to temporarily offload stuff. If I'm able to do that, am I correct to think that unRAID will be ok with that? Can I add data to the 2TB drive (it'd be the only drive installed, so there'd be no parity initially), then move in the 4 1TB drives, make the 2TB drive into the parity, and unRAID will distribute the data w/o data loss? Or is that a pipedream? Thanks for any help
October 4, 201015 yr Author It distributes it among the drives of the array, doesn't it? Could I add all five drives without assigning a parity, then make the 2TB the parity drive, and let it handle shuffling the data? Or will that definitely not work?
October 4, 201015 yr Your options are limited Go through your old server an DELETE anything you don't need to migrate. You might be able to free up some space. You will need to network the two servers so the copies can be performed over the LAN. Then, Install the 2TB drive in the unRAID server. DO NOT ASSIGN IT JUST YET. Instead, Run the preclear_disk.sh script on it. The reason is simple. You'll be needing to trust it with a large amount of your data until you can establish parity protection again after breaking your existing array and you do not have enough disk space to have both arrays with parity protection. In the same way, if you can connect the two 120Gig drives and the old 160Gig drive to the unRAID array, connect and pre-clear them too. After you are sure the 2TB drive is working properly, assign it as a data drive in the unRAID array along with the other three smaller SATA drives. This will give you 2.4TB if all used as data drives. Assign no parity drive at this time. You are still faced with a problem. You have 2.6 Gig of data to move, and only 2.4Gig of space to put it in. You could free 1 1TB drive from your existing array and re-allocate it to the unRAID array. That would give you 3.4Gig of un-protected space in the new array and a degraded RAID5 array. Doing this will leave the RAID5 array in a degraded state... If you do not trust it, get a second 2TB drive and do not risk your data. Now, migrate all the data from the degraded RAID5 array to the unRAID array. Move as much as possible onto the smaller disks and as little as possible onto the 2TB drive. If you are lucky you can end up with the data something like this: 1TB drive full 120 Gig drive-A full 120 Gig drive-b full 160 Gig drive full 2TB drive 55% full (1200 Gig of data) Once all the data is on the unRAID array, migrate the 1.5 TB OS drive from the old server to the new. At this point you can still re-group by re-loading the OS drive on the old server and having it bring up the degraded array. Hopefully you'll not need to. Now, move all the data off of the 2TB drive onto the 1.5TB data drive. The 2TB drive will then be empty. Now, assign the 2TB drive as parity. Let it do a full parity calculation. Then, do a full parity check to make sure it is readable. Last move the other 1TB drives to the unRAID array. Pre-clear them as you will not want the extended down time if unRAID clears them for you. You will be at risk of a disk failure through this procedure from the time you remove a disk from your existing RAID5 array until you establish parity on the unRAID array. It will take many days... do not expect the entire migration to occur in an afternoon. Joe L.
October 4, 201015 yr It distributes it among the drives of the array, doesn't it? Could I add all five drives without assigning a parity, then make the 2TB the parity drive, and let it handle shuffling the data? Or will that definitely not work? It will not move data off of the new 2TB drive onto the others for you. Yes, it will allocate space based on the allocation rules via the user-shares, but, that does not help you unless you have a second 2TB drive to assign as parity.
October 4, 201015 yr Author So a drive has to start empty to be a parity drive? Oh well, that's gonna put a crimp in plans. I'll prob. have to buy another 2TB drive. But then I'm in same position, too move to move over until drives reallocated Thanks for your detailed reply, but one problem with your sequence--the old computer, with the RAID array isn't a "server"--it's my desktop. So the 1.5 TB drive needs to stay there. Everything else can go into the new server. I suppose I could downgrade to smaller drive for the desktop and move the 1.5 TB over, but I need a fair amount of local storage for documents, photos, 160GB of music (which frequently is written to, as I track play counts and update ratings accordingly), and as much space for Recorded TV as possible (which is also write-intensive, obviously). So with a 2nd TB drive, could I: Add both 2TB drives to the unRAID (no parity yet) Copy everything to both drives, filling one, leaving other about 65% free (~600GB of data written) Move 1 1TB drive from old to new. Allow RAID5 to temporarily continue in degraded state Copy 600 GB from 1 2TB to 1TB Make now empty 2TB drive into parity Take down RAID5 array, blank drives, add to unRAID That make sense? Will the RAID5 array be functional (but not protected) when one drive down? I'd be least protected in the period when I have a degraded RAID5 but haven't yet finished moving the data off the @TB and initialized it as the parity. I'm trying to figure out what the risk would be--if a drive failed in either the unRAID or the RAID5, the other should be backup, right? Am I right to conclude that it would take a failure of both to screw me? I can live with those odds, if so. If not, I suppose I could power down the degraded RAID5 array while waiting for the unRAID steps to finish. I could live without the media for a couple days--how long would you estimate for the dangerous parts (initializing 1TB drive, moving 600GB form one drive to the other inside unRAID array, initializing parity drive)? I appreciate the sanity check.
October 4, 201015 yr So a drive has to start empty to be a parity drive? No it does not need to be empty, but its contents will be completely over-written with parity calculations, therefore it is why I said you had to move your files off of it. You could just as easily copy your files off of it as long as you have the copies of the files on a different drive. Oh well, that's gonna put a crimp in plans. I'll prob. have to buy another 2TB drive. But then I'm in same position, too move to move over until drives reallocated Thanks for your detailed reply, but one problem with your sequence--the old computer, with the RAID array isn't a "server"--it's my desktop. So the 1.5 TB drive needs to stay there. Ok... so let's say you get a second 2TB drive. So with a 2nd TB drive, could I: Add both 2TB drives to the unRAID (no parity yet) Copy everything to both drives, filling one, leaving other about 65% free (~600GB of data written) At this point the new unRAID array has all of your data. It is still on the old desktop too. You are fairly safe except for fire, flood, tornado, hurricane. Move 1 1TB drive from old to new. Allow RAID5 to temporarily continue in degraded stateAll data is on the new unRAID array, so still reasonably safe. No single disk failure will cause data loss. Copy 600 GB from 1 2TB to 1TBAgain still safe. Old RAID5 is in degraded state, but you can still get to all its data on it AND all the data is on the unRAID array with part of it (600GB having 2 copies) Again, no need to delete the 600Gig from the 2TB drive as it will be over-written with the parity calculations. Make now empty 2TB drive into parity When parity calculations are complete you then must ALSO perform a parity "Check" So far you just wrote parity, you've not verified it can be read properly until you check it. Then you are sure the new disks and hardware are working properly. Take down RAID5 array, blank drives, add to unRAID.Now you will need to pre-clear the disks removed from the old server or else the unRAID server will do it for you with the server off-line. That will be many hours with the server off-line. Strongly suggest since you have all the data migrated and you can use the unRAID server to take your time and perform the step of pre-clearing the older drives. Now, once the additional drives are pre-cleared you can add them to the unRAID array by stopping it and assigning them to the array on the "Devices" page. when you next start the array you will be presented with a "Format" button to format them. Once you do they'll also be ready for your use. You'll then have a 2TB parity drive, a single 2TB data drive, and four 1TB data drives. A total of 6TB of data space. You can also add those smaller disks if you like if you have a PRO license. That make sense? Will the RAID5 array be functional (but not protected) when one drive down? I'd be least protected in the period when I have a degraded RAID5 but haven't yet finished moving the data off the @TB and initialized it as the parity. Protected in that a second copy of the data exists on the unRAID server. You have no parity protection on either at that point. I'm trying to figure out what the risk would be--if a drive failed in either the unRAID or the RAID5, the other should be backup, right? Am I right to conclude that it would take a failure of both to screw me? I can live with those odds, if so. It would take a simultaneous failure of one disk in each of the two servers to lose data. The RAID5 should run with one drive removed. If not, I suppose I could power down the degraded RAID5 array while waiting for the unRAID steps to finish. I could live without the media for a couple days--how long would you estimate for the dangerous parts (initializing 1TB drive, moving 600GB form one drive to the other inside unRAID array, initializing parity drive)? I appreciate the sanity check. There is no real dangerous part. You never lose either parity protection or complete mirroring of data at any point. (unless, of course, you get one of those pesky fires, floods, tornado's, avalanches, hurricanes, etc that take out both the server and your old PC.) Figure between about 50MB/s to 60MB/s when writing to the array over the LAN before you assign a parity drive. At 50 MB/s it will take 20 seconds to migrate 1Gig. You have 2600 GB to move across the LAN. That equates to 20 * 2600 seconds at a minimum. (52,000 seconds) That then works out to be 14.4 hours just to perform the data copy steps from the old PC to the new server. To pre-clear a 2TB drive takes about 35 hours. If you wanted to get that over with you can do it as soon as you get the server booted. No need to assign any data drives to perform that step. You WILL want to do it, as new drives are most likely to misbehave if not burned in. Parity calculation will take 5 to 7 hours with a 2TB parity disk. The subsequent parity check will take roughly the same amount of time. I know it sounds like a lengthy process, but you have a lot of data to move. Joe L.
October 4, 201015 yr Author Awesome, thanks. Sucks that I have to get another drive but at least I understand where I'm at now.
October 5, 201015 yr Author Also, I think I'll use the 160GB drive as a cache for now. Probably won't bother with the 2 120GBers--paltry size and they're not low energy.
October 16, 201015 yr Author Alright, I've run into a point of confusion to do with user shares. I added two 2TB disks, precleared them, and added them as data disks (no parity disk set yet.) The 160GB cache is formatting right now, but shouldn't play into my question anyway. Move 1 1TB drive from old to new. Allow RAID5 to temporarily continue in degraded state All data is on the new unRAID array, so still reasonably safe. No single disk failure will cause data loss. Copy 600 GB from 1 2TB to 1TB Again still safe. Old RAID5 is in degraded state, but you can still get to all its data on it AND all the data is on the unRAID array with part of it (600GB having 2 copies) Again, no need to delete the 600Gig from the 2TB drive as it will be over-written with the parity calculations. OK, so how do I set the share(s) so that my future parity drive has the minimum written to it? And then how do I copy that data over to the newly installed 1TB drive, so that I don't lose anything when I set the 2TB drive as parity? I'll probably start a new thread about this, but I might as well ask for advice in setting up my shares, to. I have: VIDEO Movies (1.07 TB) Movie 1 Disc 1 VIDEO_TS Movie 1 Disc 2 Movie 2 Disc 1 ... Concert DVDs (170 GB) Movie 1 Disc 1 VIDEO_TS Movie 1 Disc 2 Movie 2 Disc 1 ... To Watch [new stuff] (328 GB) Movie 1 Disc 1 VIDEO_TS Movie 1 Disc 2 Movie 2 Disc 1 ... To Watch - Documentaries (203 GB) Series 1 Movie 1 Disc 1 VIDEO_TS Movie 1 Disc 2 Movie 2 Disc 1 ... Series 2 ... TV (677 GB) Series 1 _Watched Season 1 Disc 1 VIDEO_TS Season 1 Disc 2 Season 2 Disc 1 ... Series 2 ... Those are the five main divisions I need to maintain: Movies, TV, To Watch (Movies and Docs), Concerts. Note that TV has the extra wrinkle that I move folders under "_Watched" as I make my way through a series. I'm amenable to moving folders around or splitting them into multiple shares, so long as I can keep these 5 categories distinct. Thanks!
October 16, 201015 yr Your future parity drive will be ENTIRELY written... No matter how many or how little files you have in the data drives. If it is a 2TB parity drive all 2TB will be written once you assign a parity drive and start the array. Parity has no concept of files, or file systems, or formatting, it is just bits... and every bit is calculated and stored. Joe L.
October 16, 201015 yr Author No, I get that. If you look above, what I'm dealing with is moving data over to the new unraid server, then moving disks over. To avoid the risk of data loss I need to copy data to the 2x2TB drives, move 1x1TB drive over to the unraid (leaving the RAID5 degraded), copy the data from one 2TB drive to the 1TB drive, then assign that 2TB drive as parity. Then move the rest of the 1TB drives over. So I'm wondering how to force most of the data over to one of the 2TB drives; and then how to copy the data off the other (future parity) 2TB over to the newly added 1TB drive, before assigning parity to the 2TB drive.
October 17, 201015 yr Set the disk shares (SMB) to export read/write. You will see disk1 and disk2 on the network. Copy most of the data to say disk1 then the rest to disk2. Well, split the data up in some logical manner but ensure that disk2 has less than 1T of data on it. Once you get a 1T drive installed in the server then you can copy the data on disk2 to it and re-assign the drives so that disk2 becomes the parity. Any top level directories you create will automatically become a user share with default settings. I can't say if they appear immediately or if you need to reboot but they do get recognized and added as a share. unRAID treats each top level directory as a user share by default if user shares are turned on. It might make life easier of you put each of those as a root folder on the disks (and therefore a seperate share) but then it depends on what kind of player you're using to access this data. For example, apparently a Popcorn Hour requires that single directory point like you have while some other software such as MediaBrowser could actually benefit from the multiple directories. Peter
October 17, 201015 yr Author OK, I see. So I first move the data to the separate disks manually, then deal with shares later (or continue with the auto generateds ones). From my Win7 box, I wasn't seeing any shared folders except flash at first, so I was confused. All the disks are there now, so I'll proceed. Thanks for the reply.
October 31, 201015 yr Author Now, move all the data off of the 2TB drive onto the 1.5TB data drive. The 2TB drive will then be empty. Now, assign the 2TB drive as parity. Let it do a full parity calculation. Then, do a full parity check to make sure it is readable. Finally got to this point. I have: disk 1 2TB disk 2 2TB disk 3 1TB Disk 1 and 3 have identical data on them. I want to make disk 1 into parity drive. I stopped the array, unassigned the drive form disk1, assigned it to parity, toggled back to Main config page and unRaid reports "Invalid configuration." and "Too many wrong and/or missing disks!" I didn't delete all the files off disk 1--could that be the problem? I figured adding it as a parity would wipe it automatically. Do I need to run an initconfig?
October 31, 201015 yr Now, move all the data off of the 2TB drive onto the 1.5TB data drive. The 2TB drive will then be empty. Now, assign the 2TB drive as parity. Let it do a full parity calculation. Then, do a full parity check to make sure it is readable. Do I need to run an initconfig? That is exactly what you need to do.
Archived
This topic is now archived and is closed to further replies.