I have been building an unraid server for someone else, copying data from a number of USB drives to the array, checking parity, then removing the HDD from the USB and adding to the array. Everything was going great until I had some disk issues with my personal unraid server, which made me run SMART reports on all the drives in the new server - one fails.

 

I can't commit to buy any more drives at this stage (to replace the failing drive), and the server in question has ~11Tb free... so I looked at a number of instructions to reduce the array size however most seemed destructive (with a re-initialisation of config + re-calculate parity). I have since been using this as a guide:  http://lime-technology.com/wiki/index.php/Shrink_array but am finding that the transfer speed is often in KB/s - if the files can even be read at all.

 

I am now thinking that it's too late to follow that guide and wondering if there's a better way to handle this?

 

Can anyone suggest a better process - such as to remove one of the unused drives from the array and then use it to replace the failing drive (without losing data)? I assume at this point I am relying on parity so need to be very careful to avoid data loss?

 

First, if you don't have backups of this data, BACK IT UP !!    UnRAID is NOT a backup ... nor does it eliminate the need for backups.    Having a current backup is by far the best way to ensure that no matter what you do with the array, you'll have a way to restore any data that you may lose in the process.

 

As for reducing the size of the array ... the simplest approach is to simply re-initialize the configuration and rebuild party with the new config.  This WILL require "running at risk" (i.e. without parity) until the new parity is computed ... but you won't lose any of the data on the data disks assigned to the array.    If the disk(s) you want to remove from the array have data you want to keep, you can then connect them to another system (e.g. a PC) and copy their data to the newly constituted UnRAID array [You'll need the free LinuxReader to read the disks on a Windows PC].    As I noted above, if anything goes awry (e.g. one of those disks fail) you can simply copy the corrupted/missing data from your backups.

 

Thanks Gary. A few thoughts though:

 

  1)  Normal backup processes do not reasonably accommodate the volume of data that we are seeing with unraid these days without using a second unraid server (the unraid homepage states "supporting up to 92TB of protected storage" - how do normal people possibly backup such large amounts of data???). If unraid cannot provide a reliable storage mechanism, then why use it in the first place? If I need to 'mirror' data, then there are easier and cheaper ways...

 

  2) "...the simplest approach is to simply re-initialize the configuration and rebuild party with the new config... you won't lose any of the data on the data disks assigned to the array. If the disk(s) you want to remove from the array have data you want to keep, you can then connect them to another system (e.g. a PC)" - that's exactly my problem - I cannot read all of the data from the dying disk (to any OS). Right now it looks like I'm relying on parity for recovery of that info - which is destroyed if I follow the normal re-initialisation process. So what am I supposed to do?

 

  3) As per my post in the RC5 forum, how are we supposed to know that there's an issue with the array integrity without constant manual monitoring?

 

My apologies - it's just occurred to me that I might be able to 'backup' the at-risk data (i.e. that on the failing disk) via the user shares (instead of the disk share) to another location (not on the same unraid server). Still a lot of work to get the right data rather than the entire contents of the user shares (Tb's...), but certainly better than nothing if I can scrounge up the space.

 

This is in contradiction to the previously linked instruction of course, so my next question would be should I remove the disk from the array and re-start to ensure that parity is used? Otherwise, I assume the failing disk will continue to be used via the user share.

OK, well maybe not... Terracopy cannot even start a copy from the user share to an offline backup....

If the disk hasn't been red-balled, it's still being used. Unraid itself doesn't monitor smart status, and it only fails a drive when a write fails. If a read fails, it spins up all the drives, and attempts to write the parity generated data back to the disk that the read failed on, and if it succeeds, it increments the error counter, but doesn't red-ball the disk. This process is time consuming, especially if the failing drive takes many tries to fail the read.

 

If you choose to force a red-ball on the failing disk by removing it, you will immediately be trusting that all the other drives are in perfect shape, and they will all be spinning constantly for any data operations. If you have any other marginal drives, you will likely lose data on both drives, the one you pulled and the second drive that is failing. If you guess wrong and pull a good drive, you will lose data on the failed drive.

how do normal people possibly backup such large amounts of data???). If unraid cannot provide a reliable storage mechanism, then why use it in the first place? If I need to 'mirror' data, then there are easier and cheaper ways...

 

As far as mirroring the data being cheaper, you will have to enlighten me on this cheap way. I know of no cheaper storage per GB than spinning platter HDD, and the cheapest way to a fault tolerant super large array is still unraid as far as I know. Also, mirroring data in real time is NOT a backup, as any data loss by user or machine error is immediately transferred to the mirror. A second unraid array rsynced at regular intervals is the cheapest backup solution I can think of for a massive array.

I can't speak to your specific data reliability needs but a few thoughts to consider as you reevaluate your data storage plan:

 

- Data reliability needs are based on a number of factors such as:

  -- Your tolerance for down time (microseconds? hours? days?)

  -- Likely loss scenarios (drive failure? system failure? natural disaster? theft?)

  -- Is there a financial impact to the loss of data and how does it interact with above

 

Once you've thought about that then start thinking about how different classes of data interact with the above questions:

- What data is 100% irreplaceable and how sad / poor will you become because of it?

- What data is replacable but at large cost to resources (time and/or money)?

- What data is replaceable with ease / withouth concern to time?

- What data will not be missed at all and is only still on your server because you are a horder loath to intentionall delete anything?

 

And there are even more issues than above but it all comes down to these considerations:

- Unraid provides a level of protection for your data to survive a drive failure and even quickly and easily recover from a system failure unlike many hardware based RAID systems

- Unraid provides zero protection from accidental deletion, natural disasters (hard lightening strike / flood / fire), theft, etc

- Unraid offers minimal "uptime" protection during a single drive failure, but WILL require some minimum downtime to replace the failed drive and even longer to come back to full protected status during the parity rebuild

 

So:

- Is your most important data (like family photos, finances, etc) only stored on one drive (unraid), or at least two (unraid + PC)? 

- Are you mirroring between the two? 

- Do you know that you are hosed either way in case of natural disaster / theft? 

- If using mirroring do you understand you are hosed in the case of accidental deletion?

- Perhaps you need to copy that most important data onto a thumb drive stored in a firebox, a safe deposit box, or even mailed / ftp'ed to a distant geographic location (natural disasters can have a big footprint).

 

I'll stop ... but hopefully you see data reliability is not a simple issue even for a home user and it can be down right expensive and critical for a business!!!

A few more thoughts to r.e. Alex's post above ...

 

=> Business data is indeed often very critical ... and businesses are well aware of this and any major business has VERY robust backups in place.    A key example:  On 9/11/2001 there was NO data loss by any of the financial firms in the World Trade Centers !!  Now THAT is what I'd call good backups 

 

=>  Home users aren't likely to be quite that well backed up !!  I'm paranoid about backups, but still don't have off-site backups (a key feature for really important data) ... although I'm probably going to change that very soon by simply subscribing to Carbonite for one PC (where I'll keep all of financial, pictures, documents, etc. data).    But I AM much better backed up than most.  I don't maintain a 2nd UnRAID server; but I DO write everything that's going to be written to the array to a backup disk at the same time I write it to the array.  The backup disk is in an eSATA caddy;  and when it gets full, I simply label it (BackupXX ... where XX is the next number); and pop a new disk in the eSATA caddy.    I then put the disk in a WiebeTech DriveBox, and add it to my others in a fireproof/waterproof safe that's bolted to the concrete foundation in a closet on the other side of the house.    So I have a complete set of backups of both of my UnRAID arrays !!

 

My initial backup disks were 2TB units; but the latest are 4GB drives .. and yes, it does cost a bit to do that [but less than a 2nd UnRAID array].    But if you consider the cost/time/hassle of reconstituting all of the data you've got on your UnRAID array I think it's pretty cheap insurance.

 

My more critical but less voluminous data (pictures, financial records, documents, etc.) is backed up in several locations on my network -- basically everything is always on at least 5 different hard drives on 4 different systems, 2 of which are fault-tolerant !!  Except for the fact that it's all in my home (albeit in several different locations), it's VERY well backed up.  But I AM thinking of adding one-more "location" via Carbonite, just to have it all off-site as well.

 

Indeed, a single off-site backup is worth 1000 onsite ones

 

Think about the type of disasters that could befall a business: system failure, theft, fire, flood, small regionally destructive hurricane, large regionally destructive earth quake, WAR etc...

 

At what level of disaster do you stop worrying about recovering your data? 

 

A small one man shop probably won't care if he ever recovers his customer and inventory data if the small town her lives in is wiped off the map.

A mom and pop with maybe two or three locations w/in a state might be able to recover at least one of those shops if it is a neiboring town, but expand the disaster to an entire region and again their money might be better spent on insurance / cash reserves and not data protection.

A trans-national busniess in the finance industry or an Airline on the other hand could motor on even in the face of War if they still had access to their data

 

The point is ... know what REALY data protection looks like, know what the real risks are, and know what YOUR tolerance is.  Until you do that, you are suffering from something worse than no protection, you are suffering from FALSE protection.

Thanks guys. I guess I was frustrated from a number of recent issues with separate unraid servers, and ended up being a bit unfair in some comments.

 

I moved to unraid for fault tolerance, but hadn't really given DR (or user error) much thought. I appreciate you spending the time to reply, and you've raised some very valid points and given me a lot to think about.

 

As for shrinking the array, given that I can barely read any data off the failing drive I think I will red ball it so I can read from parity, then re-initialise the array and copy the data back.

 

My biggest (and recurring issue) seems to be failing disks, so once I get through the current issues I'll have to look into regular SMART reporting, such as these:

http://lime-technology.com/forum/index.php?topic=20822.0

http://lime-technology.com/forum/index.php?topic=3146.0

 

And of course, an appropriate backup/DR process