Today I replaced a 750GB drive with a 1TB one. During the upgrade the array stopped after 288 errors and the new drive has been marked red.

 

1.) Attached is a screenshot of the unraid menu.

 

2.) ZIPped and attached as well is the syslog. Here you will find lots of stripe errors. Don't know what this means.

 

3.) ZIPped and attached too is the result of smartctl of that drive. Here I can't find any errors.

 

I have no idea what happened. How come that smartctl has no errors but unraid reports them?

 

What should I do next?

 

Many thanks in advance.

Harald

 

This may be a tough one.  The exception Emask errors are not being very helpful, primarily reporting things like 'device error' and DF (Device Fault) and ERR and ABRT, very generic error indicators.

 

The first series of errors actually involved Disk 15, an almost identical Samsung (slightly older firmware).  Boot up was about 7:55pm, first exception Emask error sequence was around 8:17pm, then again at 8:25pm, 8:45pm, and 9:13pm, and these Disk 15 errors would probably have continued on sporadically, except that errors with Disk 11 stopped everything at 9:39pm.  The Disk 15 errors are of the BadCRC and ATA Bus error type, which I normally associate with bad or loose cables, connectors, badly seated drives in trays, or a bad controller.  In other words, the problem is *between* the system and the drive.  When installing/uninstalling drives, it is easy to knock SATA cables a little loose, so that would be the first thing I check.

 

Unfortunately the exception Emask error sequences for Disk 11 are the unhelpful 'device error' and DF and ERR and ABRT.  There are no CRC or bus-related errors.  They don't appear to disable the drive, but all of the Write errors occur subsequently, and the Sync is aborted.  This drive is not on the same controller either.  The clean SMART report seems to indicate that this was not a physical drive problem, so I don't know what to recommend.  It is possible that the drive is bad, or there is a bug in its newer firmware (1AA01113), or there was a brief power problem, or ...  The only other unique item I can mention is that this was a drive rebuild, all of the 15 other drives were being read, and this drive was being written.  That's a lot of simultaneous I/O.

 

As to your next step, you may wish to consider re-installing the previous Disk 11, and using this procedure to restore validity to parity:  Make unRAID Trust the Parity Drive, Avoid Rebuilding Parity Unnecessarily.  Then try again...  Perhaps a different terabyte drive will succeed?  I can see why you wanted to do this, all 15 large data drives almost completely full!

 

I know this is not helpful, but I cannot say whether this is a drive, cable, controller, power, or OS problem, or just a strange coincidence of unknown factors.  This does not look like a high level software issue, much more like a hardware issue or something very close to the hardware, like a low-level driver or firmware or bus issue.  The SMART report certainly won't help you RMA it.  Hopefully someone else will have an idea.

RobJ just about covered everything from the syslog perspective.

 

The smart report looks pretty clean, but a few minor things to keep an eye on.  The Multi_Zone_Error_Rate is 0 on every drive that I have and have looked at, but this has a value of 2.  Vendors can choose to store information differently, and Samsung may use this attribute differently than other manufacturers, so it is hard to know if this is a problem.  There are a few attributes that have non-zero values (#183 and #184), but I am unable to determine what these attributes are - I haven't seen them used on my drives.

 

The smartctl report mentions some special flags (-F samsung or -F samsung2) that you might consider trying.  I have no idea what they do, but might be worth trying.

 

In general, though, the smartctl seems pretty clean.

 

With the write errors showing up at the OS level but the drive thinking everything is hunky dory, I certainly concur with RobJ thinking that something in between is the issue.  I would certainly check the cabling.  So many times I've found replacing (or even just unplugging and replugging) a cable fixes things like this.

 

RobJ - I left you a PM.  (Thanks)

RobJ and bjp999,

 

many thanks for your detailled posts. This unraid community forms a very good and informed user support. Let me thank you all.

 

Drive 15 may indeed have a problem. It's an external drive attached to the eSATA port of the MD1500/LL machine. When I attached this drive several months ago I couldn't find a matching drive bay or adapter. So I bought an external adapter that didn't fit perfectly - but it worked. Just a minute ago I searched my local dealer for an external eSATA drive bay.

 

So I will put the old 750GB (drive 11) back again. I will replace the cables/connections to drive 15 and issue a "Restore" as found in the link of RobJs post. That way I hopefully will get the old system back to normal operation (after the parity sync).

 

If this works I will retry to change the disks again. A new 1TB is on it's way if the other new one has an error.

 

Many thanks.

Harald

 

 

P.S.: Seems that I need a new LimeTech system 8-(

 

 

I have an external eSATA enclosure that I use from time to time - in fact I have a couple of them.  The eSATA cables that came with the different brands are of different lengths - one about twice as long as the other.  I have noticed that the shorter one works reliably and the longer one causes occasional errors.  I therefore stick with the shorter one.  Could this be the problem?

bjp999,

 

today I bought a new enclosure for my external drive connected to the eSATA port http://www.conrad.de/goto.php?artikel=411193. The cables seem to be ok.

 

Currently parity check is in progress (38,5%). After this is done I will retry the disk change.

 

I had a lot of problems with SATA so far. Adapter not working properly and cables not fitting well are on the list always. Last time I had so many problems with interfaces the beast was called SCART. I heard it was invented by french ;-)

 

Thanks

Harald

 

RobJ and bjp999,

 

my old system is back to normal operation again. Many thanks to you.

 

I now have that partially written 1TB drive that I would like to use to replace one of my 750GB drives. What are the desired steps now?

 

If I put the drive in how can I force a "shut up and try to fill that drive with the data from the older drive". I assume that the 1TB has been formatted during my first try. But it is not filled completely.

 

Thanks in advance.

Harald

 

RobJ and bjp999,

 

my old system is back to normal operation again. Many thanks to you.

 

I now have that partially written 1TB drive that I would like to use to replace one of my 750GB drives. What are the desired steps now?

 

If I put the drive in how can I force a "shut up and try to fill that drive with the data from the older drive". I assume that the 1TB has been formatted during my first try. But it is not filled completely.

 

Thanks in advance.

Harald

 

I recommend you download and use the "preclear_disk.sh" utility I write to write zeros to the disk while not assigned to your array, and then, once cleared, add it back to your array.

 

The pre-clear step will take hours, but your array will be available and on-line while it is occurring.  Once pre-cleared, the steps of stopping the array, assigning it to a slot in the array, re-starting and formatting will only take a few minutes.

 

You can find the utility in an attachment in this thread: http://lime-technology.com/forum/index.php?topic=2696.0

 

At least the pre-clear process will give you some confidence the new disk can be written to reliably. 

 

Joe L.

JoeL,

 

thanks again. This morning I read about your tool here in the forums. But I'm not 100% clear what this means to me. Any unassigned drive out of an working array can not lead to a functional array. What Do I miss?

 

I pull out the old drive and put in the partially written disk. I do not start the array. I unassign the new drive and let it be cleared.

 

During the process my array is not started and not working.

 

Hmm???

 

Thanks

Harald

 

JoeL,

 

thanks again. This morning I read about your tool here in the forums. But I'm not 100% clear what this means to me. Any unassigned drive out of an working array can not lead to a functional array. What Do I miss?

 

I pull out the old drive and put in the partially written disk. I do not start the array. I unassign the new drive and let it be cleared.

 

During the process my array is not started and not working.

 

Hmm???

 

Thanks

 

Harald

 

It helps only if you have a spare slot (not already assigned in the array) in which to do the pre-clear.  if you do not, then it will not work for you.  Sorry for introducing any confusion.

 

The idea is to do the pre-clear while your array is on-line and still available to play movies, etc.  The clearing process otherwise would take several hours during which your array would be off-line...  The idea of the pre-clear when NOT a part of the arrayis to keep the off-line time  of your array to a minimum.

 

Joe L.

 

 

I now have that partially written 1TB drive that I would like to use to replace one of my 750GB drives. What are the desired steps now?

When rebuilding data from parity and the remaining data drives the target drive is not formatted, since the data copied from the existing drive also holds the formatting.

 

To proceed is easy.  Assuming your array is up and running and you have successfully run a full parity check.

(I am assuming your existing parity drive is also 1T or bigger)

 

Stop the array

Power down

replace the existing 750 Gig drive with the new 1T drive. 

power up

check the "I'm sure" under the start button, then press "Start"

 

The data will be written to the 1TB drive as reconstructed from the other drives in your array.  This is exactly the same procedure as if you had a drive failure and went to replace it.

 

It is illustrated here:

http://lime-technology.com/forum/index.php?topic=1553.msg10540#msg10540

 

Remember... Use the "Start" button....  Do NOT use the button labeled "restore", it does not restore data, it deletes your existing configuration and stores a new one based on the currently assigned and working disks.  It forces any existing parity data to be invalid and forces a complete parity rebuild.    The button labeled "restore" would NOT rebuild a failed (or upgraded) drive. 

 

In your case, if you wiped out your parity by pressing "Restore" you still have a copy of the data on the 750Gig drive you removed, but if it had failed and you were replacing it, its data would be lost.  See here: http://lime-technology.com/forum/index.php?topic=1703.msg11855#msg11855

 

Joe L.

JoeL,

 

the first upgrade process stopped after some errors right in the middle and left this new disk in a bad state. So after bringing the system back to normal operation (with the old disk back in and a successful parity check) I was ready for a second try. Again I would move the 750GB out and put the 1TB in. But now this 1TB is no longer empty it is partially filled instead. The disk is simply in a worst state.

 

For me it was not clear what unraid would recognize if I would use this partially written disk again. Is this disk formatted and new or is this disk formatted and partially filled or would unraid find out that this disk is in a bad state.

 

So my idea was to not rely on unraid and wipe the disk before using it. The idea was to put the new disk in. Keep the array stopped and format the disk (however this may work). But this format should not touch the parity disk at all!!

 

By reading nearly the complete board this morning I doubt that this is possible. Perhaps unmenu could had helped here - but I didn't want to open up a second case during my problem.

 

I think, with the second unraid box just around the corner, I need a way to attach these ReiserFS drives to my Windows PC externally and a box of tools that allow format, wipe, ... ;-)

 

Harald

 

JoeL,

 

the first upgrade process stopped after some errors right in the middle and left this new disk in a bad state. So after bringing the system back to normal operation (with the old disk back in and a successful parity check) I was ready for a second try. Again I would move the 750GB out and put the 1TB in. But now this 1TB is no longer empty it is partially filled instead. The disk is simply in a worst state.

If you were using this disk to replace an existing disk, the contents on it are completely ignored.  Does not matter what it had, partitioned or not, as long as it is functional otherwise.

For me it was not clear what unraid would recognize if I would use this partially written disk again. Is this disk formatted and new or is this disk formatted and partially filled or would unraid find out that this disk is in a bad state.

If you were to add the partially filled disk into a new logical slot in your array, and not replace an existing one, it probably would recognize the formatting and just accept it as it, files and all.

So my idea was to not rely on unraid and wipe the disk before using it. The idea was to put the new disk in. Keep the array stopped and format the disk (however this may work). But this format should not touch the parity disk at all!!

That would not help... unless you clear the disk in a very special way, leaving an exact "signature" set of bytes, or put a true reiserfs on it in the first partition (as it probably already has) it would be cleared anyway.

By reading nearly the complete board this morning I doubt that this is possible. Perhaps unmenu could had helped here - but I didn't want to open up a second case during my problem.

 

I think, with the second unraid box just around the corner, I need a way to attach these ReiserFS drives to my Windows PC externally and a box of tools that allow format, wipe, ... ;-)

 

Harald

 

You are worrying too much about the existing partitlly written contents of the 1TB drive. 

 

Step 1.  Do a full parity check.    If no errors, continue...  If errors, report back here with a syslog, do not continue.

Step 2.  Stop the array

Step 3.  Power down

Step 4.  Replace the old 750Gig drive with the new 1TB drive

Step 5.  Power Up.

Step 6.  Check the "I'm sure" checkbox under the "Start" button

Step 7.  Press the "Start" button.  The original files on the 750Gig drive will be re-constructed on the 1TB drive.  This will take a number of hours during which you can use your array. (depending on your array, it could take as long as 8-10 hours... that's how long it would take on my old mostly IDE based array)

Step 8.  Once the reconstruction is complete, the array will be back to normal... Let it run for a week or so before you re-use the 750Gig drive to make sure the replacement is working properly. (after all, it still has a good copy of its contents, even if not plugged into your server)

 

Joe L.

I would try to run burn-in tests on this drive first, just to make sure the drive is fine.  I think I would recommend a thorough burn-in for any of the newest drives, with higher DOA and first month failure rates, or when there has already been a failure, as in your case.  Trying to repeat the same procedure, hoping for a better result may work, since we don't know why it aborted last time, but it would be nice to have full confidence in the drive at least.

 

I can't tell from the syslog, but it looks like your very last disk controller may have 2 more SATA ports?  If so, it would be better to burn it in within your unRAID environment.  You can mount it outside of the array, and use the SMART long test, as well as format and copy terabytes to it.  If you can't hook it up to your unRAID server, then connect it to a Windows machine, and format it any way you like, run Scandisk's surface test, and copy terabytes to it as fast as possible.  You want to try to make the drive fail, by working it as hard as possible, and using as much of the surface as possible.

Thanks again to both of you.

 

Ok, the old/new 1TB is currently been written to. The upgrade process is now at 30% - this means several hours to go (fingers crossed).

 

What I learned is that I will never use all available ports again. I will leave the eSATA port for drive activities that I can do on drives standing outside the array. disk15 is currently connected to this eSATA port. This drive will be moved to a new system as soon as I have a new machine here.

 

Regards

Harald