Hi everybody, i just made the big jump yesterday with unraid, and already got a problem

 

So i have a 15 x 1To Samsung drives with a Gigabyte EP35-DS3R with 2x promise PCI 4xSATA. I already added the first 3 to see if it was working with my hardware. Then i got a pro version key.

 

So I added all the 12 drives left, then started the array, beeing told it would clear all the disks first (thing i didn't get for the first 3). I could see all the 12 lights blinking on the case during the clear phase. I checked in the browser seeing the completed % increasing. I then went to bed...

In the morning i go to the basement, cool, all lights green, so has finished. Then i go to check in the browser, just to see only 37% complete. I can do nothing in the page, as it is starting, so no shutdown, reboot possible without pushing the reset button on the case. As this seems very bad for unraid, did nothing, still at the same point, 37% with all the discs sleeping.

 

The unraid is not dead, as after reading on the forum to see what to do, i could copy the syslog to the flash from the unraid server, then go to my vista computer and post here (so got the file from the network on the flash, as this is the only share i see).

 

So what could that be, and what should i do the right way to make it work. Did i ask him too much by adding 12 new drives in the array at the same time ? Or is this just bad luck.

Should i press the reset button on the case ?

 

Just want to tell that i am a Windows maniac, meaning a "mouse man", so not used to input text in a shell. Even more beeing french with an "azerty" keyboard while the unraid wants querty. I couldn't input a | (AltGr+6) in the console, got nothing (hopefuly the cp comand doesn't need any)

 

Thanks in advance

Another info : while watching the main page, i see that all the bullets are blinking, the green ones (the first 3 to test my hardware and get the GUID) and the 12 blue.

 

Sounds crazy, but could they have gone to sleep during the clear process, at 37% ?

login at the console, then type

 

ps -ef

 

see if there is a process named "emhttp" running.  The line in the process listing will look a bit like this:

root      1694    1  0 Aug23 ?        00:00:00 /usr/local/sbin/emhttp

 

If there is, then you can try to kill it and then re-start it.  It is the process that was doing the clearing, and it appears as if it has stopped. (has not done much of anything for the past hours)  If it exists, it might have somehow gotten itself confused?  (I've never heard of this before) or locked up waiting on an interrupt from a drive that never arrives (more likely).

 

To kill it type

killalll emhttp

 

If it does not exist, (or you kill it) you can invoke it by typing

/usr/local/sbin/emhttp &

 

You should then be able to use the web-interface to stop the array and power down and/or reboot. 

 

If you cannot get to the web-interface type:

cd /root

samba stop

for i in /dev/md*

do

umount $i

done

mdcmd stop

reboot

 

The drives should not have gone to sleep when being cleared.  That timer is in the drives themselves, and as long as they were being written to, they will  stay spinning if they are being written to.

 

You might want to run a memtest when you reboot.  It is as likely as anything..  You might also re-seat the drive controller boards in their sockets.  As things got warmer in your case, anything could cause the clear to stop.  I really don't see any clue in the syslog you posted.  It just seems the clearing stopped at 37%.  There are no kernel panic, nothing... 

 

Joe L.

Many many thanks.

 

Yes, very wierd, just stoped at 37%, without saying anything.

 

In the mean time, i activated telnet on Vista and got the french keyboard layout, so it will be easier to input commands.

 

The http was running, so i killed it and started other. Now the main page says the array is stopped. So i have all the action buttons. I took away half the discs, in case it was too much for him (on windows it takes 4 hours to format 1To, and can do several in // with the same time) here in 6 hours it was just 37%. So maybe the 2 promise in PCI slot was doing a huge botlleneck.

 

I will reboot and start the array again. I'll tell you what it did...

 

Thanks again very much for your help

Many many thanks.

 

Yes, very wierd, just stoped at 37%, without saying anything.

 

In the mean time, i activated telnet on Vista and got the french keyboard layout, so it will be easier to input commands.

Do yourself a favor, download "putty" and use it instead of the telnet client built into vista.  It is MUCH better.  With it, you can invoke the midnight commander program in unRAID and even use the mouse and have the function keys work.  With the vista telnet client, the function keys do not work, and you have to use "Escape-0" through Escape-9" for the function keys.

The http was running, so i killed it and started other. Now the main page says the array is stopped. So i have all the action buttons. I took away half the discs, in case it was too much for him (on windows it takes 4 hours to format 1To, and can do several in // with the same time) here in 6 hours it was just 37%. So maybe the 2 promise in PCI slot was doing a huge botlleneck.

 

I will reboot and start the array again. I'll tell you what it did...

 

Thanks again very much for your help

 

Keep us informed.  It might be another "clearing" bug that nobody has run into before. (Not too many people add 9 new disks at once)  Most of us purchase the drives when we see a good sale price and add them one at a time. (because the really low sale prices are often limited to 1 to a household)

 

Joe L.

I'm wondering if you got a bunch of disk errors and filled your syslog?  I'd recommend saving the syslog, rebooting (using the commands Joe L. provided), and then running a parity check. Expect lots of parity errors, but if it completes 100%, your parity should be okay. (Run a 2nd parity check to be sure)

 

Post the syslog and let the expert syslog readers here take a look.

I'm wondering if you got a bunch of disk errors and filled your syslog?  I'd recommend saving the syslog, rebooting (using the commands Joe L. provided), and then running a parity check. Expect lots of parity errors, but if it completes 100%, your parity should be okay. (Run a 2nd parity check to be sure)

 

Post the syslog and let the expert syslog readers here take a look.

He had posted his syslog earlier in this thread.  No such errors existed in it, and it was not using any abnormal amount of space.  It just showed the clear stopping at 37%.

 

I doubt he'll see many parity errors, if any at all.  He's probably in the middle of a parity calc now anyway...

Joe L.

I'm on my Blackberry and can't see attachments.

 

If you start to clear a drive and it doesn't complete, won't it come up thinking it was done?

I'm on my Blackberry and can't see attachments.

 

If you start to clear a drive and it doesn't complete, won't it come up thinking it was done?

No, after the clean reboot, the 12 new drives where new and had to be cleared again from start.

...

I doubt he'll see many parity errors, if any at all.  He's probably in the middle of a parity calc now anyway...

Joe L.

 

Well clearing 69% complete, doing 5 discs at once. Much quicker to complte 1% than with 12 discs and within that 7 on PCI bus.

 

When it is done, i think i'll go 2 by 2 for the ones on the PCI Promise cards, so i don't stress too much the server

 

But working well so far 

 

For the drives, in fact i didn't buy them all at once. It is just that it took me some time to decide myselft to jump in a shell server rather than a windows one, plus the money to build the server (it took me 6 months to buy it all). But i got 2 real good prices, that where just limited in time, not in quantity. So i took once 7 then in another even better bargain 6)

Do yourself a favor, download "putty" and use it instead of the telnet client built into vista.  It is MUCH better.  With it, you can invoke the midnight commander program in unRAID and even use the mouse and have the function keys work.  With the vista telnet client, the function keys do not work, and you have to use "Escape-0" through Escape-9" for the function keys.

 

Joe L.

Oh my god. For me at midnight i try to go to sleep ;-)

You see i am an old mouse man, so the less time i type and see a shell, the better my health is. And i know nothing about linux, so i just type commands i see in the forum.

 

So i guess for me telnet will be enough (i got the | and the french keyboard). I just use a tail right now on telnet to see the %progress (now 73). But if one day, who knows, i dive into Linux shell and do more, i promise to get putty.

 

Of course if midnight commander is like the old norton commander and i get like a windows explorer on linux filesystem, i might change very quickly my mind. I tried this morning a "cd.." command and got a nice : stay on windows old man, doesn't know that command

Do yourself a favor, download "putty" and use it instead of the telnet client built into vista.  It is MUCH better.  With it, you can invoke the midnight commander program in unRAID and even use the mouse and have the function keys work.  With the vista telnet client, the function keys do not work, and you have to use "Escape-0" through Escape-9" for the function keys.

 

Joe L.

Of course if midnight commander is like the old norton commander and i get like a windows explorer on linux filesystem, i might change very quickly my mind.

It is exactly like the old norton utility on dos.

 

To invoke it, type

mc

I tried this morning a "cd.." command and got a nice : stay on windows old man, doesn't know that command

On unix, you must put a space after the "cd" and before the ".."

So to change to the parent directory

cd ..

 

to change directory to the home directory (in this case /root)

cd

 

to change directory to the flash drive

cd /boot

To change back to the prior directory you were in before this one  (easy to toggle between two directories this way)

cd -

 

Just ask, there are lots of people here to help, especially if we know the answers because we've been using Unix commands for nearly 30 years :-)

 

Glad the clearing step is going better.  (and faster)

 

Joe L.

I wish this post was here yesterday.  I had the same thing happen clearing a 1Tb drive overnight.  Got up the next morning and while it had cleared there was no "Format" button.  So I set it clearing again and the WebGUI froze 1/2 way through and I didn't know how to get it back.  Joe - I followed your advice in another thread and upgraded to 4.3.3 from 4.1.  I set a 3rd clear going and everything finalised.  Great!  You answer here about the emhttp was what I really needed.

 

Out of interest, if you kill the emhttp process, do you have to start the clear from scratch again?

 

 

Out of interest, if you kill the emhttp process, do you have to start the clear from scratch again?

Yes... no way for it to know how far it had gotten.

I am going to shoot it  :'(

 

It freezed this time at 97%.

 

The drive lights are green on the case, meaning the drive do nothing, but the log and the main page stick to 97%

 

And flambot, once you kill http and restart it, it still consider your new drive new, so it will go through a complete clear process from start again.

 

Any new idea while i go to sleep ?

I join the new syslog file

 

I will start other, this time with only 3 drives to see what happens

Again I see no clues in the syslog...

 

At this point,  you need to start thinking some kind of hardware issue.  I'd suspect memory first, although it could be one hard drive locking up the whole bus.  Make sure you have the voltage for memory and timings set proprtly.  Run memtest overnight.

 

Then, how hot are the drives?  What model power supply?  Are you sure it can handle the additional drives?  Is there enough air movement.  Drive temps should all be in the 20's or 30's.  Temps in the higher 40's and upward need to be addressed.

 

It is as if the whole process just stops.  I'd make sure everything is plugged in securely.  CPU, memory, disk controllers, cables, everything.  heat can cause expansion and all you need is a loose card and who knows how it affects everything.

 

Try with the other set of 3 drives.  If it is a drive, you can eliminate it by process of elimination.  It is just as likely to be a loose cable, or drive rack connector.

 

 

Out of interest, if you kill the emhttp process, do you have to start the clear from scratch again?

Yes... no way for it to know how far it had gotten.

 

I figured

Thanks for trying to help me on this.

 

Temp of the drives are 30 to 34 and while clearing it goes up to 38 max. I have a coolermaster cmstacker 810 case, and a coolermaster RealPower M700 (RS-700-ASAA-A1) power supply of 700W. I have 2x 1Go Corsair DDR2. I use 3 icy dock 5 in 3 drive case.

 

This makes a total of 2x 120 rear fan and 1x 80 top fan to exhaust air outside the case (the 120 power supply fan is taking air from the bottom outside the case) and 3x 80 fan that sucks air inside the case through the drives (the one from the icy dock drive case).

I hope this is enough.

 

I don't think the trouble is the power supply, because i put a wattmeter on this server and it shows 145W idle and 189W while formatting.

I would be surprised if it is a hardware issue as unraid linux still runs perfect not showing anything. No error, no loss of anything (restarting http will show all the drive ok and the array stopped, as if they where new). All the cables are well attached on the motherboard, the promise cards and the drive case.

 

In fact only the clearing process of the drives stops by itself, but the http still thinks it is clearing while it has stopped doing so. If i do F5 on the page, it reloads but always shows the same information.

 

If it still doesn't work with 3 drives, should i try another release, as flambot mentioned ?

If it breaks this night (it is other midnight in france), i will run a memtest first morning and see what it gives coming back from work.

At least a good news. It worked with 3 discs. So i could format them. No error on the main page.

 

So i will try the 2 others to see what it does. If it still stops at 97%, maybe one drive has a trouble (and another one at 37%), but the clearing process should tell the disc has an error instead of stopping. I still doubt i got 2 faulty drives out of 12, that would make a very high % of faulty new drive.

 

Will keep you inform of my situation. Roger 

To keep you informed, it worked for the next 3 drives, now doing the next next 3 drives and doing 62% clearing in 6 hours (these are all on the PCI Promise card and i can feel the difference, very very slow compair to onboard sata)

 

Hope these will work fine so i can do the last 3. I am anxious because as it takes much longer, i am afraid it stops.

 

So with less drives, it works (the 6 done 3 by 3 wouldn't clear at the same time). Don't know but sounds more like a software trouble than a hardware one.

 

When all this is done (not before tomorrow if no trouble), i will copy some files and launch a parity check, praying a lot, as it will go through the 15 discs at the same time

grrr, freezed at 68%, i will try less drives when connected via PCI

I've had the same problem with clearing multiple drives. I only do one at a time now. I've never had a problem running a parity check though. I have 16 drives in my box. You shouldn't have a problem.

When adding drives to an array I recommend the following:

 

1. ALWAYS run a parity check and make sure all is well before beginning

 

2. Burn in your drives.  There is a 2 drive burn in process documented in the “Best of the Forums” section (see my sig).  Running a long smartctl test is another option.  Optimally you want to read each drive fully, write each drive fully, and then read each drive fully again.  This gives sector remapping a chance to do its thing.  The incidence of failure of brand new drives is significantly higher than a drive with some use – so running the drives for the hours it will take to complete these tests is a good safety precaution.

 

3. Add the drives to the array.  unRaid should clear the drives and then format the drives.

 

4. Run a full parity check again, to make sure all is well.

 

5. After several days of activity, run a full parity check again.  I’d recommend running it several times over the first month of so to make sure all is working well.

 

The clearing of the drives should not fail.  Yours failed twice.  Although it could be many things, my first instinct would be to suspect the cabling from the motherboard / controller to the drive.  (If you are using some sort of drive trays / enclosures this as a cabling issue from the drive’s perspective.)  While doing a burn-in, I had a hang just like you are describing when using a long eSATA cables to an external enclosure.  Replacing with a shorter one fixed the problem.  I had used that exact cable setup in Windows many times without a problem.  unRAID stresses this cabling MUCH harder than Windows.  One marginal connection and you are in trouble!

 

If the clear process does not work reliably, I’d recommend trying to figure out why.  Even if you are able to get all the drives assigned, you might still have a problem in the future due to the underlying issue. Of course there could be a bug when trying to clear a lot of drives. I have only tried to clear 2 drives at once which has worked for me a couple of times.

 

If the goal was to add the drives as quickly as possible, ignoring the clear disk problem, you could have hit the restore button.  That would have dropped the parity protection on the couple drives that were in the array.  You could then have added all the drives to the array (except parity), started the array, and allowed unRAID to format the new drives (unRAID would not have tried to clear the drives since there is no parity.)  Formatting drives without parity assigned is very fast.  Finally, you could have assigned parity which would have built parity from your whole array.

 

Good luck!

 

It's a LOT easier to boot up a live CD like Knoppix or Slax, partition and format the new drives with ReiserFS, then reboot into unRaid, add them as data drives and do a parity sync... who cares if they are blank or not ??

 

 

It is easier to format them outside of unRAID, but then you are bypassing unRAID and not really stress testing your hardware

 

bjp999 provides really good recommendations since there are a number of new drives added all at once.

 

If you are hanging during the clear process, then one day you may hang during heavy writes.