It's a LOT easier to boot up a live CD like Knoppix or Slax, partition and format the new drives with ReiserFS, then reboot into unRaid, add them as data drives and do a parity sync... who cares if they are blank or not ??

 

 

If you do this then you will lose the data on a drive if it fails during the parity sync.

 

The clearing is supposed to ensure the drive is clear so it can be added to the array without changing parity. If it is cleared for sure then when it is added to the array the parity does not have to change (the blank disk which I believe is all 0 will have no effect on parity) meaning the array is not left vulnerable to a disk failure.

 

If the disk is stuck in and then the Restore button is used to add it the parity is invalid and a failed disk can not be recovered.

 

It seems to me the array should only be vulnerable only when upgrading a disk or when rebuilding a failed disk. I think you could get around the failure when upgrading a disk by copying the right files on the flash and then putting back the disk that was upgraded basically returning the array to the configuration it was before you tried upgrading a disk.

 

Actually, I wish there was a way to clear and then cleanly remove a disk too. Maybe it's a post here somewhere. Instead of upgrading just add then new disc, copy the data over, then cleanly remove the old one never leaving the array unprotected.

 

Peter

 

Thanks all for your support. I really don't think this is a hardware trouble, as without changing anything, just clearing less disc at the same time, it works perfect. Also i don't seem to be the same person having this particular trouble. It seems the more time it spends clearing, the more chance it gets to stop. When a drive is bad, you should get some kind of error, not the clearing process that stops but the main page thinking it is still going on

 

For disc on the PCI, clearing 2 at the same time worked. So they are formatting right now, after i add 2 new and clear them. I have no error at all.

 

When finaly i get my server running (i have 0 file on it, trying just to create the full array since a week), then i will launch a parity check to see how long it takes and if i get in trouble. But i am more confident now

 

I've had the same problem with clearing multiple drives. I only do one at a time now. I've never had a problem running a parity check though. I have 16 drives in my box. You shouldn't have a problem.

Thanks a lot, that makes me feel a lot better for using my new unraid server when it will be ready. I was begining to think this was a bad idea. Because in the end if you don't feel confident with the server, it starts bad.

 

Going 2 by 2 for the last ones, only 2 more clear cycles to go and i'll have my 14To array up and running, after a week !

I really don't think this is a hardware trouble, as without changing anything, just clearing less disc at the same time, it works perfect. Also i don't seem to be the same person having this particular trouble.

 

Either you do or you don't have a hardware problem.  The fact that your entire multi-drive drive clearing went to 97% means that the problem occurs very infrequently.

 

It could be an unRAID bug, but I would doubt it given that it got so far.  Seems a subtle timing issue that is common with cabling problems.

 

When finaly i get my server running (i have 0 file on it, trying just to create the full array since a week), then i will launch a parity check to see how long it takes and if i get in trouble. But i am more confident now

 

If you have zero files on the array, the way you are adding drives is terribly inefficient.  You could have easily added all your drives in step, taking < 1 day, without clearing a single one.

It could be an unRAID bug, but I would doubt it given that it got so far.  Seems a subtle timing issue that is common with cabling problems.

Most hardware problems would leave some clues in the syslog.  There are no failures logged.  Other than emhttp freezing the clearing process, but continuing to respond to browser requests, no other symptoms exist.  In my opinion, the failures all point to a bug somewhere in software.  Now, it could be a bug in the BIOS, or in the device driver for one of the disk controller, or possibly, in the emhttp thread used to clear the disks, but emhttp is apparently waiting for something before continuing to write zeros to the target disks.... and that event never occurs.

 

There are no errors in the syslog, the emhttp process still responds to browser requests, but the progress marks in the syslog and the clearing process just stops at some point.  It seems to do this only when clearing higher numbers of multiple disks, and at random times. I suspect a race condition leading to a deadlock of some kind.  Multiple clearing processes, all attempting to use some resource, all thinking some other thread has something locked for exclusive use.  All you need are two, each waiting for a resource the other has locked for a deadlock to occur.

 

It could be anything involved, even the firmware in the disk controller card and/or BIOS.  Or, it could be the parallel clearing process Tom wrote into emhttp.  I really suspect the BIOS and/or disk controller firmware first... or we would have run into this more frequently, but who knows, not too many people add more than a few disks at a time so it could be a deadlock in the disk clearing code.

 

Joe L.

Cables, cables, cables.....

 

 

Here are the last news... I finaly got the server up running with all drives formated. So i now have 14To to play with.

 

What i can say now is that if the clearing process last longer than 6 hours, i got into trouble. When i tried the all 12 drives left (i tested with 3 first), it stoped after around 6 hours at 37% (as i remember). Then i tried half, it stoped after around 6 hours at 97%. Then going with 3 worked, until i did 3 on the PCI bus where it went other 6 hours and stoped again. The last 2 drives stoped also after 6 hours. Then after rebooting (what i didn't do for 2 clearing session that went fine), i started other the 2 last drives and worked.

Now my trouble doesn't seem an isolated problem

 

So i would more likely go the Joe way, a software trouble somewhere. If it was cable, something would have been detected by Linux.

Now i will see that pretty fast as i have around 10To to transfert on it, meaning the drives will run much longer now. Well as it takes more than 10 hours for 1To, it will take time, but the disc will still work much longer.

 

Even if the init was a disaster (finaly 5 days to make it up and running), i am very happy now, as the transfert rate is doing an average 26.3Mo/s for my first To of data, without cache drive 

(i didn't expect that much from what i read in the forum)

 

After few To, i will run a parity check to see what happens.

If you have zero files on the array, the way you are adding drives is terribly inefficient.  You could have easily added all your drives in step, taking < 1 day, without clearing a single one.

 

Well i didn't choose the way it was done, the main page did. To start the array it told me it would clear all the new drives first. Then i had to format them. Was there another way to initiate the array ?

Even if the init was a disaster (finaly 5 days to make it up and running), i am very happy now, as the transfert rate is doing an average 26.3Mo/s for my first To of data, without cache drive 

(i didn't expect that much from what i read in the forum)

 

After few To, i will run a parity check to see what happens.

 

Let us know what happens. Good or bad.. 

Very bad news when copying some files, but more luck, because this time, i have a kernel panic. But having this, impossible to connect either remote or local, meaning no way for me to get the syslog (or i didn't know how to). So i took a picture of the screen with my camera. Here it is.

 

In the panic i see a lot of timer and something about jmicron and promise. I have 2 Promise TX4 connected on PCI in the server, with 7 discs connected to them (4 and 3, at the end of the array) and the Gigabyte GA-EP35-DS3R has 8 sata port (6 from Intel ICH9R chipset and 2 from a called "Gigabyte Sata2 chip"). Everything is setup as native Sata AHCI

 

Now there is some error message, what do you think ?

Perhaps already mentioned ...

 

I seem to recall a problem running multiple TX4 cards in one system.  Something worth googling.

 

 

Bill

As far as I know, a kernel panic always halts the system, so a screen picture is all you could do.  It does try to leave as much info on the screen as possible.

 

The first thing I would do is reboot and start the memory tester, and let it run all night.  We have to make sure it is not bad memory.

 

The list of linked modules is not very helpful in this case, probably misleading.  And the routines shown are so low-level, that they don't really help us either.  If I trusted them, I would suspect something wrong in the timer hardware, and that is extremely rare, and would indicate a hardware fault in the motherboard, which is probably not fixable.  Test the memory first.

 

Hopefully, someone else will have better ideas...

Perhaps already mentioned ...

 

I seem to recall a problem running multiple TX4 cards in one system.  Something worth googling.

 

 

Bill

Well i took these because it is the one used in the 1500 server sell by unraid. During post, it consolidate the drives from the 2 cards and list the 7 discs at once under Promise. But you can not add more than 2 of these card in a server.

 

As far as I know, a kernel panic always halts the system, so a screen picture is all you could do.  It does try to leave as much info on the screen as possible.

 

The first thing I would do is reboot and start the memory tester, and let it run all night.  We have to make sure it is not bad memory.

 

The list of linked modules is not very helpful in this case, probably misleading.  And the routines shown are so low-level, that they don't really help us either.  If I trusted them, I would suspect something wrong in the timer hardware, and that is extremely rare, and would indicate a hardware fault in the motherboard, which is probably not fixable.  Test the memory first.

 

Hopefully, someone else will have better ideas...

Yesterday i checked again all the cable and disc in the cages, then launched a memtest for 2 hours, without any error. But as i very often get errors after 6 hours, i launched once again the memtest and will let him the day to run. I will keep you informed.

 

If it is a rare motherboard trouble, it is going to be hard to get a new one. I imagine telling the seller my card is bad, timer trouble, after running unraid for more than 6 hours. As they try to reproduce first, this is going to be a tough one

(just hope the memtest will crash the system after some time, this would be easy to reproduce for them)

 

Thanks for your help, i apreciate, i was really wondering what to do

Well he are the latest news. I run a memtest for more than 8 hours, it passed 26 times without any trouble, no error. I am happy to see that my good Corsair memory is doing fine.

 

As this was my first kernel panic, i searched a little to see what that means. So now i know the lib listed (all the sata drivers) are just the loaded modules when the crash happened, so doesn't mean they are concerned. In fact just the text next to the kernel panic seems important and related to the error. And searching for it, it could be anything : i saw graphic card, memory module, bug in the kernel/drivers.

So not of any help for my case.

 

I changed few things in the BIOS while rebooting after the memtest (disabled floppy, virtualisation and High Precisoon Timer). At reboot unraid launched a parity check because of the preceeding kernel panic. I find it very slow (15Mo/s, for all SATA II drives). I guess this is again the 2 PCI promise sata card that are the bottleneck, as all the drives are read at the same time. Knowing this, it will take more than 17 hours to complete.

 

I might go for a PCI Express x8 SATA card. I was thinking of an Adaptec one. Do you know if their RAID card are supported by Linux/unRaid ?

I know those do hardware raid6 and are expensive, but it is the only solution i found with at least 8 sata ports on pci express. So i will be using it for just JBOD. Didn't find plain jbod pci express sata ii card.

 

During the parity check, i leave a tail on the syslog in case something occurs. It should be better than a picture of the screen ;-)

Well, what is nice now is that the kernel panic comes more quickly, so i don't have to wait untill morning to detect a trouble, giving me the possibility to do more tests.

 

So same kernel panic reason as last time (i took a picture in case someone wants to see, as the tail syslog gave nothing at all, no message before the panic).

 

So i read while parity and found a post where it said to update the bios, which i did from F2 to latest F3. Then power off, then power on, bios, loading safe default, enabled AHCI, disable HPT, floppy and virtualisation (didn't help last time but who knows) and boot on unraid flash

 

So back again for a parity check... After memtest that was ok, cable checked, now latest bios, maybe it will fall working 

I give up for now. In less than 3 hours i get another kernel panic while checking parity.

 

So latest bios, good memory, this is for sure.

 

Now if i have a drive that has a trouble, linux shouldn't panic but tell it clearely and unraid mark it as bad or try to isolate the bad sector, right ?

If it is some hardware, what could i do to isolate the fault device ?

If it is a cable, is there a way to isolate without buying all 15 expensive cables to replace the generics one i bought ?

If it is a bug in the kernel, is this bug known and do i have a chance it will work for next release ?

 

I know, a lot of question and no answer. But if it is hardware, i need to know which to send it back.

It is a pity it worked with 3 discs (i put 1.5To of data) and doesn't work at all with 15 discs (14To). All my 15 discs are Samsung Spinpoint F1 HD103UJ.

Is your PSU capable of handling the current from the loading of all your drives?  When I added two more drives (12 up to 14), my PSU couldn't hack the extra load.  It was fine initially, but attempting a long parity check caused a kernal panic.  I figured it was the loading as everything had been fine until then.  I swapped out a 600w for an 850w and everything is now fine.

Is your PSU capable of handling the current from the loading of all your drives?  When I added two more drives (12 up to 14), my PSU couldn't hack the extra load.  It was fine initially, but attempting a long parity check caused a kernal panic.  I figured it was the loading as everything had been fine until then.  I swapped out a 600w for an 850w and everything is now fine.

Thanks for the advise.

 

I took a 700W power supply, the Cooler Master Real Power Pro 700W (i went for 700W as the 1500 unraid server is sold with a 620W for 15 discs, and also because the samsung eats as much as 13W writing). I put in front of the server a wattmeter, to check the power used. During parity check, i was surprised to see a little less than 200W instant usage. The max power drained was at startup with 306W. When all discs are sleeping, it uses 101W.

 

But the Amper needed may exceed what the power supply can give. I would think when this happens that it puts itself in security mode, meaning it would powerdown the server, not behave erratic.

 

But now if you could give me a model that is sure of working full loaded (so with 17 discs, in case) with a lot of molex (i need minimum of 9 for my 3 ici dock MB-455) and enough power left (i don't want to use it near its edge, i would rather leave some horse power free so it gets loaded far less than 70%), i am interested.

 

To test this, i am going to unplug all the drives on the 2 promise card, leaving only 8 discs in the system (i will unassign first in unraid before unplugin). I will take of the parity disc the time loading plenty of data, then i will enable the parity (still leave 8 discs) and let him build the parity (i guess that will mean all the discs will be running and writing to the parity disc, so heavy stressing). I will see if that works.

 

Thanks for this little light of hope 

Cabling errors, drive errors, and most peripheral hardware errors are detected by drivers and the kernel, are handled, reported, and logged.  They do not cause kernel panics.  I think that core hardware components could possibly cause a kernel panic, but you have eliminated the most likely one, memory.  So that only leaves low level software bugs, in core modules and the kernel.  You could try running older versions of unRAID, in order to test with different kernel releases.

 

I would be more likely to think power problems or temperature problems if the system were turning off, not terminating with a kernel panic.  However I'm not expert enough to say if a power or temp issue could or could not cause a kernel panic.  If the panic screen messages are consistent, with very consistent trace dumps, similar last routines listed, then I would strongly suspect a software problem somewhere.  Or could possibly still be the same faulty motherboard component.

 

Was this motherboard bought new for unRAID, or has it been running another operating system without issue?  In other words, would you consider it well tested?

Thanks Rob, i feel better with cables and drives now.

 

Everything in this server is new except the memory that was on my main PC before (2x 1Go Twinx CORSAIR DDR2 XMS PC2-6400).

 

For the power supply, last thing i tried was taking just the drives for the Intel ICH9R chipset, taking away the parity. Then i pulled out the 2 promised card and disabled the extra 2 sata port from Gigabyte. So i endend with a server having just the integrated NIC card, an ATI card and 5 drives connected to ICH9R. This takes just a maximum of 107W when writing to one disc, and less loaded drivers in unraid (700W for just 5 drives, should be more than ok).

 

But still got the kernel panic. But this time a DIFFERENT MESSAGE, therefore it could help you more find what is going wrong with my system. Maybe it shows the clear reason of my trouble.

From the new more clear message, i wonder if this means memory or some bios settings (i kept the default in bios, meaning this is a cas 4 and it gets cas 5-5-5-18 in the bios, at 800MHz, its normal speed)

From my google searches, it could be faulty ram (but they got errors on memtest which i didn't) or bug in the kernel (but i wouldn't the nearly alone with this, would i)...

 

If i have to test a different version, i wonder if my integrated NIC card was just supported in 4.3.3 (Gigabyte GA-EP35-DS3R with gigabit Realtek RTL8111B) ?

 

Thanks in advance, maybe the end of my nightmare (i even flashed my switch to get the latest firmware, just in case)

Well, progress, sort of.  You have proved that the failure is exactly repeatable.  The EIP (current CPU instruction pointer) for both images was at update_wall_time+0x2d3/0x6a6.  The higher level routine running was swapper for one and smbd for the other, so that seems to implicate a very low level timer routine, not whatever was running at a higher level.  Because that EIP is well past the update_wall_time label, I have no idea what the actual code is doing here.  But there is enough information in the 2 pictures, that someone with Linux kernel experience, especially kernel debugging experience, might be able to pinpoint something, but I don't know if we have anyone with that experience in this forum.  General protection fault was 0000 [#28] in one case and 0000 [#40] in the other.

 

It still looks like faulty motherboard component to me, since you eliminated memory, and buggy kernel seems unlikely, why would it run fine for hours and then fail.  Although you can play with different memory settings in the BIOS, and disable everything motherboard related (including networking), if it still panics, you may have to consider replacing the motherboard.

On a bad note, I just discovered another thread with the identical board, and kernel panics (but not at the same place).  His went away apparently, on installing the F3 BIOS.  Are you absolutely positive your upgrade to the F3 BIOS version was successful?

 

Otherwise, this Gigabyte board may have to be considered problematic under Linux.

 

One other possibility, is to experiment with boot options, see the Boot Codes wiki page.

Thanks, i'll check all the bios option to see if there is something here.

 

I flashed the bios to F3, and it is what it tells me at the boot (F3). After the flashing process, instead of loading the default perf parameters, i choosed the default safe parameters, as i was having already problems.

 

If it is the motherboard, i wonder what to do to get a change, as they check first and it will show it works fine (i doubt they'll install it with unraid and 6 drives and make a 6 hours test). Would there be a test to do outside unraid (or easy without all the disc) to "prove" more easely it is the motherboard that is faulty ?

The more i test and the less i understand what is going on.

Now it seems to work with the 6 internal ICH9R discs (could copy and build the parity disc).

 

As another user reported trouble with the same board and the 2 gigabyte internal Sata, i turned it on in the bios and added 2 more drives. I also changed during this last test the powersupply to a corsair 750W with only 1x 12V rail of 62A. It did rebuild the parity disc ok in 8 hours. I still don't think it is the powersupply as 6 drives will eat far less than 6A and 8 drives less than 8A all writing (reading eats 0.6A each, and 0.9A when writing).

 

So it seems i have a server that works, but just don't know why it is now working while it didn't from the start.

 

I will try this evening to plug the 2 promise cards and plug the 7 last drives, then let him build the parity disc, making an estimated 18 hours job where 14 discs will be read and 1 write. If it still works, well i just will never know what was wrong in the first place.

Well didn't work with 15 drives, same kernel panic again :'(

All the tests i did where unsuccessfull for now.

 

The last test i am trying is to use my backup flash, which is from different brand and speed. I don't expect any good result but now any test will do it. Like this, it is a brand new system for that flash. It is building parity...

 

After that i will have to wait for a pci express sata card that works with unraid to replace the 2 pci promise card