July 10, 201015 yr Hello! It has been a while since I am happily using unRAID. Recently, as the capacity of my server grows, I have been experiencing freezing during the parity check. Freezing means that the servers hangs, no response on ping or web interface, black screen when I plug one on it. The freezing occurred at a random moment during the parity check, but eventually I was able to complete one after several reboots. Now the problem is more serious : I am upgrading my parity drive to 2TB and the parity rebuild doesn't last longer than 2 minutes before the whole server crashes. Again the timing is random but it does not go over 2 minutes of checking. I am unable to provide a syslog of the error as when it happens I loose all connectivity with the server. I am able to use the (unprotected) server without problem (browsing shares, copy files from/to it) . Syslog during those "normal" operations does not indicates any error whatsoever. Temperatures are not too high either (max 40 for the hotter disks, CPU @ 48°C) I am using V 4.5.4 of unRAID with 7 disks for a total of 6.32TB (incl 2TB parity) CPU is AMD Athlon 3000+ RAM is DDR 1GB PSU is 550W seasonic Disks are connected through 2 PCI-E controller card as well as a SATA2 port on the motherboard. Any idea what could cause this issue and how to solve it would be much appreciated EDIT >> Issue caused by faulty disk controller. SOLVED by hardware change.
July 10, 201015 yr Motherboard Model? Exact PSU Model? Model of Controllers? Please post a syslog. Please read this article. http://lime-technology.com/wiki/index.php?title=Troubleshooting
July 10, 201015 yr Author Hello WeeboTech, Here are the requested info : MB : Asrock 939 Dual SATA2 CPU : AMD 64 3000+ Ram : 2* 512 Mb DDR 266 PSU : SeaSonic S12 Energy Plus SS-550HT Click here for syslog. So I tried something : rearranging the power connectors from the PSU to the disks as I noticed I still had a spare cable with 3 connectors on it. I figured it might be on another power rail. So far so good, it has been rebuilding for about 20minutes now (still about 1700 to go ...) That is the reason why I can't give you more info on the controller cards : I have to read the chip to tell you what they are. They come from eBay, cheap cards but as I said they work fine from some time now (about 2 years without issues) Do you see something wrong in my server with the data I provided?
July 10, 201015 yr Author Hmneh! Crashed again after 35minutes EDIT >> Sooo the controller chips are : PCI-Express 1x 4 ports : silicon image Sil 3124 ACBHU QU9903.1-11 0908 AD03AX2 PCI-Express 1x 2ports : silicon image Sil 3132 CNU QK 8059.1-1 0645 AD02KX2 Do you know if there are a lot of people with over 5TB of usable disks?
July 10, 201015 yr 1. Apply the patch described in this post: http://lime-technology.com/forum/index.php?topic=6127.msg59095#msg59095 2. Test your RAM! From the boot menu select "Memtest86+". Let it run for a few hours. Preferrably, leave it running overnight.
July 10, 201015 yr Do you know if there are a lot of people with over 5TB of usable disks? Yes, Absolutely. There's a high number of folks have anywhere from 12 - 20 disks of 1TB / 1.5TB / 2TB drives for well over 11 / 16.5 / 22 TB of usable space (excluded parity). You should perform a memtest overnight to determine if it's memory related. After that, perform the other steps listed in the Troubleshooting Guide of the Wiki: http://lime-technology.com/wiki/index.php?title=Troubleshooting
July 10, 201015 yr Please post a syslog for review. Upon quick review, the PSU should be adequate for 7 disks. Hard lockups are usually an indication of a hardware conflict. My first question comes up with this card. PCI-Express 1x 4 ports : silicon image Sil 3124 ACBHU QU9903.1-11 0908 AD03AX2 This card is a hybrid, it is usually a PCI/PCI-X card. http://www.siliconimage.com/products/product.aspx?pid=27 So if it is a PCIe card, there is a bridge chip in there. I would move as many HDD's off this controller as possible. (Just my quick thoughts). Without a syslog it's hard to determine if there is a real hardware/driver issue.
July 10, 201015 yr Author Hello! Here is the last "current" syslog I retrieved from the USB key. I am afraid it does not indicate much more Memtest is running now, I'll let it go for a few hours and let you know the outcome. Thank you all for your suggestions EDIT >> WeeboTech, I am afraid the card you mention is all I have to plug 4 of my disks, I just do not have another option instead buying another. Do you see in my syslog something that may indicate a conflict? I already disabled as much as I could from the BIOS (audio, FDD, printer, serial,...)
July 10, 201015 yr I do not see anything that stands out. You could try some boot code hardware options. See Joe's post and the relevant Wiki page. I know I had to use some of these options for certain boards to prevent a hardware freeze. try some of the boot codes that modify the interrupt handling... 1) noapic nolapic OR 2) noapic acpi=noirq OR 3) noapic acpi=off OR 4) noapictimer OR 5) noapictimer irqpoll OR 6) noapic acpi=off OR 7) noapic acpi=noirq nolapic These would be added to the syslinux.cfg file on your flash drive as described in the wiki: http://lime-technology.com/wiki/index.php?title=Boot_Codes
July 10, 201015 yr Author Hi there! I ran 8 passes of memtest : no errors, I guess we can rule out that option. I also had a look in the BIOS and noticed the old SATA1 controller on the mobo was still active. As I do not use it I deactivated it. At the same time I tried the noapic nolapic option at unRAID boot (non permanent) at the "Press tab for boot options". I hope it is OK to input those that way instead of the syslinux.cfg. It is rebuilding it's parity now (12minutes and counting). I'll keep you posted. EDIT >> So far so good : 1 hour and counting. A bit early to say I am out of the woods but I think I see the light
July 10, 201015 yr Hello! Here is the last "current" syslog I retrieved from the USB key. I am afraid it does not indicate much more That's your "current" syslog, from after the reboot. There can be no traces of the crash in it. What's interesting is the "last" syslog, the one before the reboot. That one can contain useful info from the last moments just before the crash. If you've applied the patch I mentioned above, the last syslog should be on the flash disk. Look for file called "syslog.last"
July 10, 201015 yr Author Hello purko, I am afraid I do not have a syslog.last file on the flash drive with the patcch you provided. Does it disappear when the flash drive is replaced in the server and the server rebooted?
July 10, 201015 yr Hello purko, I am afraid I do not have a syslog.last file on the flash drive with the patcch you provided. Does it disappear when the flash drive is replaced in the server and the server rebooted? Let me try to explain a little better... You can have one of the following two scenarios: (A) -- Your server crashed, and you did not reboot it back into unRAID, but unplugged the flash disk and examined in in another computer, then the interesting file on the flash disk is called "syslog.current". or (B) -- Your server crashed, and you rebooted it back into unRAID, without touching the flash disk, then the interesting file would be "syslog.last" -- the one from before the reboot. (Maybe I should have called this one "syslog_before_the_reboot") See, at unRAID boot time, (if you've put my patch in yout 'go' script), the (old) "syslog.current" if found on the flash disk is renamed to "syslog.last", and a fresh new file is started by the name of "syslog.current". Does my explaination make some sense now?
July 10, 201015 yr Author OK I see... But silly me I made the server crash earlier : I guess copying some files from it to another machine during a parity rebuild was not a good idea... So, two good news : 1°) I have a syslog.last file on the flash drive. It does not appear to contain any useful data however : See for yourself. 2°) Current rebuild is running for over 3 hours now, I lit a candle and sent Murphy away. I hope this will be enough. Ah, by the way, this new rebuild is made without the noapic nolapic option : I wanted to see if the bios modifications had any influence at all, apparently it does.
July 10, 201015 yr But silly me I made the server crash earlier : I guess copying some files from it to another machine during a parity rebuild was not a good idea... This is not true, you can use unRAiD while it is busy. The fact remains that there is a hardware issue and this issue is what is causing the problem. It should not be this sensitive to use. There is a larger underlying problem. I lit a candle and sent Murphy away. I hope this will be enough. unRAID is not magic, no rituals needed LOL. Seriously, there is some hardware issue. it should not need special conditioning other then some options or it's failing hardware.
July 10, 201015 yr Author I hear you Unfortunately that hardware issue caused the server to crash again after 4 hours Here is the last syslog but it does not indicate any hardware issue or IRQ conflict is it? Considering I have over 20 hours of rebuild ahead of me, I still have a long way to go EDIT>> Strange thing is : It seems the rebuild goes farther at every reboot, does this make any sense to you? Anyway, I am relaunching another rebuild with noapic nolapic and see how far it goes...
July 11, 201015 yr From your syslog I see that you've installed "unraid notify". It is trying to send you some email, every hour, and it is failing. Did you set it up to email you every hour even if the server is OK? Or did you set it up to email you only when it detects some problem? If it's the latter, I'll show you how we can save the "dead letter" output to the flash disk. Speaking of which, can you please issue the following command, and post the output... ls -l /var/log/packages (these are small "L"s, not capital "i"s) Is there a monitor attached to that server? If so, what do you see on the screen after the server freezes? What disk-temperatures is the management web page showing while you're building parity? Do you by any chance have a multimeter? You could try measuring the voltages from the power supply while the server is running and building parity. See if the readings hold steady, or if they fluctuate a lot (bad power supply)
July 11, 201015 yr Author Here is the list of the packages : total 364 -rw-r--r-- 1 root root 7430 2010-07-11 01:27 binutils-2.18.50.0.9-i486-1 -rw-r--r-- 1 root root 829 2010-07-11 01:27 cxxlibs-6.0.9-i486-1 -rw-r--r-- 1 root root 8911 2010-07-11 01:27 gcc-4.2.4-i486-1 -rw-r--r-- 1 root root 304357 2010-07-11 01:27 glibc-2.7-i486-17 -rw-r--r-- 1 root root 20753 2010-07-11 01:27 kernel-headers-2.6.27.7_smp-x86-1 -rw-r--r-- 1 root root 1084 2010-07-11 01:27 mailx-12.3-i486-1 -rw-r--r-- 1 root root 3363 2010-07-11 01:27 make-3.81-i486-1 -rw-r--r-- 1 root root 1621 2010-07-11 01:27 socat-1.7.0.0-i486-2bj -rw-r--r-- 1 root root 798 2010-07-11 01:27 unraid_notify-2.54-noarch-unRAID Yes, I have setup unraid notify to send a mail every hour no matter what, I planned to input mail filters to get an alert for big issues. Somehow I got troubles configuring it with gmail and forgot about it, hence the errors in the syslog, I figured I would have a look at it later on then some real troubles came : I upgraded the parity disk Just joking here, I need to blow of some steam as the server crashed again, now I am trying the noapic acpi=off option. Yes I plugged a monitor & keyboard to the box in order to get some more info or troubleshooting help but they get as frozen as the box when it crashes. Even the numlock key does not light anymore :-/ Disks temp are well within normal behaviour between 35 for the coolest and 43 for the hottest. That is during the rebuild. Unfortunately I do not have a multimeter so I won't be able to get more info on that side. Again, thanks a lot for your time and precious help on this matter, it really cheers me up More info when the rebuild finishes or the next option.
July 11, 201015 yr Author Hmneh! noapictimer option... Could you please confirm that if I input these options at the "tab" prompt of unRAID it takes them? Or should I edit the syslinux.cfg file each time?
July 11, 201015 yr I'll show you how we can save the "dead letter" output to the flash disk. Post this anyway.
July 11, 201015 yr Author Hey! Some news I was checking the last syslog and found at the end some new lines indicating a PCI error : Jul 11 03:09:59 Alpha emhttp: shcmd (32): killall -HUP smbd Jul 11 03:09:59 Alpha emhttp: shcmd (33): /etc/rc.d/rc.nfsd restart | logger Jul 11 03:10:04 Alpha syslogd 1.4.1: restart. Jul 11 03:10:04 Alpha kernel: klogd 1.4.1, log source = /proc/kmsg started. Jul 11 03:25:01 Alpha kernel: sata_sil24: IRQ status == 0xffffffff, PCI fault or device removal? Jul 11 03:25:01 Alpha kernel: sata_sil24: IRQ status == 0xffffffff, PCI fault or device removal? Jul 11 03:25:01 Alpha kernel: Clocksource tsc unstable (delta = 2789658751 ns) Jul 11 03:25:01 Alpha kernel: Switching to clocksource jiffies So, sil24 seems to indicate the 4 port card as you (WeeboTech) pointed yesterday. Good hinch Now the question is : if it is a conflict is there a way to correct it? Still rebuilding for over an hour now... irqpoll is the next on the list
July 11, 201015 yr I'll show you how we can save the "dead letter" output to the flash disk. Post this anyway. I didn't want to confuse him, because in his setup those hourly emails are likely "You server is OK". Anyway, here it is: Open a telnet session to your server, and Copy/Paste the following three lines into the telnet window: touch /boot/dead.letter ln -sf /boot/dead.letter /root/dead.letter echo 'ln -sf /boot/dead.letter /root/dead.letter' >>/boot/config/go That file gets appended with every ssmtp failure, and it will now grow on the flash disk forever. Not the brightest thing to set-and-forget, but may prove useful while we're troubleshooting the current problem. (I mean, one of those emails may have been trying to notify us about a real problem)
July 11, 201015 yr Yes I plugged a monitor & keyboard to the box in order to get some more info or troubleshooting help but they get as frozen as the box when it crashes. Even the numlock key does not light anymore I wasn't asking about the numlock, I was wondering if you see anyting displayed on the screen when the computer freezes. Unfortunately I do not have a multimeter so I won't be able to get more info on that side. Could you maybe borrow one from somewhere? Or buy one? Even the cheapest one will do. (And they are cheap nowadays, I can get one for like 10 bucks from my local wallmart)
Archived
This topic is now archived and is closed to further replies.