January 25, 201115 yr I need some help with my server. For some reason it has crashed 3 times in 3 days. It will work fine after a hard shutdown, but a call to spin up the disks and it will crash, or if you try to access something that is on one of the disks that are spun down it will crash. Parity checks after the hard reboot all are good no errors. Nothing can be done from the console the keyboard is unresponsive, the power button will do nothing. Give me a direction and lets figure it out. gb ep45ud3p mother board 2 g ram 12 drives Rosewill RC-217 3 drives on here AOC-SASLP-MV8 1 on here
January 25, 201115 yr Have you already run an overnight memtest diagnostic? Have you already troubleshooted the Power Supply?
January 25, 201115 yr I also had a bad hard disk cause similar problems. The memtest is the easiest thing do first, run it overnight. If you can, swap in another power supply. If still no luck, run a "test" array with a smaller set of disks and minimal expansion cards and memory to see if you can get the system stable. Slowly add SATA cards and additional disks piecemeal. When I had a difficult stability problem to solve, I ended up buying a new motherboard to eliminating that as a potential cause. Troubleshooting can get expensive.
January 25, 201115 yr Author I will run memtest tonight. What methods do you suggest to troubleshoot the power supply i have CORSAIR Enthusiast Series CMPSU-650TX 650W ATX12V / EPS12V was supposed to be a good one. I also attached a picture of the console before hard shutdown.
January 25, 201115 yr you have cache_dirs running ? if yes... then i think you are caching a bit tooo much and your machine is running out of memory if no... swap enabled ? any corruption on your swap file? check your logs for any problem with the disk where the swap file is on ? Else... wait till a more experienced user gives comment on that screenshot
January 25, 201115 yr Author Server is back up and running I'm gonna let it finish its parity check. Should be done tomorrow afternoon. I also set the spin down to never. If I come home tomorrow and everything is working and the parity was good and I can access each disc separately I would think we can then rule out the power supply and disks and cabling, possibly even network functions. While it's running its parity tonight I'm gonna stream some movies.
January 25, 201115 yr Author I have a cache drive aside from that I do nothing special with it. Other than it is enabled. No swap enabled that I know of if it is then I didn't do it.
January 25, 201115 yr I also set the spin down to never. If I come home tomorrow and everything is working and the parity was good and I can access each disc separately I would think we can then rule out the power supply and disks and cabling, possibly even network functions. Not really. Typically drives use the most power when they spin up from power down, not when they are sitting idle or even if they're in use. Do you have any addons installed? If so, double check to see if any have excessive logging that would overfill the ram-based filesystem, possibly producing an out of memory crash.
January 25, 201115 yr Author Only add ons that I can think if that now is unmenu,notify, I think that pretty much covers it. I understand the rush current of spinning up the drives. I will try to spin them up and down tonight
January 25, 201115 yr Just a shot in the dark, but since it just happened to me, I'll mention it - How COLD is the room where your server is? I live in the northeast US and temps are really cold right now. My server crashed when I tried to start up all of the drives and it took a few reboots before I got things running properly again...just a thought.
January 25, 201115 yr Author Well mine is in a cold room to but the temps have warmed. The drives were down in the 10c range saturday and warmer on sunday. I wonder if there could be a link.
January 25, 201115 yr You should read my thread from earlier tonight: http://lime-technology.com/forum/index.php?topic=10479.0
January 25, 201115 yr Author Man I hope that's all it is. for the short term I will leave them spun up. Long term need to find a method to control the fans when its cold like this. Would be great to spin them down when the temps on the drives are to cold.
January 25, 201115 yr Author Parity check still running this morning no errors yet 700 min to go. All drive temps were in the 20's parity was 30.
January 25, 201115 yr I don't know for sure what temps might be right but single digits seems to cold to me, even when spun down. My drives start at a temp of around 20C from a spun down state. Once they are all going for a little while they hover around the 30C mark. If a full parity check is going on I will see a couple of my drives just peak above 40C. My case happens to be in a "computer nook" in a computer desk, so it is not the best place for proper hot air ventilation.
January 26, 201115 yr Author Parity ck completed yesterday at 11 am with no errors. Server was set the previous night to not spin down drive. watched movies various things with no problems transfered 100+ gb to it yesterday mover completed no problems. All drives were in the mid to high 20's this morning, spun them down and tried to spin them back up and crash.
January 26, 201115 yr Parity ck completed yesterday at 11 am with no errors. Server was set the previous night to not spin down drive. watched movies various things with no problems transfered 100+ gb to it yesterday mover completed no problems. All drives were in the mid to high 20's this morning, spun them down and tried to spin them back up and crash. That sounds like a power supply problem (could have gone bad) and my bet is on that. The only other thing I can think of is the Rosewill and the SASLP not playing nice together (or the gigabyte board not liking it). I would first remove the Rosewill card and connect all drives to the SASLP and do your tests again.
January 26, 201115 yr Author Here is a log of the spin down then up note it did not crash on this instance spin_down_spin_up.txt
January 26, 201115 yr Here is a log of the spin down then up note it did not crash on this instance Try doing the above things I mentioned to see if there is a conflict between the cards. It might not be fun, but spinning up your drives should not be causing this.
January 26, 201115 yr Author checked power supply never dropped below 12v on the single 12v rail and also 5v volt never dropped . removed the rosewill card leaving only the supermicro card. Still doing the same thing on spinup. Jan 26 11:14:33 Tower login[4222]: ROOT LOGIN on `pts/0' from `192.168.0.199' Jan 26 11:14:44 Tower emhttp: Spinning down all drives... Jan 26 11:14:44 Tower kernel: mdcmd (26): spindown 0 Jan 26 11:14:45 Tower kernel: mdcmd (27): spindown 1 Jan 26 11:14:46 Tower kernel: mdcmd (28): spindown 2 Jan 26 11:14:47 Tower kernel: mdcmd (29): spindown 3 Jan 26 11:14:48 Tower kernel: mdcmd (30): spindown 4 Jan 26 11:14:49 Tower kernel: mdcmd (31): spindown 5 Jan 26 11:14:49 Tower kernel: mdcmd (32): spindown 6 Jan 26 11:14:51 Tower kernel: mdcmd (33): spindown 7 Jan 26 11:14:52 Tower kernel: mdcmd (34): spindown 8 Jan 26 11:14:53 Tower kernel: mdcmd (35): spindown 9 Jan 26 11:14:54 Tower kernel: mdcmd (36): spindown 10 Jan 26 11:14:55 Tower emhttp: shcmd (26): /usr/sbin/hdparm -y /dev/sdc >/dev/null Jan 26 11:15:07 Tower emhttp: Spinning up all drives... Jan 26 11:15:07 Tower emhttp: shcmd (27): /usr/sbin/hdparm -S0 /dev/sdc >/dev/null Jan 26 11:15:07 Tower kernel: mdcmd (37): spinup 0 Jan 26 11:15:07 Tower kernel: mdcmd (38): spinup 1 Jan 26 11:15:07 Tower kernel: mdcmd (39): spinup 2 Jan 26 11:15:07 Tower kernel: mdcmd (40): spinup 3 Jan 26 11:15:07 Tower kernel: mdcmd (41): spinup 4 Jan 26 11:15:07 Tower kernel: mdcmd (42): spinup 5 Jan 26 11:15:07 Tower kernel: mdcmd (43): spinup 6 Jan 26 11:15:07 Tower kernel: mdcmd (44): spinup 7 Jan 26 11:15:07 Tower kernel: mdcmd (45): spinup 8 Jan 26 11:15:07 Tower kernel: mdcmd (46): spinup 9 Jan 26 11:15:07 Tower kernel: mdcmd (47): spinup 10 Jan 26 11:15:14 Tower kernel: sas: command 0xc43f33c0, task 0xc4217e00, timed out: BLK_EH_NOT_HANDLED Should I be looking at the (sdc) drive as a possible problem maybe try moving it to a different spot and see if the problem follows
January 26, 201115 yr Author I swapped that drive to a different port and I still get the problem (sdc) The drive is a WD 1TB EADS
January 26, 201115 yr Ditch the PCI card for now as the three disk on this one are causing you the slow parity checks. Then post your complete syslog.
January 26, 201115 yr Author Well I'm not doing parity checks as of now, and the pci card has been out, parity is good. I think something may be up with that one drive.
January 26, 201115 yr Author Problem has been found. Something is up with the cache drive. I removed it from the system after trying various tests. To include trying to spin each drive up and down separately all worked except for that one. Now to determine if it is in fact a drive or something else possibly with the software. What do you suggest as a plan of attack, I'm contemplating putting the said drive into the array and seeing if it acts the same.
January 26, 201115 yr It still could be Power Supply related. You removed 1 drive's worth of power draw from the PSU. Before you might have been borderline, now with 1 less drive you may be under the threshold. If you have yet another drive you can spare for testing, connect it to the system and try the spin-down/spin-up tests again. If the new drive is fine and it crashes, your issue is likely PSU related.
Archived
This topic is now archived and is closed to further replies.