June 24, 201313 yr Hi, I'm encountering unclean shutdown on my Unraid server. One instance I have seen this happen is when the mover is moving files from the cache to the data disk but I have also seen it when copying files to my cache drive. The problem I see is 1) I will copy files onto the array using the cache drive. 2) The mover attempts to do its thing overnight (I forced the mover to start last night instead of waiting till 3:40am). 3) The next morning I check the web UI for the server and it shows unclean shutdown detected. 4) I start the array and do a parity check and it reports sync errors but parity check completes System info Unraid Pro 5.0 RC 12a 1 3TB WD Green Parity drive (EARX or EZRX - cant remember the name) 1 2TB WD Green cache drive (EARX or EZRX - cant remember the name) 1 2TB WD Green data drive (EARX or EZRX - cant remember the name) 8GB Ram ASROCK H77 Pro MVP4 Mobo Parity drive powered from PSU directly Cache and Data drive powered in Norco SS-500 Plugins None installed Shares Movies - split level 2 - Most Free allocation method TV Shows - split level 2 - Most Free allocation method I have run preclear on all the drives before hand and did not notice any errors. (this was before transferring data to the array) Between finishing preclearing the drives and starting to transfer data to the array I did take the system down to install some cooling fans. I will take apart the server and check the cables and reseat them all tonight. Should I run memtest to rule out the RAM? How do I do this? From a telnet session? The SATA cables for the drives are all zip tied together. I will separate them incase there is some cross talk going on between the signals. Is there anything else I should try and run to collect information or debug this? I have uploaded 3 Syslogs (this is the chronological order that they were recorded in) 1) Syslog after previous unclean shutdown and parity check completed 2) Syslog after moving files to cache drive and just before invoking the mover 3) Syslog after unclean shutdown Thanks in advance for helping me with this problem Syslog_unclean_shutdown.zip
June 24, 201313 yr When you boot unRAID there will be an option to test memory, Select that let it run overnight. Is your unRAID server on a UPS? What PSU are you using?
June 24, 201313 yr Is there anything else I should try and run to collect information or debug this? Click on 'Log' button in webGui menu bar to bring up separate window with real-time syslog displayed. Make problem happen. Select/copy/paste contents of the log and post.
June 24, 201313 yr Author When you boot unRAID there will be an option to test memory, Select that let it run overnight. Is your unRAID server on a UPS? What PSU are you using? PSU is Corsair HX650 Gold Edition http://www.newegg.ca/Product/Product.aspx?Item=N82E16817139012 Originally the server was on one of those Costco APC surge protectors (shared with laptop, router, etc) but I moved it to a power bar with nothing else on it just to rule that out. I thought about a PSU issue but I thought the largest current draw is when all the drives spin up when the array is started. I've never see any problems when that happens. Could still be a loose cable as the supply is modular.
June 24, 201313 yr Author Is there anything else I should try and run to collect information or debug this? Click on 'Log' button in webGui menu bar to bring up separate window with real-time syslog displayed. Make problem happen. Select/copy/paste contents of the log and post. Thanks! I will do that tonight before I shutdown the array and check the wiring. I updated my OP to state how the drives were being powered. 2 of them are in a Norco SS-500.
June 27, 201313 yr Author I tried running the mover 2 or 3 times and could not reproduce the problem. I started memtest on the server and I see some weird behavior. I took a picture of the monitor while the self test was running (this was the screen about 5 or 10 mins into the test). The test stayed on this screen for a while (5 maybe 10 minutes) and then the server rebooted itself. Is there a smoking gun here? Memory or Mobo? I have not checked the wiring yet but will try that tomorrow. Thanks
June 27, 201313 yr That looks suspiciously like memory corruption. I'd pull all the memory cards and insert them into the first DIMM slot one at a time, running the memory test for each card. You should be able to identify the culprit that way.
June 30, 201313 yr Author After doing a whole bunch of testing I'm thinking it is my mother board but like a second opinion I tested both RAM sticks individually in slot 1 (my mobo has four slots) with Memtest and they ran for 8+ and 10+ hours without any errors. I put both sticks into the motherboard and ran the tests overnight and the system rebooted it self by morning. I put both sticks into the other 2 slots and ran the tests and that test run did not also last very long before it rebooted. What do people think the problem could be? The RAM sticks work individually but not when installed at the same time. That leads me to think it is the motherboard but there were no problems with just one stick installed. THe customer support for the RAM suggested I try and setup the RAM (which is low voltage - 1.25) to run at a higher voltage like 1.4 or 1.45. Thanks, Valens
June 30, 201313 yr After doing a whole bunch of testing I'm thinking it is my mother board but like a second opinion I tested both RAM sticks individually in slot 1 (my mobo has four slots) with Memtest and they ran for 8+ and 10+ hours without any errors. I put both sticks into the motherboard and ran the tests overnight and the system rebooted it self by morning. I put both sticks into the other 2 slots and ran the tests and that test run did not also last very long before it rebooted. What do people think the problem could be? The RAM sticks work individually but not when installed at the same time. That leads me to think it is the motherboard but there were no problems with just one stick installed. THe customer support for the RAM suggested I try and setup the RAM (which is low voltage - 1.25) to run at a higher voltage like 1.4 or 1.45. Thanks, Valens Almost certain it is the voltage. clock speed. or timing set by your BIOS for your RAM and the RAM itself is fine. Most BIOS will set the parameters correctly, but some will not. Certainly you may not just randomly populate ram sockets. If only populating some, they must be in pairs and in certain sockets. In any case, the parameters must be set for your specific make/model RAM strips. You may have to set them manually for your specific RAM.
June 30, 201313 yr RAM voltage is NOT an SPD parameter ... so your BIOS doesn't automatically adjust it for low voltage (or high voltage) modules => YOU have to do that. If your RAM modules are supposed to be set at 1.25v, and you've not adjusted that, then you've been running them at the standard DDR3 voltage of 1.5v -- a 20% over-volt !! I wouldn't be surprised if this has damaged one or both of the modules. You can try the following: (a) Set the voltage to 1.25v and see if the modules work okay (if so, breath a sign of relief) (b) Buy a pair of new RAM modules (I'd suggested standard 1.5v modules) and see if the fixes your problem. © Experiment with slowing down your current modules, which can sometimes help resolve issues like this ... add a cycle or two to the latency settings; reduce the clock speed; or both.
June 30, 201313 yr Author After doing a whole bunch of testing I'm thinking it is my mother board but like a second opinion I tested both RAM sticks individually in slot 1 (my mobo has four slots) with Memtest and they ran for 8+ and 10+ hours without any errors. I put both sticks into the motherboard and ran the tests overnight and the system rebooted it self by morning. I put both sticks into the other 2 slots and ran the tests and that test run did not also last very long before it rebooted. What do people think the problem could be? The RAM sticks work individually but not when installed at the same time. That leads me to think it is the motherboard but there were no problems with just one stick installed. THe customer support for the RAM suggested I try and setup the RAM (which is low voltage - 1.25) to run at a higher voltage like 1.4 or 1.45. Thanks, Valens Almost certain it is the voltage. clock speed. or timing set by your BIOS for your RAM and the RAM itself is fine. Most BIOS will set the parameters correctly, but some will not. Certainly you may not just randomly populate ram sockets. If only populating some, they must be in pairs and in certain sockets. In any case, the parameters must be set for your specific make/model RAM strips. You may have to set them manually for your specific RAM. Hi Joe, Not to worry I have been mindful of populating the RAM in the right sockets for all my testing I think I might be one of the few who reads an actual BIOS manual. RAM voltage is NOT an SPD parameter ... so your BIOS doesn't automatically adjust it for low voltage (or high voltage) modules => YOU have to do that. If your RAM modules are supposed to be set at 1.25v, and you've not adjusted that, then you've been running them at the standard DDR3 voltage of 1.5v -- a 20% over-volt !! I wouldn't be surprised if this has damaged one or both of the modules. You can try the following: (a) Set the voltage to 1.25v and see if the modules work okay (if so, breath a sign of relief) (b) Buy a pair of new RAM modules (I'd suggested standard 1.5v modules) and see if the fixes your problem. © Experiment with slowing down your current modules, which can sometimes help resolve issues like this ... add a cycle or two to the latency settings; reduce the clock speed; or both. Hi garycase, I looked over my BIOS settings and it was actually setting the DRAM voltage to 1.25V. The RAM I'm using is on the Asrock's approved memory list. The funny thing is that the Technical support at GSkill actually suggested increasing the RAM voltage to 1.4 or 1.45 V as they stated the RAM might be more stable. Kind of contradicts the advice you gave me but I decided to try it before I saw your response. They also suggesting loading the XMP defaults but I believe my BIOS has the XMP settings set to automatically load. The test has been running for 1.5hrs with a higher voltage so I'll continue to monitor it. I'll try forcing the XMP settings as well if this test does not work. Valens
July 3, 201313 yr Author I think my problem has been solved. Increasing the DRAM voltage to 1.4V (as per GSkill Tech Support suggestion) and adjusting one of the timing parameters has made my RAM (GSkill Sniper Low Voltage series) stable. I ran memtest for 19hrs without any errors and last night I did not see an unclean shutdown. I was seeing them every night before. Thanks to all the forum members who helped out!
Archived
This topic is now archived and is closed to further replies.