October 12, 201411 yr Please everyone don't take this the wrong way: but really, it does absolutely no good to say "this config exhibits the issue, this config doesn't". The only way this is going to be solved is you MUST post a system log coincident with the problem occurring. There is no other way to debug this. Almost certainly this is going to be an issue with a disk driver.
October 12, 201411 yr I usually see 2 different results from the system. The one is attached which shows errors from the system related to the controller. Other times I don't see a single error from a tail of syslog and the machine becomes un-reachable and the console is blank so I can't see any errors and I have to do a power button reset to get the server to restart. I can try rebooting back to non-Xen tonight and run a parity check to try to get another log. Do you have any suggestions on any additional data capture I can do other than the tail of syslog? syslog.09232014.zip
October 12, 201411 yr I usually see 2 different results from the system. The one is attached which shows errors from the system related to the controller. Other times I don't see a single error from a tail of syslog and the machine becomes un-reachable and the console is blank so I can't see any errors and I have to do a power button reset to get the server to restart. I can try rebooting back to non-Xen tonight and run a parity check to try to get another log. Do you have any suggestions on any additional data capture I can do other than the tail of syslog? Couple things: first you're running -beta9, please always use the latest beta, currently -beta10a. Second: there are massive disk error messages in the system log, no wonder parity sync/correct does not work. Please check all your hardware: cables, controller seating, disk seating, etc. Very well could be a bad or inadequate PSU as well.
October 12, 201411 yr Sorry about that, I thought that was a more recent syslog. I must not have saved any of the newer ones. Those disk errors only occurred under the non-Xen boot, when I boot under Xen, I don't see any of those disk errors. Let me reboot over and run a parity check tonight and get a new log..
October 13, 201411 yr On the second parity check, but now seeing the other issue I had after upgrading to 10a, it corrects the same parity error on each run. Usually if I reboot under Xen, it runs without any parity differences. mdcmd (45): check CORRECT md: recovery thread woken up ... md: recovery thread checking parity... md: using 3584k window, over a total of 3907018532 blocks. md: correcting parity, sector=390025208 [sched_delayed] 4CE: hpet increased min_delta_ns to 11521 nsec mdcmd (46): spindown 2 mdcmd (47): spindown 3 mdcmd (48): spindown 4 mdcmd (49): spindown 6 mdcmd (50): spindown 7 md: sync done. time=45083sec md: recovery thread sync completion status: 0 mdcmd (51): check CORRECT md: recovery thread woken up ... md: recovery thread checking parity... md: using 3584k window, over a total of 3907018532 blocks. md: correcting parity, sector=390025208 Ill post the full syslog once it completes.
October 13, 201411 yr xen/non-xen shouldn't make any difference. If it did, lots more people would be reporting problems. I would suspect your PSU. There could be a power draw difference in xen vs non-xen and perhaps that server is right on the edge. Maybe remove a couple hard drives and see if issue persists.
October 13, 201411 yr bkastner had the same issue: http://lime-technology.com/forum/index.php?topic=35659.0 Once the parity check is completed, I can try migrating data to another drive and remove a drive to see what happens...
December 2, 201411 yr Ran the test under beta12. Running a parity check under non-Xen, after about 2 hours I lost connectivity and the tail of syslog from ssh shows no errors, and console is dark and had to hard power cycle to reboot. Booted up under Xen, and parity check ended without any issues. Looks like whatever the cause of the issue is still present with beta12 or some interaction with unraid and the hardware.
December 2, 201411 yr Ran the test under beta12. Running a parity check under non-Xen, after about 2 hours I lost connectivity and the tail of syslog from ssh shows no errors, and console is dark and had to hard power cycle to reboot. Booted up under Xen, and parity check ended without any issues. Looks like whatever the cause of the issue is still present with beta12 or some interaction with unraid and the hardware. Thanks for testing this again. Still not sure what's causing this and can't replicate this issue on our test systems. We will keep looking into this though.
December 2, 201411 yr Thanks for keeping this on your radar, I know it has been difficult to diagnose...
December 11, 201411 yr Other than jphipps, has anyone else seen this issue on non-Xen boot mode with beta12? We need to ascertain how wide-spread this issue may be or if it could be a one-off with specific hardware. For those having this issues, please post your current syslinux.cfg here. In addition, please move /etc/rsyslog.conf to /boot/config/rsyslog.conf and modify the configuration file to point your syslog to a persistent storage location. This way when the host crashes and you reboot, you should be able to get the previous syslog in its state JUST before the crash.
December 12, 201411 yr Please try the CPU freq fix provided by Eric and let me know if this issue persists. See here: In KVM mode, there might be a cpu scaling driver issue with certain hardware combinations. One of those drivers is called Intel-Pstate. This is the chosen driver if your Intel cpu is Sandy Bridge (2011) or newer. On my Haswell-class cpu (i7-4771) the Intel-Pstate driver is too sensitive and seems to keep the cpu frequency near the max frequency even when idle but occasionally it does scale the frequency down. You can disable the Intel-Pstate driver by editing your /boot/syslinux/syslinux.cfg and adding a intel_pstate=disable parameter to the append line below: ... label unRAID OS menu default kernel /bzimage append intel_pstate=disable initrd=/bzroot ... Save the file, stop the array and reboot unRAID. Doing this on my Haswell machine caused it to use the acpi-cpufreq scaling driver instead of the intel_pstate one. It scales the frequency down like a rockstar now! Usually keeps it around 800MHz - 1000MHz during idle now. On the flip side, my other test machine, a year older Intel cpu (i5-3470) was able to scale down to 1600MHz (minimum) pretty consistently when using the intel_pstate driver... but when I disabled intel_pstate then there wasn't a scaling driver available. For some reason the acpi-cpufreq driver wasn't compatible with this cpu. Your mileage may very. Give this a try and let me know if it helps you. Either way, if it helped or not, let me know which cpu you tried with this command: grep -m 1 'model name' < /proc/cpuinfo
January 20, 201511 yr Please try the CPU freq fix provided by Eric and let me know if this issue persists. See here: In KVM mode, there might be a cpu scaling driver issue with certain hardware combinations. One of those drivers is called Intel-Pstate. This is the chosen driver if your Intel cpu is Sandy Bridge (2011) or newer. On my Haswell-class cpu (i7-4771) the Intel-Pstate driver is too sensitive and seems to keep the cpu frequency near the max frequency even when idle but occasionally it does scale the frequency down. You can disable the Intel-Pstate driver by editing your /boot/syslinux/syslinux.cfg and adding a intel_pstate=disable parameter to the append line below: ... label unRAID OS menu default kernel /bzimage append intel_pstate=disable initrd=/bzroot ... Save the file, stop the array and reboot unRAID. Doing this on my Haswell machine caused it to use the acpi-cpufreq scaling driver instead of the intel_pstate one. It scales the frequency down like a rockstar now! Usually keeps it around 800MHz - 1000MHz during idle now. On the flip side, my other test machine, a year older Intel cpu (i5-3470) was able to scale down to 1600MHz (minimum) pretty consistently when using the intel_pstate driver... but when I disabled intel_pstate then there wasn't a scaling driver available. For some reason the acpi-cpufreq driver wasn't compatible with this cpu. Your mileage may very. Give this a try and let me know if it helps you. Either way, if it helped or not, let me know which cpu you tried with this command: grep -m 1 'model name' < /proc/cpuinfo Just checking in with folks to see if this issue still persists.
January 20, 201511 yr Author I must have missed this suggestion previously. I have made the necessary changes, rebooted and am starting a parity check now. I will post the results. On an un-related note, I did confirm that after rebooting the Dynamix Scheduler did not auto-start (again) so I had to manually set it so that the parity check will hopefully kick off normally on the 1st of Feb.
January 21, 201511 yr Author Okay, so I finished a parity check on non-Xen with no issues. However... I had forgot to allow my 6TB Red disk to spin down. Since it has completed this morning I changed the settings on the Red disk to spin down based on default settings, forced a spin down of all drives and kicked off a parity check. I am 0.8% done and have the 128 write errors to the parity disk, so I would say there is still an issue.
January 23, 201511 yr Must have missed this post... Not sure it would apply to the machine I am having an issue with since it is AMD? model name : AMD Athlon 64 X2 Dual Core Processor 5600+ Would this still apply?
January 27, 201511 yr Must have missed this post... Not sure it would apply to the machine I am having an issue with since it is AMD? model name : AMD Athlon 64 X2 Dual Core Processor 5600+ Would this still apply? Nah, the pstate fix was Intel-specific.
Archived
This topic is now archived and is closed to further replies.