October 28, 201411 yr I've been running beta10a since it was released. 2 days ago I got some drive issues resolved, and decided to start trying to create a windows VM. I installed dmacias's webvirt manager plugin and libvert plugin. everything was going fine, and I was making progress towards getting a windows 7 VM running, but ran into trouble when the install failed due to a problem with the windows iso I was using. since then unRAID becomes unresponsive after a couple hours. I lose all access to the server, the GUI, the shares, putty; all unavailable. I hard booted, then got back into the GUI, and turned off the new plugins, but it became unresponsive again. I had been running the XEN version for the last several weeks, but before starting on the windows VM process, I switched to using the KVM version of unRAID. I will have to hard boot again, then open the log window until it crashes, and report back. I just wanted to report the issue in case anyone else is seeing the same thing.
October 28, 201411 yr So I think I have to formally renounce my title of "bug magnet" and pass it to you. I just cannot compete with your bug finding ways. Well done sir well done.
October 28, 201411 yr Author Wow, I wish I could say I'm honored to receive such an honor, but I pretty feel exactly the opposite. I'm so tired of having issues, and all seem pretty unrelated. It feels like one step forward, 2 punches in the face. :'(
October 28, 201411 yr I feel with you. You seem to have bad luck from day one. Maybe you are cursed or something.. [emoji26]
October 28, 201411 yr Author I hard booted again, was able to access everything fine for a little while. I opened the logging window, which worked fine, then went and stopped all the running dockers. I went to work on something else for a bit, and when I looked again, unRAID was totally frozen. the logging window also went blank, so I have no syslog info to provide to help diagnose this. What can I do to get to the bottom of this issue? my server is useless right now
October 28, 201411 yr I hard booted again, was able to access everything fine for a little while. I opened the logging window, which worked fine, then went and stopped all the running dockers. I went to work on something else for a bit, and when I looked again, unRAID was totally frozen. the logging window also went blank, so I have no syslog info to provide to help diagnose this. What can I do to get to the bottom of this issue? my server is useless right now Justin try running the tail syslog on putty.
October 28, 201411 yr Hmmmmm... I logged in to report the same. My unRAID 6.0 beta 10a keeps HARD CRASHING where keyboard wont respond on console, or anything... Have to press RESET button to get it back up and it starts a parity check every time I do that. I'm killing my drives here... hehe BTW, I have 3 VM's running: Windows 8, Windows 2008 R2 and Ubuntu 14.04 1 docker running: AirVideo 12 plugins (including unRAID Server OS): transmission.64bit, subsonic.64bit, sickbeard.64bit, serviio.64bit, sabnzbd.64bit, Powerdown Package, plexmediaserver.64bit, headphones.64bit, Docker Manager, couchpotato.64bit, APC UPS support, unRAID Server OS How can I check previous log files to see if there is any messages I should be aware of? I have had 0 errors since I started using unRAID (I did have a bad Sata cable that reported errors but doing a SMART check said drive was perfect, replaced cable and it's been rock solid until now that I upgraded to unRAID 6 beta 10a) Not sure what could be causing this...
October 28, 201411 yr So with this tail command I can redirect it to a file in my array and review it later upon reboot?
October 28, 201411 yr Author I ran the command, in putty, and it's adding lines in the window as things happen. I would like to assume this information will not be erased, or removed when the server dies again, but of course, now that I'm 'prepared' to catch the problem, my server is still running okay. now that I'm thinking about it, I wonder if the webvirt and virtmanager plugins being enabled at startup might have something to do with it. Everything was fine until I tried using them, then even after doing so, it remained good, but at some point (can't remember if I rebooted prior) I lost GUI and the server. I hard booted a few times, and kept losing the server. i turned them off, but still lost the server, then hard booted again, with them disabled, and so far, the server is still doing well. time will tell.
October 28, 201411 yr Let me see how much buffer I can set on putty and I will leave it running as well... EDIT: I enabled logging of All of the session. Will report back tomorrow or whenever it crashes again.
October 29, 201411 yr Well, it didnt take long to crash unRAID. Casting Plex from Android to a Chromecast crashed it in less than 5 mins. syslig has nothing useful. Oct 28 19:32:42 MMPC kernel: br0: port 6(vif3.0-emu) entered disabled state Oct 28 19:33:20 MMPC kernel: mdcmd (58): nocheck Oct 28 19:33:20 MMPC kernel: md: md_do_sync: got signal, exit... Oct 28 19:33:20 MMPC kernel: md: recovery thread sync completion status: -4 Oct 28 19:33:29 MMPC kernel: br0: port 5(vif3.0) entered listening state Oct 28 19:33:29 MMPC kernel: br0: port 5(vif3.0) entered listening state Oct 28 19:33:44 MMPC kernel: br0: port 5(vif3.0) entered learning state Oct 28 19:33:59 MMPC kernel: br0: topology change detected, propagating Oct 28 19:33:59 MMPC kernel: br0: port 5(vif3.0) entered forwarding state Oct 28 19:59:28 MMPC sshd[20008]: Accepted password for root from 192.168.25.8 port 1557 ssh2 Oct 28 20:47:33 MMPC kernel: mdcmd (59): spindown 2 Oct 28 20:47:44 MMPC kernel: mdcmd (60): spindown 12 Oct 28 20:47:55 MMPC kernel: mdcmd (61): spindown 10 Oct 28 20:48:16 MMPC kernel: mdcmd (62): spindown 3 Oct 28 20:48:16 MMPC kernel: mdcmd (63): spindown 4 Oct 28 20:48:27 MMPC kernel: mdcmd (64): spindown 5 Oct 28 20:48:28 MMPC kernel: mdcmd (65): spindown 6 Oct 28 20:48:28 MMPC kernel: mdcmd (66): spindown 7 Oct 28 20:48:29 MMPC kernel: mdcmd (67): spindown 9 Oct 28 20:48:29 MMPC kernel: mdcmd (68): spindown 11 Oct 28 20:52:00 MMPC kernel: mdcmd (69): spindown 0 Oct 28 20:52:00 MMPC kernel: mdcmd (70): spindown 13 Oct 28 20:52:40 MMPC kernel: mdcmd (71): spindown 1 Oct 28 20:52:41 MMPC kernel: mdcmd (72): spindown 8 Oct 28 21:18:37 MMPC emhttp: sendFile: sendfile /usr/local/emhttp/plugins/vendor/shadowbox/shadowbox.js: Broken pipe Oct 28 21:18:38 MMPC emhttp: sendFile: sendfile /usr/local/emhttp/plugins/vendor/jquery/jquery-1.10.2.min.js: Broken pipe Oct 28 21:19:44 MMPC emhttp: sendFile: sendfile /usr/local/emhttp/plugins/webGui/images/banner.jpg: Broken pipe Oct 28 21:22:24 MMPC emhttp: sendFile: sendfile /usr/local/emhttp/plugins/webGui/images/banner.jpg: Broken pipe Oct 28 21:22:24 MMPC kernel: mdcmd (73): clear Oct 28 21:22:24 MMPC kernel: mdcmd (74): clear
October 29, 201411 yr Author it took me a bit longer. I decided to try installing windows into a new VM, and after one false start, then making changes to the xml and trying again, windows started installing. it got to at least 66% installed, and when I went to look on the progress, the window was unavailable. then I checked and the server is unavailable also. Here is the log from my putty session. It also seems to be short on clues... root@media:~# tail -f /var/log/syslog Oct 28 18:34:26 media emhttp: shcmd (79): /etc/rc.d/rc.atalk status Oct 28 18:34:26 media emhttp: shcmd (80): /usr/local/sbin/emhttp_event svcs_restarted Oct 28 18:34:26 media emhttp_event: svcs_restarted Oct 28 18:34:26 media emhttp: shcmd (81): /usr/local/sbin/emhttp_event started Oct 28 18:34:26 media emhttp_event: started Oct 28 18:34:36 media kernel: docker0: port 1(veth1bc3) entered forwarding state Oct 28 18:34:37 media kernel: docker0: port 2(veth36f9) entered forwarding state Oct 28 18:34:37 media kernel: docker0: port 3(veth477f) entered forwarding state Oct 28 18:34:38 media kernel: docker0: port 4(veth1e56) entered forwarding state Oct 28 18:51:49 media sshd[7495]: Accepted password for root from 192.168.20.134 port 52039 ssh2 Oct 28 19:13:18 media php: /usr/bin/docker stop MariaDB Oct 28 19:13:20 media php: MariaDB Oct 28 19:13:20 media php: Oct 28 19:13:20 media kernel: docker0: port 4(veth1e56) entered disabled state Oct 28 19:13:20 media kernel: device veth1e56 left promiscuous mode Oct 28 19:13:20 media kernel: docker0: port 4(veth1e56) entered disabled state Oct 28 19:13:23 media php: /usr/bin/docker stop SABnzbd Oct 28 19:13:24 media php: SABnzbd Oct 28 19:13:24 media php: Oct 28 19:13:24 media kernel: docker0: port 2(veth36f9) entered disabled state Oct 28 19:13:24 media kernel: device veth36f9 left promiscuous mode Oct 28 19:13:24 media kernel: docker0: port 2(veth36f9) entered disabled state Oct 28 19:13:30 media php: /usr/bin/docker stop ownCloud Oct 28 19:13:31 media php: ownCloud Oct 28 19:13:31 media php: Oct 28 19:13:31 media kernel: docker0: port 3(veth477f) entered disabled state Oct 28 19:13:31 media kernel: device veth477f left promiscuous mode Oct 28 19:13:31 media kernel: docker0: port 3(veth477f) entered disabled state Oct 28 19:13:35 media php: /usr/bin/docker stop NzbDrone Oct 28 19:13:35 media php: NzbDrone Oct 28 19:13:35 media php: Oct 28 19:13:35 media kernel: docker0: port 1(veth1bc3) entered disabled state Oct 28 19:13:35 media kernel: device veth1bc3 left promiscuous mode Oct 28 19:13:35 media kernel: docker0: port 1(veth1bc3) entered disabled state Oct 28 19:49:00 media emhttp: shcmd (82): /usr/sbin/hdparm -y /dev/sdb &> /dev/null Oct 28 20:54:58 media php: /etc/rc.d/rc.libvirt start Oct 28 20:54:58 media php: Starting libvirtd... Oct 28 20:54:58 media kernel: tun: Universal TUN/TAP device driver, 1.6 Oct 28 20:54:58 media kernel: tun: (C) 1999-2004 Max Krasnyansky <[email protected]> Oct 28 20:54:58 media php: Oct 28 20:55:21 media php: /etc/rc.d/rc.webvirtmgr buttonstart Oct 28 20:55:21 media php: Starting Web Virtual Manager Oct 28 20:55:22 media sudo: root : TTY=unknown ; PWD=/usr/local/emhttp ; USER=nobody ; COMMAND=/bin/bash -c nohup /usr/bin/python /mnt/cache/appdata/webvirtmgr/manage.py runserver 0:8888 > /dev/null 2>&1 & Oct 28 20:55:24 media php: ... Start OK Oct 28 20:55:25 media php: Oct 28 20:56:06 media kernel: device vnet0 entered promiscuous mode Oct 28 20:56:06 media kernel: br0: port 2(vnet0) entered forwarding state Oct 28 20:56:06 media kernel: br0: port 2(vnet0) entered forwarding state Oct 28 20:56:11 media kernel: kvm: zapping shadow pages for mmio generation wraparound Oct 28 20:56:21 media kernel: br0: port 2(vnet0) entered forwarding state Oct 28 21:00:29 media kernel: br0: port 2(vnet0) entered disabled state Oct 28 21:00:29 media kernel: device vnet0 left promiscuous mode Oct 28 21:00:29 media kernel: br0: port 2(vnet0) entered disabled state Oct 28 21:22:46 media kernel: device vnet0 entered promiscuous mode Oct 28 21:22:46 media kernel: br0: port 2(vnet0) entered forwarding state Oct 28 21:22:46 media kernel: br0: port 2(vnet0) entered forwarding state Oct 28 21:22:50 media kernel: kvm: zapping shadow pages for mmio generation wraparound Oct 28 21:23:01 media kernel: br0: port 2(vnet0) entered forwarding state
October 29, 201411 yr Is anybody looking into this????? Im going to disable my 3 VM's and leave unRAID running by itself and try casting to a chromecast via my Plex app on my android phone remotely from another location (which is what seems to cause mine to lockup) I left it running overnight without casting and it downloaded a few tv shows and a movie (couchpotato, sickbeard and sabnzbd worked for the entire day).... I guess we're on our own... no syslog, no dmesg, no nothing to help troubleshoot... just wonderful... Can you post the plugins, dockers and VM's you are running on your system? Maybe we can narrow it down to like a plugin or something? EDIT: I just disabled XBoost off on my ASRock motherboard. I know that might not have any effect but it would be 1 less thing on the mix.... I'll give it a spin for a bit and see if it crashes like it did before.
October 30, 201411 yr Author Well, it doesn't seem to be a priority for LimeTech at the moment. Hopefully they are going to release a new beta that they think/hope will resolve this issue. As for me, I've uninstalled the WebVirtMan plugin and installed the vmMan plugin instead, but that still didn't fix this issue. Then, I uninstalled all my plugins, except for docker, and I also removed this part of my syslinux.cfg file (pcie_acs_override=downstream); which I had added based on jonp's suggestion in another thread, saying it must be in the syslinux.cfg in order to do passthru to a VM. I've asked in a couple of other threads if others have this in their syslinux files, but so far, no one has responded. Do you have this in your syslinux.cfg file? The only other plugins I had installed were apcuspd and powerdown. Everything else I had running in dockers. I had those plugins installed for quite some time before having these issues, so I suspect they are not the problem either. However, it is possible they conflict in some way with the vmMan or virtMan plugins, and/or the section above from the syslinux file. After removing that section and the plugins, I've rebooted, and installed the virtMan plugin, since it's not really supposed to "do" anything, and so far, no more crashes. I'm currently rebuilding/customizing my Windows7 image so I can try installing it later. Please post back about the syslinux stuff and plugins you may be using on your machine. Hopefully we can find some common ground to help isolate the cause of this issue. thanks.
October 30, 201411 yr We have nothing to go on from these posts and cannot replicate the issue. Until we get more info in the form of steps to replicate, we can't really help. Know that we are following this thread though.
October 30, 201411 yr Author I appreciate the feedback. I wish I could give you more info, but at this point I think I've already provided all the information I can. However, I will try to summarize, in case that brings something to the fore. I have docker plugin installed, on disk1, 10GB I have SABnzbd, SickRage, nzbDrone, ownCloud, mariaDB and deluge dockers installed, deluge and SickRage are not set to auto start, the others are. I have since changed that so that none of them auto start, and all are not running. I have apcuspd and powerdown plugins installed. I have screen-4.0.3-x86_64-4 and utempter-1.1.5-x86_64-1 in the Extras folder. All of that is pretty consistent, and has been that way for a couple of weeks, but was all running under the XEN install option. A few days ago, i changed to installing under KVM so I could try to get a windows VM installed. After changing to booting KVM mode, I then installed WebVirtMan and virtMan plugins. In my research I saw a post from jonp saying that the only thing that needed to be added to the syslinux.cfg file to get passthru working was "pcie_acs_override=downstream", so I changed my syslinux to match exactly (copy/paste) that of jonp. All of those changes happened in a short time (minutes/hours). Since making those changes, this issue started happening. I removed the WebVirtMan plugin, and installed the vmMan plugin, but the issue persisted. I cannot honestly remember if I rebooted after making that change, but I don't think I did, so perhaps there was something persistent/left behind from WebVirtMan, I'm not sure. After the last time it hung/crashed, i removed the USB, removed all the plugins and the extras and removed the section from the syslinux.cfg file. I then booted into the KVM version again, and installed the virtMan plugin only. That was several hours ago, and the server is still working fine. I'm going to install the vmMan plugin in a little bit, and see if that changes anything. I don't plan to make any more changes tonight. As I said before, I have been unable to capture any logging, or other information which looked helpful. I've provided all I could capture so far. If there is anything at all you want me to do or test, please just let me know. I'm out of ideas at this point.
October 30, 201411 yr In my research I saw a post from jonp saying that the only thing that needed to be added to the syslinux.cfg file to get passthru working was "pcie_acs_override=downstream", so I changed my syslinux to match exactly (copy/paste) that of jonp. I recall this mention, and I also think it can be a bit misleading (I will speak without being any type of expert on this in anyway). From my reading on this flag prior to it being included in a beta a couple of versions prior it assumes exclusions from other devices and ACS support on the device, however it does not know that is the case. The main need for this is to get devices into their own C groups and then allow for pass-thru. However using this can lead to an issue that is assumed exclusion between another device, and that not being the case and causing issues. "Therefore, we need to add some flexibility. The pcie_acs_override=boot option lets users opt-in specific devices or sets of devices to assume ACS support. The "downstream" option assumes full ACS support on root ports and downstream switch ports." "It's suggested to use the most limited set of options necessary to avoid completely disabling ACS across the topology." Article here: https://lkml.org/lkml/2013/5/30/513 Now I only have that in my syslinux.cfg and I am having no issues, so I don't think it is universally a wrong thing however it may be causing your specific setup issues.
October 30, 201411 yr Can you boot into safe mode (no plugins)? We need to narrow down the issue. You have too many variables in your setup for us narrow down the cause.
October 30, 201411 yr EDIT: I just disabled XBoost off on my ASRock motherboard. I know that might not have any effect but it would be 1 less thing on the mix.... I'll give it a spin for a bit and see if it crashes like it did before. This solved my issue. I was overclocking my cpu from 4GHz to 4.5GHz but under info and on the VM cpu info it was only showing 4GHz, so turning off Xboost on my ASRock mobo fixed the crashes. Are you o/c'ing? If you are try not to and see if that helps....
October 30, 201411 yr In my research I saw a post from jonp saying that the only thing that needed to be added to the syslinux.cfg file to get passthru working was "pcie_acs_override=downstream", so I changed my syslinux to match exactly (copy/paste) that of jonp. I recall this mention, and I also think it can be a bit misleading (I will speak without being any type of expert on this in anyway). From my reading on this flag prior to it being included in a beta a couple of versions prior it assumes exclusions from other devices and ACS support on the device, however it does not know that is the case. The main need for this is to get devices into their own C groups and then allow for pass-thru. However using this can lead to an issue that is assumed exclusion between another device, and that not being the case and causing issues. "Therefore, we need to add some flexibility. The pcie_acs_override=boot option lets users opt-in specific devices or sets of devices to assume ACS support. The "downstream" option assumes full ACS support on root ports and downstream switch ports." "It's suggested to use the most limited set of options necessary to avoid completely disabling ACS across the topology." Article here: https://lkml.org/lkml/2013/5/30/513 Now I only have that in my syslinux.cfg and I am having no issues, so I don't think it is universally a wrong thing however it may be causing your specific setup issues. There is another option for the pcie_acs_override setting which is "multifunction" and it will only apply the ACS overrides to multifunction devices. Example: Almost all GPUs nowadays have an audio processor on them and represent that audio device as a "function" of the parent PCI device. In lspci, this looks like so: 01:00.0 VGA compatible controller: AMD/ATI [Advanced Micro Devices, Inc.] RV710 [Radeon HD 4350/4550] 01:00.1 Audio device: AMD/ATI [Advanced Micro Devices, Inc.] RV710/730 HDMI Audio [Radeon HD 4000 series] 02:00.0 VGA compatible controller: NVIDIA Corporation Device 1004 (rev a1) 02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio [GeForce GTX Titan] (rev a1) So for devices that DON'T have multi-function, this wouldn't apply (e.g. SATA controllers, USB controllers, etc.). The "boot" option instead of downstream or multifunction is more complicated in that you have to specify the specific devices in the syslinux append line that you want to enable for ACS. This would introduce another level of complexity for folks, but may be necessary to support various hardware configurations and pass-through scenarios. Ok, so what do we do now? Justin / any others having this issue: please adjust your setting from downstream to multifunction and report back if the issues continue. - Jon
October 30, 201411 yr Author My status at this point is this. I've removed the section from the syslinux completely, I'm not starting any dockers, I've not installed the apcupsd or the powerdown plugins, but I did install the vmMan plugin. I spend a couple hours last night trying to get my windows VM installed, and eventually succeeded. I let it install overnight, and am currently installing updates. So far everything is working fine. EDIT: I just disabled XBoost off on my ASRock motherboard. I know that might not have any effect but it would be 1 less thing on the mix.... I'll give it a spin for a bit and see if it crashes like it did before. This solved my issue. I was overclocking my cpu from 4GHz to 4.5GHz but under info and on the VM cpu info it was only showing 4GHz, so turning off Xboost on my ASRock mobo fixed the crashes. Are you o/c'ing? If you are try not to and see if that helps.... I don't recall enabling that setting, and am not overclocking anything in the BIOS. Next time I reboot the server, I'll go into BIOS and check that setting. Can you boot into safe mode (no plugins)? We need to narrow down the issue. You have too many variables in your setup for us narrow down the cause. I think this might not be necessary at this point, since I've eliminated the problem with the steps above. There is another option for the pcie_acs_override setting which is "multifunction" and it will only apply the ACS overrides to multifunction devices. Ok, so what do we do now? Justin / any others having this issue: please adjust your setting from downstream to multifunction and report back if the issues continue. - Jon I'll add that back after windows finishes updating, then reboot. Just to confirm, this is what I need in the syslinux.cfg label unRAID OS menu default kernel /bzimage append pcie_acs_override=multifunction initrd=/bzroot exactly like that, correct?
October 30, 201411 yr I'll add that back after windows finishes updating, then reboot. Just to confirm, this is what I need in the syslinux.cfg label unRAID OS menu default kernel /bzimage append pcie_acs_override=multifunction initrd=/bzroot exactly like that, correct? Correct.
Archived
This topic is now archived and is closed to further replies.