February 22, 20179 yr Hi All, I've been using Unraid for years without any issues that couldn't be resolved by searching on the board, but this is new to me. I normally have months of uptime. I updated to 6.3.2 last week (from 6.2.1, I think) Yesterday I noticed that the drives spun up without any activity. Whe I looked after a few hours it was running a Parity Check. While checking a few things in the UI it stopped responding and after a couple of minutes it came back online but had restarted the Parity Check. Not thinking I rebooted the server, and a Parity check was scheduled for last night so I left it to see what would happen. When I checked this afternoon a Parity check was running and I looked in the parity-checks.log and it had an "Unavailable" item from 7PM last night (see atttached). I saved diagnostics and the UI died again soon afterwards, restarting a few minutes later and running a new Parity check. I ran another diagnostic and I've attached that too. There are errors in one of the SMART logs - could this be the cause? Do I need to buy a new drive? I'm running these plugins: CA Auto Update Applications CA Backup / Restore Appdata CA Cleanup Appdata Community Applications Dynamix SSD TRIM Dynamix System Information Dynamix System Statistics Dynamix webGui Nerd Tools Preclear Disks SNAP non-array drive mount and share And these Dockers: Dropbox PlexMediaServer plexpy Transmission I was running CUPS Docker but recently stopped it. Thanks in advance for any assistance. tower-diagnostics-20170222-1508.zip tower-diagnostics-20170222-1500.zip parity-checks.log Edited February 23, 20179 yr by emmcee
February 22, 20179 yr Author OK - it seesm that the server is rebooting - not just UI being unresponsive. I ran FCP just now and it reported a numeber of issues, one of which was to run mcelog. I just tried to connect over telnet with Putty and I get Connection Refused. If I go into FCP and click Rescan then the server reboots. Edit: I've just noticeed that there are many instances of this in the system log: Tower root: error: plugins/fix.common.problems/include/fixExec.php: wrong csrf_token Edited February 22, 20179 yr by emmcee
February 22, 20179 yr Very early in your start-up sequence you have a MCE around the point where your system is switching over from running on a single CPU core to SMP: Quote Feb 22 14:14:58 Tower kernel: x86: Booting SMP configuration: Feb 22 14:14:58 Tower kernel: .... node #0, CPUs: #1 Feb 22 14:14:58 Tower kernel: mce: [Hardware Error]: Machine check events logged Feb 22 14:14:58 Tower kernel: #2 #3 #4 #5 #6 #7<6>x86: Booted up 1 node, 8 CPUs Feb 22 14:14:58 Tower kernel: smpboot: Total of 8 processors activated (52764.34 BogoMIPS) I'd reboot and select MemTest from the boot menu and check your RAM for 48 hours. If your RAM passes the test then install mcelog from the Nerd Tools package that you already have installed. It might reveal something. Look for a newer BIOS. The disk that showed errors recorded them 10,000 power-on hours ago (a third of their age) so they are probably not an issue any longer. The SMART parameters look good but your disks are too hot at 49 to 52 Celsius. You need to improve their cooling. Is the rest of your server running hot too?
February 22, 20179 yr Author Thanks for the tips. The drive temps have only spiked to that as it's constantly running parity checks. I'll see if I can increase the fan speed in the BIOS. And look for a BIOS update. Running mcelog is going to be difficult if I can't telnet.
February 22, 20179 yr 31 minutes ago, emmcee said: OK - it seesm that the server is rebooting - not just UI being unresponsive. I ran FCP just now and it reported a numeber of issues, one of which was to run mcelog. I just tried to connect over telnet with Putty and I get Connection Refused. If I go into FCP and click Rescan then the server reboots. Edit: I've just noticeed that there are many instances of this in the system log: Tower root: error: plugins/fix.common.problems/include/fixExec.php: wrong csrf_token Not gospel truth, but FCP causing the server to reboot on a scan implies power supply issues, since one of the tests (unless you disabled it) is to spin up all the drives to attempt writes to them The wrong csrf_token means that you've got browser windows still open and pointed to somewhere within unRaid's UI from before the restart. And FCP didn't suggest to run mcelog, it suggested to install it (via the NerdPack) Then restart, put FCP into troubleshooting mode, and use the server. Hopefully another mce error will get logged (with more information) before a restart (and hopefully FCP won't trigger the restart by going in and starting troubleshooting mode) At that point, post up the last diagnostics file generated within /logs on the flash drive (along with the FCPsyslog_tail.txt file) Unfortunately, if the mce is very early in the logs (as in before NerdPack installs), and does not repeat then no extra info will get logged for it. And definitely run a memtest as already suggested as its at least an easy test
February 22, 20179 yr Author OK, memtest is running. Not sure I can do 48 hours as it's in my office and with the fan cranked up it's no longer silent. Is there any way to control the system fan from within unRAID? BIOS is latest non beta version. PSU issues is interesting. I'm running a 160W PicoPSU but with a 120W transformer. Power meter ordered. I've always wanted one anyway!
February 22, 20179 yr Community Expert 15 minutes ago, emmcee said: I'm running a 160W PicoPSU Never heard of anyone trying that with unRAID. Not gonna be enough. Also, the SNAP plugin was deprecated a very long time ago and is incompatible with V6.1+. You should be using Unassigned Devices plugin.
February 22, 20179 yr Author I haven't had an issue with it up to now and It's been running 24/7 since it was built back in June. I calculated peak consumption based on the CPU/mobo and 5 HDD/1 SSD and there was overhead. I guess the only thing to do is measure it.
February 22, 20179 yr Thing is, you've got a hardware problem so it isn't really true to say that you haven't had an issue with it so far. The fault could be the PSU or RAM or something expensive - best to eliminate the easy and cheap things first.
February 22, 20179 yr Author Ok, so I now have a power meter. It peaks at 80W when spinning up all drives but normally hovers ~60W with drives up. With drives spin down it's ~30W. I normally run headless, but I have it connected to a monitor for memtest (completed one pass - no issues). When I restarted and tried to load the GUI from my PC I get a kernel panic - see attached. There was no spike in power usage at this point. Is is possible (or advisable) to roll back to 6.2.1 to see if it's a bug? Like I said I've had 24/7 uptime with this rig since June and only updated to 6.3.2 on Sunday. Edited February 22, 20179 yr by emmcee
February 22, 20179 yr Assuming you updated by means of the Plugin page of the web GUI you can roll back to the version you were using previously by copying the three bz* files from the previous folder on your boot device into the root of your boot device, overwriting the three files already there. cp /boot/previous/bz* /boot However your problem is really a hardware one.
February 23, 20179 yr 1 hour ago, emmcee said: Ok, so I now have a power meter. It peaks at 80W when spinning up all drives but normally hovers ~60W with drives up. With drives spin down it's ~30W. How close to the rated available power are you on the +12V leg? Total power draw is only a part of the picture, a power supply that small running 5 spinner HDD is going to be VERY close to the available amperage on the +12. I bet you are actually overloading that part of the PSU, causing dirty power to the motherboard when the drives are spun up. Try temporarily running with a normal ATX PSU hung out the side of the case and see if the symptoms continue.
February 23, 20179 yr Author 3 minutes ago, jonathanm said: How close to the rated available power are you on the +12V leg? Total power draw is only a part of the picture, a power supply that small running 5 spinner HDD is going to be VERY close to the available amperage on the +12. I bet you are actually overloading that part of the PSU, causing dirty power to the motherboard when the drives are spun up. Try temporarily running with a normal ATX PSU hung out the side of the case and see if the symptoms continue. Yeah - thats a good idea. If only I had a spare... I may have to see what I can find in work. I've finally got the errors from FCP fixed (after many panics). Now waiting for it to panic again and I can post the logs and go to bed...
February 23, 20179 yr Author And the last Kernel Panic of the night. Logs attached. Now going to run memtest. Thanks everyone for their helpful suggesitons. Hopefully tomorrow will be more fruitful. FCPsyslog_tail.txt tower-diagnostics-20170223-0023.zip
February 23, 20179 yr 11 minutes ago, emmcee said: Yeah - thats a good idea. If only I had a spare... I may have to see what I can find in work. I've finally got the errors from FCP fixed (after many panics). Now waiting for it to panic again and I can post the logs and go to bed... FYI, Troubleshooting mode from FCP is cancelled after every reboot so you got to ensure that its back up and running. (if that's the approach you were taking)
February 23, 20179 yr Author 1 minute ago, Squid said: FYI, Troubleshooting mode from FCP is cancelled after every reboot so you got to ensure that its back up and running. (if that's the approach you were taking) Doh. Not sure if I ran it again. it did update the tail file, so maybe I did? In any case, my bed is calling and tomorrow is another day.
February 23, 20179 yr Yes, you did remember to start it but it doesn't reveal anything new. The system booted, gave the same MCE at the same place. You started the array. It began its parity check due to the previous unclean shut down. The last thing recorded is the mover running. I'm not happy about the MCE and I'm dubious about the PicoPSU. I found the manual here. It says it's for Atoms and other low power CPUs. I don't think Broadwell i7s quite fall into that category: 65 W TDP, according to the ARK. Couple that with five mechanical disks and it's surely too much.
February 23, 20179 yr Author 8 hours ago, John_M said: I'm not happy about the MCE and I'm dubious about the PicoPSU. I found the manual here. It says it's for Atoms and other low power CPUs. I don't think Broadwell i7s quite fall into that category: 65 W TDP, according to the ARK. Couple that with five mechanical disks and it's surely too much. This page recommends a 120W for a 65W TDP chip: http://www.mini-itx.com/store/psu and I have Turbo Boost turned off so it's unlikely to hit that at any stage. In any case, the PSU is 160W nominal/200W peak so it should be able to handle momentary spikes. I've looked at CPU usage during transcoding on Plex and it doesn't peg the CPU - it peaks for a few seconds then falls back as it caches the transcoded data. WD reds are rated at 4.4W during read/write so that's a max of 22W during parity check. The only other things drawing power is the SSD and the fans. Edited February 23, 20179 yr by emmcee
February 23, 20179 yr 9 minutes ago, emmcee said: WD reds are rated at 4.4W during read/write so that's a max of 22W during parity check Read / write are negligible compared to spin up power There you are looking at 1.79A = 21.48 Watts per drive x 5 drives = 107 Watts. As I said, FCP crashing the system when it runs a check suggests power issues because one of its default tests is to spin up all the drives to attempt a write to them...
February 23, 20179 yr Author OK, so I rolled back to 6.2.4 (not 6.2.1 as I thought...) No mce errors in the log on startup and no Kernel Panics so far with FCP Troubleshooting and a Parity Check running. Uptime is 46 minutes so far. The only other difference is the case is open, so it could possibly be heat related. Once I've had some coffee I'll put it back together and test again. FCPsyslog_tail.txt tower-diagnostics-20170223-1032.zip
February 23, 20179 yr Author I have reassembled the machine and it started up again with no mce errors....but.... I've just noticed that Docker is not running. I tried disabling/enabling in the settings, but no joy. The docker.log contains multiple entries like this: time="2017-02-23T11:38:26.636932870Z" level=warning msg="could not create bridge network for id 2c6d8cffa9ff6de49c2d5863d7df3a8c4b7c17441cae3d17a2fb0c181b13b088 bridge name docker0 while booting up from persistent state" time="2017-02-23T11:38:26.665783763Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" time="2017-02-23T11:38:26.759714339Z" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: Failed to setup IP tables, cannot acquire Interface address: Interface docker0 has no IPv4 addresses" Also, the GUI is slow to load pages - about 7 - 8 seconds. Any ideas? tower-diagnostics-20170223-1138.zip Edited February 23, 20179 yr by emmcee
February 23, 20179 yr Docker errors are network related, possibly caused by this Quote Feb 23 11:26:04 Tower kernel: r8169 0000:02:00.0 eth0: link down Feb 23 11:30:27 Tower kernel: r8169 0000:02:00.0 eth0: link up Feb 23 11:30:28 Tower kernel: r8169 0000:02:00.0 eth0: link down ... Feb 23 11:30:51 Tower kernel: r8169 0000:02:00.0 eth0: link up which happened just as docker was starting. Perhaps a faulty network cable?
February 23, 20179 yr Author I have dual ethernet. I have been swapping them as it seems eth0 isn't working. In any case, I swapped for a known good cable and rebooted the server. Docker still refuses to run.
Archived
This topic is now archived and is closed to further replies.