Unraid dropping from network

[JukeBug]

I have been running Unraid for many years with no problems. Lately, the server drops off my network after 5-10 minutes of powering up. I recently upgraded my parity drive to a larger disc, as well as upgrading to Unraid 6.8.3. I have swapped out network cables, but the problem persists. While I am using a static IP address for the Unraid server, the problem began before I started doing so. I downloaded the Fix Common Problems plugin, which gives a couple warnings, but no errors. Hopefully the attached syslog captured the latest time the server dropped from the network. I am unable to connect to the GUI whenever this happens, so I have to force a shutdown. Any help would be greatly appreciated.

syslog.zip

Edited August 16, 2020 by JukeBug

[Squid]

You should post the diagnostics (Tools - Diagnostics), as it gives far more information.  But, unless you're running with very little RAM, this

Aug 16 10:06:44 Unraid root: Fix Common Problems: Warning: Rootfs file is getting full (currently 77 % used)

is something that is effectively an error

[JukeBug]

My apologies on not attaching the diagnostics the first time. I've attached them and will start looking at the forums for solutions to the Rootfs problem. Thank you in advance!

unraid-diagnostics-20200816-1357.zip

[ChatNoir]

If I am not mistaking, unraid runs from memory. Rootfs would be there.

 

I see you are running with one 2GB DIMM, I would guess that 6.8.3 requires more ressource than your previous version.

[trurl]

1 hour ago, Squid said:

unless you're running with very little RAM

And you are running with little RAM. But looks like you only have basic NAS functionality configured so that might be OK.

 

Can you add more RAM?

[JukeBug]

You are correct that I am only using the most basic functionality on this server. I tried rolling back my Unraid to the version I used previously, but the server still dropped from the network after a short while. The only other real change I have made is upgrading my parity to a bigger drive and adding the old parity drive as additional storage. I am clueless on this, but could the addition of a new disk be chewing up more RAM somehow?

 

I have more RAM being delivered in a few days and will report back then. Thanks for the help everyone.

[JukeBug]

I dropped in 4GB of RAM today and the server stayed online for an hour or so. Just as I was getting my hopes up, it dropped from the network again. I managed to clear up the other warnings regarding the cache drive (I had to toggle cache off/on), but the main problem remains. If anyone would be willing to look at the latest diagnostics, I would appreciate it. I also attached the syslog to show the last time it dropped from the network. I am guessing this is the pertinent portion:

 

Aug 19 20:25:51 Unraid rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="8177" x-info="https://www.rsyslog.com"] start
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Aug 19 20:28:16 Unraid kernel: RTL8211B Gigabit Ethernet r8169-300:00: Master/Slave resolution failed, maybe conflicting manual settings?
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: Link is Down
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: rtl_phyar_cond == 1 (loop: 20, delay: 25).
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: rtl_phyar_cond == 1 (loop: 20, delay: 25).
Aug 19 20:28:16 Unraid dhcpcd[1439]: eth0: carrier lost
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Aug 19 20:28:16 Unraid kernel: r8169 0000:03:00.0 eth0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Aug 19 20:28:16 Unraid dhcpcd[1439]: eth0: deleting route to 192.168.1.0/24
Aug 19 20:28:16 Unraid dhcpcd[1439]: eth0: deleting default route via 192.168.1.1
Aug 19 20:28:17 Unraid ntpd[1507]: Deleting interface #1 eth0, 192.168.1.17#123, interface stats: received=13, sent=13, dropped=0, active_time=514 secs
Aug 19 20:28:17 Unraid ntpd[1507]: 82.209.245.153 local addr 192.168.1.17 -> <null>

unraid-diagnostics-20200819-2024.zip syslog.zip

[JorgeB]

Try another NIC if available, ideally from Intel.

[JukeBug]

I don't have an NIC sitting around, but am willing to buy one if that is likely the cause. At the same time, I don't want to keep buying new parts just to see if it fixes the problem. How likely do you think it is to be the NIC? Prior to upgrading my OS and parity drive, it had worked for many years with no problem.

[JorgeB]

7 minutes ago, JukeBug said:

How likely do you think it is to be the NIC?

Difficult to say, only that Realtek NIC drivers have been hit-and-miss for years with Linux, that's why we recommend using an Intel NIC whenever possible, much less likely to run into trouble.

[JukeBug]

I have seen this NIC suggested in another thread.

https://www.amazon.com/Intel-Gigabit-Network-Adapter-EXPI9301CTBLK/dp/B001CY0P7G?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o00_s00

 

I am already using my sole PCE 2.0x1 slot, but can I throw the above card into a PCE 2.0x16 slot? I will probably mess around with cables and resetting my network one more time before I make another purchase. TIA

[JorgeB]

1 hour ago, JukeBug said:

but can I throw the above card into a PCE 2.0x16 slot?

Yes, and if possible buy the NIC from someplace that allows returns in case it doesn't help, though like mentioned Intel NIC is always preferred.

[JukeBug]

I am not sure if this helps as an additional bit of evidence, but I let parity check run for the past 25 hours or so. The server stayed online the whole time. The minute it finished, the server dropped from the network. I wasn't mirroring my syslog to the flash drive, so nothing to show for it, but thought I would throw it out there in case it is helpful at all. I will go ahead and order the Intel NIC.

[JukeBug]

Just wanted to say thanks to all for the help. The new NIC seems to have done the trick so far.

[JukeBug]

Unfortunately I am running into problems again with my server dropping out. I have already replaced the RAM and the NIC. I am starting to wonder if the PSU is going bad on me. When the network drops, I am unable to press the button to turn the server off (even after holding down the power button for a while). I have to physically unplug it. I have attached the syslog from several dropouts today and the most recent diagnostics if anyone is willing to help.

syslog unraid-diagnostics-20200918-2251.zip

[JorgeB]

I don't see any issues in the syslog.

[JukeBug]

If there is no issue with the syslog, would it be feasible that the PSU has either hit its capacity or is failing? Maybe that could explain all the hardware problems I have seen lately. I have an old Corsair CX500, 8 drives at 5400rpm, no graphics card, an old Sempron CPU, an NIC in a PCI slot, and a PCI sata expansion card. I would think 500W could handle that without problem though.

 

As always, I appreciate your help.

[JorgeB]

It's possible, any hardware can fail at any time, if you have a spare one it's worth trying.

[JukeBug]

Unfortunately a new PSU did not fix this issue. After swapping out the PSU, NIC and RAM, I tried resetting my CPU with new thermal paste. I can try using a new USB drive for booting, but I don't have any reason to think the USB is bad.

 

To possibly help further diagnose this problem, UNRAID never boots up on the first try. It just hangs. If I force a shutdown via the PSU button and try a second time, it usually boots fine. Even if it boots, it always ends up freezing up at some point (sometimes minutes later, sometimes hours later), leaving me unconnected to the shared drives. When UNRAID hangs, I always have to force a shutdown via the PSU switch. The power button on the case does not do the trick. I have attached the syslog, but I am not seeing any pattern before the system freezes. The following commands are what I usually see before losing a connection.

 

Unraid root: Starting Nginx server daemon...
Unraid root:  issuing standby command
Unraid kernel: mdcmd (54): spindown 4

 

I came across a thread that sounded similar to my problems. It indicated problems using NICs in the graphics card slot on some motherboards, so I will mess with the BIOS settings as soon as I get an HDMI to VGA adapter to get onto the machine. If anyone has any other crazy ideas to try before swapping out the MB, I would love to hear them. Thanks as always.

 

syslog

[civic95man]

Maybe try booting your system with a live version of another flavor of linux to see if you have similar issues (lockup/boot problems).

 

Could also try clearing the bios to defaults and checking the battery as a last resort.

 

I didn't see anything in the manual about not using the pcie x16 slot for anything but graphics.

[Gelbert]

Looks like I am the 3rd person in the last day or so having this issue, but I am on 6.8.2, but getting the same issues, and I have a HP ML150g9 server.

 

Looks to affect more than just you.

[JukeBug]

Apologies for the basic question, but can I simply replace my Unraid USB with a LUBUNTU bootable USB (different USB drives) without creating any problems for my HDD data? I believe this should be OK, but want to double check before I do something dumb.

 

If this works and stays online for a day or two, should I assume the Unraid USB is bad?

[civic95man]

11 minutes ago, JukeBug said:

can I simply replace my Unraid USB with a LUBUNTU bootable USB (different USB drives)

Yes, remove the unraid USB and insert the bootable flavor of our choise.

 

12 minutes ago, JukeBug said:

without creating any problems for my HDD data

as long as you do not try to mount any of your disks outside of unraid, then you will be fine and parity will stay valid.  If you don't trust yourself then maybe disconnect all of your drives. I don't know if the live versions will try to auto mount disks now - its been a while since I've used one, but my assumption is that they won't mount until you try to access them (via GUI if you prefer) 

[JukeBug]

So unfortunately the computer still froze while using the Lubuntu OS from a different USB drive. I am guessing this means the motherboard is toast.

 

If that is correct, what is my best bet for finding a motherboard with an AM2 socket, at least 6 sata connections, and a couple of PCIE slots? I only use the server for storage, so I would like to avoid a complete upgrade of RAM and CPU along with the motherboard.

Edited September 30, 2020 by JukeBug

[civic95man]

1 minute ago, JukeBug said:

So unfortunately the computer still froze while using the Lubuntu OS from a different USB drive. I am guessing this means the motherboard is toast.

Yes, that would mean that its at the hardware level

2 minutes ago, JukeBug said:

what is my best bet for finding a motherboard with an AM2 socket, 6 sata connections, and a couple of PCIE slots

if you are comfortable with used, then i would check ebay.  I acquired most of my build through parts from ebay.  Just test everything really well before you are comfortable with it and any return policy expires.

 

otherwise you would need to look into NOS, but at that point it may be cheaper to use lower end modern equipment and buy everything new.