February 25, 201511 yr unRAID OS Version: Beta 14 Description: After the server has been up for more than 24 hours the network stops working. Cannot even ping anything from the console of the unraid box. The console responds fine and the syslog shows no errors. This all worked wonderfully on Beta 12. How to reproduce: Enable bridge br0 and wait 24 hours Expected results: ARP would show IP to MAC for all recent traffic Actual results: ARP will show incomplete for IPs attempting to communicate with the unraid box Other information: Motherboard: ASROCK 970 Extreme 4 LAN: Realtek RTL8111E It appears to have something to do with the bridge setup br0. I stop building ARP entries on the bridge after a certain period of time. If I manually ad an arp entry it can reach that device.
February 25, 201511 yr Author I've now disabled bridge br0 and shutdown my VMs in an attempt to see if the problem persists. I will know tomorrow
February 25, 201511 yr I'd recommend updating to 14a, if you submit a bug report on an old version, the first thing anybody is going to say is "does it do it on the current beta?"
February 25, 201511 yr So without a bridge setup, this works fine? Any chance you have an alternate network adapter you can use to help us narrow down if this is specific to the RTL8111E?
February 26, 201511 yr Author Ok, So disabling the Bridge br0 fixed the issue. This worked fine in beta 12. Updating to the latest beta 14b and re-enabling the bridge to see if the problem is still present. I will see if I can get out to purchase another network card today to test with. Is there a specific chipset you want me to try to obtain for the test?
February 26, 201511 yr Ok, So disabling the Bridge br0 fixed the issue. This worked fine in beta 12. Updating to the latest beta 14b and re-enabling the bridge to see if the problem is still present. I will see if I can get out to purchase another network card today to test with. Is there a specific chipset you want me to try to obtain for the test? I hate to push you to buying new hardware just to test. Was hoping you had something maybe lying around, but if you're willing to buy a new one, Intel NICs tend to be very solid and reliable for many in this forum. I just want to confirm if it's a device specific issue.
February 27, 201511 yr Confirmed issue still occurring on 14b with bridge br0 enabled Yeah I wouldn't have expected a change there. Same kernel version so therefore same driver. Assuming you are still using the realtek, yes?
February 27, 201511 yr Author Confirmed issue still occurring on 14b with bridge br0 enabled Yeah I wouldn't have expected a change there. Same kernel version so therefore same driver. Assuming you are still using the realtek, yes? Correct, haven't had a chance to head out and get another card. It's snowing and the roads are bad. I'll head out once the roads are clear and get another card. I'll look for an Intel, Frys is bound to have something that'll work.
February 27, 201511 yr Confirmed issue still occurring on 14b with bridge br0 enabled Yeah I wouldn't have expected a change there. Same kernel version so therefore same driver. Assuming you are still using the realtek, yes? Correct, haven't had a chance to head out and get another card. It's snowing and the roads are bad. I'll head out once the roads are clear and get another card. I'll look for an Intel, Frys is bound to have something that'll work. I hear yah. I live in Chicago and we've had our fair share of snow this year. Keep me posted!
February 27, 201511 yr Only after updating from beta12 to beta14 (a,b) the last few days has my unRAID stopped responding in about a 24 hour period, twice. I remember having this issue back in 5.x but was remedied by using an Intel NIC. When I migrated to 6.x I removed the Intel NIC and used the onboard Realtek LAN and it has worked just fine until 6.0-Beta14. This morning I installed a new Intel NIC on my 6.0-Beta14b in hopes it won't freeze again. Thanks for the reminder JonP.
February 28, 201511 yr I can report the same issue. And I have no open PCIe slots, so this is a major issue for me. I will try an external NIC and see if that fixes it. If not I will have to roleback.
March 1, 201511 yr It has been two days now since I have switched from onboard LAN (Realtek) to a new Intel NIC and can report that my unRAID 6.0-Beta14b has not lost LAN connectivity. I will try to "stress test" the Intel NIC by increasing LAN traffic (inbound and outbound) from local and remote clients and then see if it holds up after a couple of more days, but so far so good.
March 3, 201511 yr Author Been running a couple days now with no issues on an Intel NIC. Seems to be an issue with Realtek NICs in the latest builds.
March 3, 201511 yr Author Also my workaround of disabling bridge Br0 seems to hold up as well on the Realtek NIC. If I enable Realtek NIC then ARP stops working after a period of time with bridge Br0 on.
March 17, 201511 yr Author Been running for a while on the Intel NIC. The original issue has gone away but a different issue has surfaced. After a while running (4-5 days), the web interface becomes un-usable (Page cannot be displayed). All other items continue to work. About a day later the VMs using the bridge can no longer see the network. Within a week the SMB shares are no longer visible on the network. A reboot fixes the issue. The other option is for me to disable the bridge all together then no issues occur. It seems this all stems from the use of bridging. I also notice that ARP entries begin to stop working in the ARP table, they show "INCOMPLETE", even though they are devices on the same subnet and switch. shutting down the interface br0 and re-upping it does not fix the issue. removing eth0 from the bridge and re-adding it does not fix the issue. Spanning-Tree is disabled on the bridge and "brctl showstp br0" shows the bridge is forwarding on all ports. Deleting the bridge, and re-adding it then readding eth0 to the bridge and re-issuing the ip address does not fix the issue. A sniffer on the network shows the ARP requests are not leaving the NIC when issuing a ping to a device from the unraid box. The sniffer also shows the unraid box is not responding to ARP requests. Adding static ARP entries to both machines trying to talk does not repair the issue, the sniffer show the unraid box is not responding to any network traffic. Removing the bridge completely and adding the IP address to eth0 fixes the issue immediately. ifconfig br0 down brctl delbr br0 ifconfig eth0 down ifconfig eth0 192.168.1.5 netmask 255.255.255.0 up This somehow stems from the bridging feature and is independent of the NIC being used. For now I will keep my VMs shutdown and cease to use bridging until a new version comes out that attempts to fix these issues. If you need me to perform further troubleshooting, let me know I will gladly assist. I have attached the syslog from before I did any troubleshooting but when the problem was manifesting itself. syslog.txt
March 17, 201511 yr Been running for a while on the Intel NIC. The original issue has gone away but a different issue has surfaced. After a while running (4-5 days), the web interface becomes un-usable (Page cannot be displayed). All other items continue to work. About a day later the VMs using the bridge can no longer see the network. Within a week the SMB shares are no longer visible on the network. A reboot fixes the issue. The other option is for me to disable the bridge all together then no issues occur. It seems this all stems from the use of bridging. I also notice that ARP entries begin to stop working in the ARP table, they show "INCOMPLETE", even though they are devices on the same subnet and switch. shutting down the interface br0 and re-upping it does not fix the issue. removing eth0 from the bridge and re-adding it does not fix the issue. Spanning-Tree is disabled on the bridge and "brctl showstp br0" shows the bridge is forwarding on all ports. Deleting the bridge, and re-adding it then readding eth0 to the bridge and re-issuing the ip address does not fix the issue. A sniffer on the network shows the ARP requests are not leaving the NIC when issuing a ping to a device from the unraid box. The sniffer also shows the unraid box is not responding to ARP requests. Adding static ARP entries to both machines trying to talk does not repair the issue, the sniffer show the unraid box is not responding to any network traffic. Removing the bridge completely and adding the IP address to eth0 fixes the issue immediately. ifconfig br0 down brctl delbr br0 ifconfig eth0 down ifconfig eth0 192.168.1.5 netmask 255.255.255.0 up This somehow stems from the bridging feature and is independent of the NIC being used. For now I will keep my VMs shutdown and cease to use bridging until a new version comes out that attempts to fix these issues. If you need me to perform further troubleshooting, let me know I will gladly assist. I have attached the syslog from before I did any troubleshooting but when the problem was manifesting itself. So are you talking about if you use the bridge in unRAID's network settings, or a bridge you self-created using the brctl command?
March 17, 201511 yr Author So are you talking about if you use the bridge in unRAID's network settings, or a bridge you self-created using the brctl command? Bridge in unRAID's network settings. I was just using the brctl to assist in troubleshooting.
March 19, 201511 yr Author I'm wondering, since I have 2 NICs now. Is it possible to have the main unRAID IP bound to eth0, while having the bridge setup on eth1. This way if the bridging crashes again it won't take down my networking to unRAID.
March 19, 201511 yr Author So I've got bridging working on 1 NIC while regular networking is working on the other NIC. I did this by altering 2 files. /boot/config/network.cfg: USE_DHCP[0]="no" IPADDR[0]="192.168.1.5" NETMASK[0]="255.255.255.0" GATEWAY[0]="192.168.1.1" DHCP_KEEPRESOLV[0]="no" DNS_SERVER1[0]="192.168.1.1" DNS_SERVER2[0]="" DNS_SERVER3[0]="" IFNAME[0]="eth0" BONDING="no" BONDING_MODE="1" BRIDGING="no" BRNAME="br0" BRSTP="no" Added to /boot/config/go: brctl addbr br0 brctl addif br0 eth1 ifconfig eth1 up ifconfig br0 up My VMs seem to be working on br0 and my unRAID is up and bound to eth0. Now we wait and see if the VMs on br0 become unresponsive after 24 hours.
March 20, 201511 yr Author So just a quick update here. After 24 hours, the unRAID on eth0 is having no issues at all. The VMs sitting on br0 which is using eth1 have stopped having network connectivity. So whatever this issue is, is specific to the bridging portion for sure. Not sure where else to go here. I guess I could NAT my VMs instead of bridging them but that's not ideal for me. Suggestions are welcome here. Is it safe to roll back to a previous version of unRAID or will that corrupt anything?
March 20, 201511 yr Author rolling back to beta12 to see if it fixes the issues. This is the last stable version I had.
March 21, 201511 yr My unraid installation also stops every night (14b), it would appear the mover script is causing an issue. I'm not really a linux guy, so the log files are a bit of a mystery. I've attached part of the error log, basically it's filled with the following lines for over an hour before the error log appears to stop collecting information Mar 21 01:04:02 Tower kernel: BTRFS info (device sdl1): no csum found for inode 7906352 start 5101334528 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] Mar 21 01:04:02 Tower kernel: Result: hostbyte=0x04 driverbyte=0x00 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] CDB: Mar 21 01:04:02 Tower kernel: cdb[0]=0x28: 28 00 80 e6 d0 df 00 00 20 00 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] Mar 21 01:04:02 Tower kernel: Result: hostbyte=0x04 driverbyte=0x00 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] CDB: Mar 21 01:04:02 Tower kernel: cdb[0]=0x28: 28 00 80 f6 d0 df 00 00 20 00 Is this disk corruption of some sort, and can it be fixed? Cheers Glen. error.zip
March 21, 201511 yr My unraid installation also stops every night (14b), it would appear the mover script is causing an issue. I'm not really a linux guy, so the log files are a bit of a mystery. I've attached part of the error log, basically it's filled with the following lines for over an hour before the error log appears to stop collecting information Mar 21 01:04:02 Tower kernel: BTRFS info (device sdl1): no csum found for inode 7906352 start 5101334528 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] Mar 21 01:04:02 Tower kernel: Result: hostbyte=0x04 driverbyte=0x00 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] CDB: Mar 21 01:04:02 Tower kernel: cdb[0]=0x28: 28 00 80 e6 d0 df 00 00 20 00 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] Mar 21 01:04:02 Tower kernel: Result: hostbyte=0x04 driverbyte=0x00 Mar 21 01:04:02 Tower kernel: sd 1:0:4:0: [sdl] CDB: Mar 21 01:04:02 Tower kernel: cdb[0]=0x28: 28 00 80 f6 d0 df 00 00 20 00 Is this disk corruption of some sort, and can it be fixed? Cheers Glen. Your drive which is connected to ATA13.00 (your syslog didn't go back far enough to tell which drive) has some issues. Presumably its your cache drive since its formatted BTRFS. BTRFS *thinks* that there is corruption on it. However if you go back in the error log to where it begins you have errors coming from ATA13.00 which imply a bad / loose sata cable or power connector, and could result in the BTRFS errors that you're seeing I would post the smart reports for the drive in question, and also reseat the cables going to that drive and mobo / HBA (or reseat the drive if its in a hotswap bay) If after reseating the connectors the BTRFS errors keep popping up, then I would run scrub on it.
Archived
This topic is now archived and is closed to further replies.