ati Posted March 7, 2023 Share Posted March 7, 2023 I am working my way towards upgrading from 6.8 to 6.10. As part of that I needed to update a few settings on my Seagate drives, which required a reboot. I made the changes and rebooted, but the server didn't come back right away. I later learned it was because I left a second USB drive plugged in and it messed up the boot order. I removed that USB, rebooted and the server came up as expected. I then verified the changes using the SeaChest tools and rebooted again. This is when it all went sideways. Everything comes up fine, but I have no network access. The /boot/config/network.cfg matches my previously backed up config (the CLI shows the correct address) I checked syslog and didn't see anything glaring. I pulled the link the syslog shows it going down - so the NIC isn't dead. I deleted /boot/config/network.cfg and rebooted, it did not re-create the file I rebooted again into GUI mode and entered in my static IP in the GUI and rebooted again for the settings to apply. No luck. This is the diagnostics for today from the USB from the last reboot. I am going crazy. This is so simple. Really makes my nervous that a simple reboot caused this much of an issue. What am I missing. I am beyond frustrated with something so simple. unraid-diagnostics-20230307-1115.zip unraid-diagnostics-20230307-1406.zip unraid-diagnostics-20230307-1451.zip Quote Link to comment
ati Posted March 7, 2023 Author Share Posted March 7, 2023 I am even more lost now. I shut down, to get the diagnostics off the USB and upon reboot it booted normally and I was able to access it via the network. I logged into the web GUI to startup the array. Hell I even got an email alert that my UPS was unplugged (to get the USB out). About 30 seconds into starting the array I lost network again. I connected back to the server to checkout the syslog and saw this: Mar 7 15:14:38 unRAID root: Starting diskload Mar 7 15:14:38 unRAID emhttpd: Mounting disks... Mar 7 15:14:38 unRAID emhttpd: shcmd (197): /sbin/btrfs device scan Mar 7 15:14:38 unRAID root: Scanning for Btrfs filesystems Mar 7 15:14:38 unRAID emhttpd: shcmd (198): mkdir -p /mnt/disk1 Mar 7 15:14:38 unRAID emhttpd: shcmd (199): mount -t xfs -o noatime,nodiratime /dev/md1 /mnt/disk1 Mar 7 15:14:38 unRAID kernel: SGI XFS with ACLs, security attributes, no debug enabled Mar 7 15:14:38 unRAID kernel: XFS (md1): Mounting V5 Filesystem Mar 7 15:14:38 unRAID kernel: XFS (md1): Ending clean mount Mar 7 15:14:38 unRAID emhttpd: shcmd (200): xfs_growfs /mnt/disk1 Mar 7 15:14:38 unRAID root: meta-data=/dev/md1 isize=512 agcount=8, agsize=244188659 blks Mar 7 15:14:38 unRAID root: = sectsz=512 attr=2, projid32bit=1 Mar 7 15:14:38 unRAID root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 7 15:14:38 unRAID root: = reflink=1 Mar 7 15:14:38 unRAID root: data = bsize=4096 blocks=1953506633, imaxpct=5 Mar 7 15:14:38 unRAID root: = sunit=0 swidth=0 blks Mar 7 15:14:38 unRAID root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 7 15:14:38 unRAID root: log =internal log bsize=4096 blocks=476930, version=2 Mar 7 15:14:38 unRAID root: = sectsz=512 sunit=0 blks, lazy-count=1 Mar 7 15:14:38 unRAID root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 7 15:14:38 unRAID emhttpd: shcmd (201): mkdir -p /mnt/disk2 Mar 7 15:14:38 unRAID emhttpd: shcmd (202): mount -t xfs -o noatime,nodiratime /dev/md2 /mnt/disk2 Mar 7 15:14:38 unRAID kernel: XFS (md2): Mounting V5 Filesystem Mar 7 15:14:38 unRAID kernel: XFS (md2): Ending clean mount Mar 7 15:14:38 unRAID emhttpd: shcmd (203): xfs_growfs /mnt/disk2 Mar 7 15:14:38 unRAID root: meta-data=/dev/md2 isize=512 agcount=8, agsize=244188659 blks Mar 7 15:14:38 unRAID root: = sectsz=512 attr=2, projid32bit=1 Mar 7 15:14:38 unRAID root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 7 15:14:38 unRAID root: = reflink=1 Mar 7 15:14:38 unRAID root: data = bsize=4096 blocks=1953506633, imaxpct=5 Mar 7 15:14:38 unRAID root: = sunit=0 swidth=0 blks Mar 7 15:14:38 unRAID root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 7 15:14:38 unRAID root: log =internal log bsize=4096 blocks=476930, version=2 Mar 7 15:14:38 unRAID root: = sectsz=512 sunit=0 blks, lazy-count=1 Mar 7 15:14:38 unRAID root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 7 15:14:38 unRAID emhttpd: shcmd (204): mkdir -p /mnt/disk3 Mar 7 15:14:38 unRAID emhttpd: shcmd (205): mount -t xfs -o noatime,nodiratime /dev/md3 /mnt/disk3 Mar 7 15:14:38 unRAID kernel: XFS (md3): Mounting V5 Filesystem Mar 7 15:14:39 unRAID kernel: XFS (md3): Ending clean mount Mar 7 15:14:39 unRAID emhttpd: shcmd (206): xfs_growfs /mnt/disk3 Mar 7 15:14:39 unRAID root: meta-data=/dev/md3 isize=512 agcount=6, agsize=268435455 blks Mar 7 15:14:39 unRAID root: = sectsz=512 attr=2, projid32bit=1 Mar 7 15:14:39 unRAID root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 7 15:14:39 unRAID root: = reflink=1 Mar 7 15:14:39 unRAID root: data = bsize=4096 blocks=1465122442, imaxpct=5 Mar 7 15:14:39 unRAID root: = sunit=0 swidth=0 blks Mar 7 15:14:39 unRAID root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 7 15:14:39 unRAID root: log =internal log bsize=4096 blocks=521728, version=2 Mar 7 15:14:39 unRAID root: = sectsz=512 sunit=0 blks, lazy-count=1 Mar 7 15:14:39 unRAID root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 7 15:14:39 unRAID emhttpd: shcmd (207): mkdir -p /mnt/cache Mar 7 15:14:39 unRAID emhttpd: cache uuid: 7cb7e73a-56e0-4226-b453-ed9ec86cc38f Mar 7 15:14:39 unRAID emhttpd: cache TotDevices: 2 Mar 7 15:14:39 unRAID emhttpd: cache NumDevices: 2 Mar 7 15:14:39 unRAID emhttpd: cache NumFound: 2 Mar 7 15:14:39 unRAID emhttpd: cache NumMissing: 0 Mar 7 15:14:39 unRAID emhttpd: cache NumMisplaced: 0 Mar 7 15:14:39 unRAID emhttpd: cache NumExtra: 0 Mar 7 15:14:39 unRAID emhttpd: cache LuksState: 0 Mar 7 15:14:39 unRAID emhttpd: shcmd (208): mount -t btrfs -o noatime,nodiratime -U 7cb7e73a-56e0-4226-b453-ed9ec86cc38f /mnt/cache Mar 7 15:14:39 unRAID kernel: BTRFS info (device sdh1): using free space tree Mar 7 15:14:39 unRAID kernel: BTRFS info (device sdh1): has skinny extents Mar 7 15:14:56 unRAID emhttpd: shcmd (209): /sbin/btrfs filesystem resize 1:max /mnt/cache Mar 7 15:14:56 unRAID root: Resize '/mnt/cache' of '1:max' Mar 7 15:14:56 unRAID kernel: BTRFS info (device sdh1): resizing devid 1 Mar 7 15:14:56 unRAID kernel: BTRFS info (device sdh1): new size for /dev/sdh1 is 3000592928768 Mar 7 15:14:56 unRAID emhttpd: shcmd (210): /sbin/btrfs filesystem resize 2:max /mnt/cache Mar 7 15:14:56 unRAID root: Resize '/mnt/cache' of '2:max' Mar 7 15:14:56 unRAID kernel: BTRFS info (device sdh1): resizing devid 2 Mar 7 15:14:56 unRAID kernel: BTRFS info (device sdh1): new size for /dev/sde1 is 3000592928768 Mar 7 15:14:56 unRAID emhttpd: shcmd (211): sync Mar 7 15:14:56 unRAID emhttpd: shcmd (212): mkdir /mnt/user0 Mar 7 15:14:56 unRAID emhttpd: shcmd (213): /usr/local/sbin/shfs /mnt/user0 -disks 14 -o noatime,allow_other |& logger Mar 7 15:14:56 unRAID shfs: use_ino: 1 Mar 7 15:14:56 unRAID shfs: direct_io: 0 Mar 7 15:14:56 unRAID emhttpd: shcmd (214): mkdir /mnt/user Mar 7 15:14:56 unRAID emhttpd: shcmd (215): /usr/local/sbin/shfs /mnt/user -disks 15 2048000000 -o noatime,allow_other -o remember=0 |& logger Mar 7 15:14:56 unRAID shfs: use_ino: 1 Mar 7 15:14:56 unRAID shfs: direct_io: 0 Mar 7 15:14:57 unRAID emhttpd: shcmd (217): /usr/local/sbin/update_cron Mar 7 15:14:57 unRAID root: Delaying execution of fix common problems scan for 10 minutes Mar 7 15:14:57 unRAID unassigned.devices: Mounting 'Auto Mount' Devices... Mar 7 15:14:57 unRAID emhttpd: Starting services... Mar 7 15:14:57 unRAID emhttpd: shcmd (220): /etc/rc.d/rc.samba restart Mar 7 15:14:59 unRAID root: Starting Samba: /usr/sbin/smbd -D Mar 7 15:15:00 unRAID root: /usr/sbin/nmbd -D Mar 7 15:15:00 unRAID root: /usr/sbin/wsdd Mar 7 15:15:00 unRAID root: /usr/sbin/winbindd -D Mar 7 15:15:00 unRAID emhttpd: shcmd (234): /usr/local/sbin/mount_image '/mnt/cache/system/docker/docker.img' /var/lib/docker 40 Mar 7 15:15:00 unRAID kernel: BTRFS: device fsid 1e042287-9021-48d2-8ad1-772169654339 devid 1 transid 1747553 /dev/loop2 Mar 7 15:15:00 unRAID kernel: BTRFS info (device loop2): using free space tree Mar 7 15:15:00 unRAID kernel: BTRFS info (device loop2): has skinny extents Mar 7 15:15:00 unRAID root: Resize '/var/lib/docker' of 'max' Mar 7 15:15:00 unRAID kernel: BTRFS info (device loop2): new size for /dev/loop2 is 42949672960 Mar 7 15:15:00 unRAID emhttpd: shcmd (236): /etc/rc.d/rc.docker start Mar 7 15:15:00 unRAID root: starting dockerd ... Mar 7 15:15:04 unRAID avahi-daemon[10987]: Joining mDNS multicast group on interface br-4ead1d45ff54.IPv4 with address 172.18.0.1. Mar 7 15:15:04 unRAID avahi-daemon[10987]: New relevant interface br-4ead1d45ff54.IPv4 for mDNS. Mar 7 15:15:04 unRAID avahi-daemon[10987]: Registering new address record for 172.18.0.1 on br-4ead1d45ff54.IPv4. Mar 7 15:15:04 unRAID kernel: IPv6: ADDRCONF(NETDEV_UP): br-4ead1d45ff54: link is not ready Mar 7 15:15:04 unRAID avahi-daemon[10987]: Joining mDNS multicast group on interface br-66b05052a9b2.IPv4 with address 192.168.240.1. Mar 7 15:15:04 unRAID avahi-daemon[10987]: New relevant interface br-66b05052a9b2.IPv4 for mDNS. Mar 7 15:15:04 unRAID avahi-daemon[10987]: Registering new address record for 192.168.240.1 on br-66b05052a9b2.IPv4. Mar 7 15:15:04 unRAID kernel: IPv6: ADDRCONF(NETDEV_UP): br-66b05052a9b2: link is not ready Mar 7 15:15:04 unRAID avahi-daemon[10987]: Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1. Mar 7 15:15:04 unRAID avahi-daemon[10987]: New relevant interface docker0.IPv4 for mDNS. Mar 7 15:15:04 unRAID avahi-daemon[10987]: Registering new address record for 172.17.0.1 on docker0.IPv4. Mar 7 15:15:04 unRAID kernel: IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready Mar 7 15:15:09 unRAID rc.docker: b1f70fe360063b35c8f89ccb8a5befd5f30feb5c090801074f6f4ddcbd41b28f Mar 7 15:15:11 unRAID emhttpd: shcmd (250): /usr/local/sbin/mount_image '/mnt/cache/system/libvirt/libvirt.img' /etc/libvirt 1 Mar 7 15:15:11 unRAID kernel: BTRFS: device fsid bc304288-554a-4fd8-a63c-b4a0a37f5e68 devid 1 transid 262 /dev/loop3 Mar 7 15:15:11 unRAID kernel: BTRFS info (device loop3): using free space tree Mar 7 15:15:11 unRAID kernel: BTRFS info (device loop3): has skinny extents Mar 7 15:15:11 unRAID root: Resize '/etc/libvirt' of 'max' Mar 7 15:15:11 unRAID kernel: BTRFS info (device loop3): new size for /dev/loop3 is 1073741824 Mar 7 15:15:11 unRAID emhttpd: shcmd (252): /etc/rc.d/rc.libvirt start Mar 7 15:15:11 unRAID root: Starting virtlockd... Mar 7 15:15:11 unRAID root: Starting virtlogd... Mar 7 15:15:11 unRAID root: Starting libvirtd... Mar 7 15:15:11 unRAID kernel: tun: Universal TUN/TAP device driver, 1.6 Mar 7 15:15:11 unRAID kernel: virbr0: port 1(virbr0-nic) entered blocking state Mar 7 15:15:11 unRAID kernel: virbr0: port 1(virbr0-nic) entered disabled state Mar 7 15:15:11 unRAID kernel: device virbr0-nic entered promiscuous mode Mar 7 15:15:11 unRAID emhttpd: nothing to sync Mar 7 15:15:11 unRAID kernel: bond0: link status definitely down for interface eth0, disabling it Mar 7 15:15:11 unRAID kernel: bond0: making interface eth1 the new active one Mar 7 15:15:11 unRAID kernel: device eth0 left promiscuous mode Mar 7 15:15:11 unRAID kernel: device eth1 entered promiscuous mode Mar 7 15:15:11 unRAID avahi-daemon[10987]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. Mar 7 15:15:11 unRAID avahi-daemon[10987]: New relevant interface virbr0.IPv4 for mDNS. Mar 7 15:15:11 unRAID avahi-daemon[10987]: Registering new address record for 192.168.122.1 on virbr0.IPv4. Mar 7 15:15:11 unRAID kernel: virbr0: port 1(virbr0-nic) entered blocking state Mar 7 15:15:11 unRAID kernel: virbr0: port 1(virbr0-nic) entered listening state Mar 7 15:15:11 unRAID dnsmasq[14786]: started, version 2.80 cachesize 150 Mar 7 15:15:11 unRAID dnsmasq[14786]: compile time options: IPv6 GNU-getopt no-DBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify dumpfile Mar 7 15:15:11 unRAID dnsmasq-dhcp[14786]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Mar 7 15:15:11 unRAID dnsmasq-dhcp[14786]: DHCP, sockets bound exclusively to interface virbr0 Mar 7 15:15:11 unRAID dnsmasq[14786]: reading /etc/resolv.conf Mar 7 15:15:11 unRAID dnsmasq[14786]: using nameserver 8.8.8.8#53 Mar 7 15:15:11 unRAID dnsmasq[14786]: using nameserver 8.8.4.4#53 Mar 7 15:15:11 unRAID dnsmasq[14786]: read /etc/hosts - 2 addresses Mar 7 15:15:11 unRAID dnsmasq[14786]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Mar 7 15:15:11 unRAID dnsmasq-dhcp[14786]: read /var/lib/libvirt/dnsmasq/default.hostsfile Mar 7 15:15:11 unRAID kernel: virbr0: port 1(virbr0-nic) entered disabled state Mar 7 15:15:11 unRAID kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details. Mar 7 15:15:12 unRAID tips.and.tweaks: Tweaks Applied Mar 7 15:15:12 unRAID unassigned.devices: Mounting 'Auto Mount' Remote Shares... Mar 7 15:15:12 unRAID sudo: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237 Mar 7 15:15:13 unRAID kernel: br0: port 2(vnet0) entered blocking state Mar 7 15:15:13 unRAID kernel: br0: port 2(vnet0) entered disabled state Mar 7 15:15:13 unRAID kernel: device vnet0 entered promiscuous mode Mar 7 15:15:13 unRAID kernel: br0: port 2(vnet0) entered blocking state Mar 7 15:15:13 unRAID kernel: br0: port 2(vnet0) entered forwarding state Mar 7 15:15:14 unRAID kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Mar 7 15:15:14 unRAID kernel: bond0: link status definitely up for interface eth0, 1000 Mbps full duplex Mar 7 15:15:15 unRAID avahi-daemon[10987]: Joining mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe00:f65e. Mar 7 15:15:15 unRAID avahi-daemon[10987]: New relevant interface vnet0.IPv6 for mDNS. Mar 7 15:15:15 unRAID avahi-daemon[10987]: Registering new address record for fe80::fc54:ff:fe00:f65e on vnet0.*. Mar 7 15:16:20 unRAID ntpd[1875]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Mar 7 15:17:13 unRAID unassigned.devices: Cannot 'Auto Mount' Remote Shares. Network not available! Mar 7 15:18:16 unRAID login[10849]: ROOT LOGIN on '/dev/tty1' What the heck is happening? Quote Link to comment
JorgeB Posted March 8, 2023 Share Posted March 8, 2023 You are using a valid config but to simply things disable the bond and set eth1 as eth0 (or swap the cables) Quote Link to comment
ati Posted March 8, 2023 Author Share Posted March 8, 2023 4 hours ago, JorgeB said: You are using a valid config but to simply things disable the bond and set eth1 as eth0 (or swap the cables) Tried that many times. Tried each port individually even. When I deleted the /config/network.cfg I set it back up as a single interface. After lots of reboots (to remove the USB and get diagnostic files off it) it just worked after no changes. Any idea where there are lots of references in the syslog to virbr0 using an IP address that isn't anywhere on my network or unRAID configuration? Regardless, I am afraid to reboot for fear of having this ordeal happen again. I spent 3 hours rebooting my server in circles for it to miraculously work again using the same configuration I started with 3 hours prior. Quote Link to comment
JorgeB Posted March 8, 2023 Share Posted March 8, 2023 8 minutes ago, ati said: Any idea where there are lots of references in the syslog to virbr0 using an IP address that isn't anywhere on my network or unRAID configuration? Those are normal. Quote Link to comment
MalronWall Posted March 5 Share Posted March 5 Hello everyone! I have the same problem... ifconfig gives me bond0 br0... but they have no ip. Like ati, after several reboots, it works again! And after the next one, it doesn't work anymore... It's very frustrating. Here's my diagnosis, can you please help me? I'm not sure what else to do... This has been going on for several months now, and the only temporary solution I've found is........ Avoid stopping it... 😅 Thanks in advance unraid-diagnostics-20240305-2231.zip Quote Link to comment
JorgeB Posted March 6 Share Posted March 6 Link detected: no No link being detect on the NIC, check cables/switch or try a different NIC. Quote Link to comment
MalronWall Posted March 6 Share Posted March 6 (edited) Thanks for your reply! Indeed, but the cables are physically fine (already disconnected and reconnected several times following this message). For me, it's more a question of a false negative and the problem is more soft than hard. Edited March 6 by MalronWall Add precision Quote Link to comment
JorgeB Posted March 6 Share Posted March 6 It will never work while Linux sees the link as down. Quote Link to comment
MalronWall Posted March 6 Share Posted March 6 I know, but what else can I do? 😕 Quote Link to comment
JorgeB Posted March 6 Share Posted March 6 15 minutes ago, JorgeB said: check cables/switch or try a different NIC. Quote Link to comment
MalronWall Posted March 6 Share Posted March 6 I did both and I still have the problem... 😔 The last time the server was restarted (a few hours ago) I had this problem. After several reboots I still had this problem... I went to bed and, as usual after several hours, tried to restart it again and 0 problems. I don't know where the "random" variant of this problem comes from... Quote Link to comment
JorgeB Posted March 6 Share Posted March 6 2 hours ago, MalronWall said: I did both and I still have the problem... Do you mean different cable/switch and also NIC? Quote Link to comment
uninformed Posted March 6 Share Posted March 6 (edited) I believe I am having the same problems. I updated, left everything running for over a week, no issues. Rebooted and the networking hasn't worked since. I tried several fixes mentioned, turning off and on bridging, bonding, cpu virtualization in the bios, different cables, its plugged directly into the modem and back into the router, got nada. I only have the onboard networking, it's a Realtek brand, I don't have a network card to test with, but I did boot windows and tried it, the ethernet is working just fine. Unfortunately I deleted my backup of the previous version, after running this one for a few days, so I cannot revert either. Any ideas? tower-diagnostics-20230814-2255.zip Edited March 6 by uninformed might as well add the diagnostics Quote Link to comment
JorgeB Posted March 6 Share Posted March 6 19 minutes ago, uninformed said: I believe I am having the same problems It's no the same since your NIC detects a link, rename/delete /boot/config/network.cfg to restore the network settings to default DHCP, then reboot and post new diags. 1 Quote Link to comment
uninformed Posted March 6 Share Posted March 6 will do, deleted network.cfg before, and tried again, but I will do this now, thank you Quote Link to comment
uninformed Posted March 6 Share Posted March 6 here's the updated diagnostics tower-diagnostics-20240306-0724.zip Quote Link to comment
trurl Posted March 6 Share Posted March 6 ethtool says no link detected https://docs.unraid.net/unraid-os/release-notes/6.12.8/#problems-due-to-realtek-network-cards 1 Quote Link to comment
MalronWall Posted March 6 Share Posted March 6 7 hours ago, JorgeB said: Do you mean different cable/switch and also NIC? Cable/switch yes, but what do you mean by "NIC"? Or rather, how do you want me to test this (not forgetting that when this problem occurs I only have the CLI) and what result is expected? 🤔 Sorry... I'm not performing very well yet Quote Link to comment
uninformed Posted March 6 Share Posted March 6 working, no changes made.... most annoying problem is an intermittent problem.... anyways I did install the the Realtek plugin for 8111, just incase, thank you very much tourl and Jorgeb Quote Link to comment
JorgeB Posted March 7 Share Posted March 7 13 hours ago, MalronWall said: but what do you mean by "NIC"? The Ethernet controller. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.