No more connection after update to 6.7.0. Server crash

[hi2hello]

Today I was updating to the latest version of unRaid. After the process was finished I rebooted my machine but unfortunately, it does not come up. Connecting a display to the server showed, that it no longer can get a connection to the network. There is an error on booting that says:

modprobe: FATAL: Module bonding not found in directory /lib/modules/4.19.41-Unraid
Cannot find device "bond0"

Also trying to boot in safe mode leads me to a browser, that can not get any connection neither to localhost, nor to any other ip. The server is not present in the network either. I still can type into the servers internal terminal. 

 

Seems there was something wrong with the update. Can i trigger this once more from the terminal? How do I proceed?

Would it be possible to roll back to 6.6.7? If so, how?

 

Thanks for any help as my server is crashed right now. 

PS: Even don’t know how to post a "diagnostics". I ran the command and it was saved to the server, but without any connection …

Edited May 12, 2019 by hi2hello

[itimpi]

Try running the ‘df’ command to check that the USB stick is being mounted as /boot.   Another issue that seems to be causing an issue is the contents of the config/network.cfg file on the USB stick.  Try renaming and then rebooting to get a new default one created to see if that helps.

[hi2hello]

There is indeed nothing mounted as /boot. Which device should be mounted as boot?
And how can I get this done?
 

Here is what df spit out

rootfs  … mounted on /
tmpfs … mounted on /run
devtmpfs … mounted on /dev
tmpfs … mounted on /dev/shm
group_root … mounted on /sys/fs/cgroup
tmpfs … mounted on /var/log

 

Edited May 12, 2019 by hi2hello

[itimpi]

If there is nothing mounted as /boot (which is where the USB stick gets mounted) then there is some sort of problem with it that needs resolving.    The first thing is to plug it into a Pc or Mac to check it.    Do you have a backup of its contents?

[hi2hello]

Just a really old backup (from May 2017). Another newer one is on the server. 
Got lazy as things were running totally smooth till now. Sorry for that.

 

Plugin the USB Drive into my mac seems to work fine. 
What am I looking for?

Edited May 12, 2019 by hi2hello

[itimpi]

Just now, hi2hello said:

Just a really old backup (from May 2017). Another newer one is on the server. 
Got lazy as things were running totally smooth till now. Sorry for that. I ll plug out the USB-Drive into my Mac. Than what?

There should be an option to check the filesystem on the usb stick, and make sure the Mac can read its contents.  Not being a Mac user I am not sure if the details for doing that.

[hi2hello]

No problems with reading content from scratch. Plugging in the stick mounts and reads fine. 

There is a folder called "previous" … Could it be an option to overwrite the actual content of the stick with the stuff from this folder?

 

How would i check filesystem of the stick on a PC? 

Edited May 12, 2019 by hi2hello

[hi2hello]

Ok. I reverted from 6.7.0 to 6.6.7 by moving the content from the USB-Stick’s "Previous"-Folder to the root and overwriting existing files. Worked like a charm. Server back up.

 

I guess I will wait for 6.7.1 though

 

PS: Anything I can help with to get this fixed? Seems that I am not the only one having problems updating. Would you like my diagnostics file?

Edited May 12, 2019 by hi2hello

[hi2hello]

I just found some time to have another look on that matter. 

The error can be reproduced (multiple times): As soon as I update to 6.7.0 and reboot, I m running into the same "bond0" error as described above. Rolling back to 6.6.7 (from the "previous" folder on the USB stick) and everything works as expected. 
 

I would really like to gain some further info but "running" 6.7.0 gives me errors on the "diagnostics" command, I also event don’t know where those infos are stored. Can’t find them when back on 6.6.7. 

So right now, I dont have the slightest clue on how to dive deeper to find or to help with error research. 

Here’s my server specs:

unRAID server Basic, version 6.6.7
Motherboard:Gigabyte Technology Co., Ltd. - H270N-WIFI-CF
Processor:Intel® Pentium® CPU G4560 @ 3.50GHz
HVM:Enabled
IOMMU:Enabled
Cache:L1 Cache = 128 kB (max. capacity 128 kB)
L2 Cache = 512 kB (max. capacity 512 kB)
L3 Cache = 3072 kB (max. capacity 3072 kB)
Memory:8 GB (max. installable capacity 64 GB)
ChannelA-DIMM0 = 4096 MB, 2400 MT/s
ChannelB-DIMM0 = 4096 MB, 2400 MT/s
Network:eth0: 1000Mb/s, full duplex, mtu 1500
eth1: not connected
Kernel:Linux 4.18.20-unRAID x86_64
OpenSSL:1.1.1a
P + Q algorithm:12023 MB/s + 18421 MB/s

 

[bonienl]

The error "modprobe: FATAL: Module bonding not found" indicates Unraid is unable to mount the specific module from the flash device.

Either there is file system corruption on the flash device, which needs to be repaired (easiest is to use a Windows PC) or the file itself is broken.

You can download the ZIP file and extract its contents to the flash device to see if it solves your problem.

 

[itimpi]

30 minutes ago, hi2hello said:

So right now, I dont have the slightest clue on how to dive deeper to find or to help with error research. 

If you can login at the console then it might be worth using the 'df' command to check that the USB stick is being mounted correctly.    There seem to have been a spate of cases recently where this has not been the case for some reason.

 

The other common problem with similar symptoms is problems with the config/network.cfg  file on the USB stick where invalid entries are present which you get away with on 6.6 but which cause a problem on 6.7.  In such a case renaming this file and letting the system generate a new default one seems the easiest way forward.

[hi2hello]

52 minutes ago, bonienl said:

The error "modprobe: FATAL: Module bonding not found" indicates Unraid is unable to mount the specific module from the flash device.

Either there is file system corruption on the flash device, which needs to be repaired (easiest is to use a Windows PC) or the file itself is broken.

You can download the ZIP file and extract its contents to the flash device to see if it solves your problem.

 

The stick itself seems to be fine. It mounts under OS X without problems, no specific report when checking file system either. I will check the zip file and see If that helps. Thank you for the info and your help!

[hi2hello]

1 hour ago, itimpi said:

If you can login at the console then it might be worth using the 'df' command to check that the USB stick is being mounted correctly.    There seem to have been a spate of cases recently where this has not been the case for some reason.

 

The other common problem with similar symptoms is problems with the config/network.cfg  file on the USB stick where invalid entries are present which you get away with on 6.6 but which cause a problem on 6.7.  In such a case renaming this file and letting the system generate a new default one seems the easiest way forward.

df shows that the stick does not get mounted under 6.7.0. (see my post #3 with the output of the df command above). 

Could you maybe be a bit more specific on what entries in the config/network.cfg could cause problems under 6.7.0 while being fine on 6.6?

I d like to avoid setting up a new server config if possible. 

Thank you for your reply and the kind help

Edited May 13, 2019 by hi2hello

[itimpi]

4 minutes ago, hi2hello said:

df shows that the stick dies not get mounted under 6.7.0. (see my post above). 

Could you maybe a bit more specific in what entries in the config/network.cfg could cause problems under 6.7.0 while being fine on 6.6?

I d like to avoid setting up a new server config If possible. 

Thank you for your reply and the kind help

I would suggest that you first sort out the problem with getting the USB stick to boot correctly.    Copying across the bz* type files from the ZIP version of Unraid would ensure that they are all OK.

 

If you can then do that you can get system diagnostics so that we can see what is in the various configuration files including the network.cfg file.   The suggestion for renaming it was a 'quick' solution so that you could get up and running and then compare the contents of the new file against the old ones (assuming there is even a problem at all in the first place).

Edited May 13, 2019 by itimpi

[hi2hello]

1 hour ago, itimpi said:

I would suggest that you first sort out the problem with getting the USB stick to boot correctly.    Copying across the bz* type files from the ZIP version of Unraid would ensure that they are all OK.

Wouldn’t that be the same than running an update? 

I suppose that If I ll do this the Server wont get up again, running into the exact same error than with updating via web GUI. Or what would be the difference?
Don’t get me wrong, really happy to do so and also really appreciate your help, but already tried three updates via Web-GUI, which -to my understanding- replaced the BZ* type files on the boot stick (that should be identical to the one from the zip?!?). 

Also checked my network.cfg under 6.6. Nothing special in here. Or do you see anything, that could cause an error under 6.7?
(Note: IPADDR[0], GATEWAY and DNS_SERVER1 anonymized, they are fine)

# Generated settings:
IFNAME[0]="br0"
BRNAME[0]="br0"
BRSTP[0]="no"
BRFD[0]="0"
BRNICS[0]="eth0"
DESCRIPTION[0]=""
USE_DHCP[0]="no"
IPADDR[0]="***.***.*.**"
NETMASK[0]="255.255.255.0"
GATEWAY="***.***.*.*"
DHCP_KEEPRESOLV="yes"
DNS_SERVER1="***.***.*.**"
DNS_SERVER2="8.8.8.8"
DNS_SERVER3=""
MTU[0]="1500"
SYSNICS="1"

Sorry to bother you, just try to understand what I’m doing, while doing! 

Edited May 13, 2019 by hi2hello

[itimpi]

That network.cfg file looks fine - the reports I have seen of problems in this area always seem to be on servers with multiple Ethernet ports which does not seem to be the case for your server.

 

Writing the bz* file from the Zip file SHOULD be identical to doing it via the GUI, but since you would be doing this on a different machine it may make a difference.  I have certainly had cases myself in the past where it did make a difference.   On the Unraid server use a USB2 port if possible as these tend to be more reliable than USB3 ones.   Also worth checking that the USB stick still has the label UNRAID (all capitals) - although I cannot think how it might get changed but worth checking.

[hi2hello]

Just one Ethernet-Port, true. Stick is labelled as it should be "UNRAID", all in capitals.
Unfortunately I really do not have any USB 2.0 Ports available on my server.
 

So i shutdown the system, copied all bz*-files from the zip on my mac to the stick, making sure to use an USB2-Port. Re-inserted it into the server, booted and what should I tell you? … 

Bond0 - Error 
No IP, no possibility to get any connection to the server (as before, not even localhost) except for the terminal. Even "diagnostics"-command throws errors and cannot be saved for further examination.  df shows stick is not mounted.

Also renamed the network.cfg and let unraid create a new one. Makes no difference: still Bond0-error. There wasn t even a new network.cfg created by unRaid. 

Rolled back to 6.6.7 - instantly works like a charm. df as expected, network connection present, …
Propably have to wait for 6.7.1 and hope the issue will be resolved

Edited May 13, 2019 by hi2hello

[hi2hello]

Any ideas what else I could try or how to proceed? Wait for an update or contact support?

[hi2hello]

Ok, I solved the problem and thought it might help to share this, just in case anyone is hazzling with similar problems. 
Sidenote: Digging a bit deeper in "the bond0-not-found"-problem in several other unRaid forum topics (and there a plenty of them!), it more or less always turned out, that there is something wrong with the USB-Stick (though I still do not get what causes a problem under 6.7 while it is working flawlessly under 6.6*).

 

Anyway, I deciced to do a fresh and clean install of unRaid 6.6.7, using the unRaid USB-Creator from the download-section.
- I made a backup of the "old" USB-Stick first
- than wiped the USB-Stick and created a new 6.6.7 (on that same stick) with the unRaid USB-Creator
- finally, I took the folder "config" from the previously made backup and put it back on the fresh USB-Stick, overwriting the existing folder (rest remained completely untouched)

- put the stick back into the server and started it

- after unRaid booted into 6.6.7, I repeated the update procedure with the web GUI (tool > update OS) to go for 6.7

- rebooted afterwards

- surprise: 6.7

 

Don’t ask, I don’t get it! A fine USB-Stick under 6.6, not working under 6.7 …
(Probably one of the downsides of running an OS from a USB-Stick. Just sort of being trustworthy, especially for a server, usually reyling on stability - but that might be just my personal opinion). I just hope the server remains stable (as it always used in the past) …

Whatever: Thanks to everyone trying to help! Much appreciated

 

 

Cheers, 
hi2hello

Edited May 16, 2019 by hi2hello

[tayshserve]

Was hoping you would resolve this, thanks for posting. I don’t really have time to tinker so I was forced to just rollback and wait. Anyway... huge help, thank you 

[phileeny]

I had the same issue this weekend and I did more less the same thing as yourself, I initially thought it was something to do with the 10Gbit network cards I have installed, so I remove it and downloaded a brand new copy of 6.6, copied over my config file and reset my network config file and everything was now working fine on 6.6, I then made sure I had a backup of version 6.6 then proceeded to download again 6.7 using tool update os but it didn't work I still lost network connection.

[phileeny]

Downloaded the 6.7 zip file to build a test usb

when extracted with 7zip this doesn't look right

 

"unRAIDServer-6.7.0-x86_64\bzimage"
"unRAIDServer-6.7.0-x86_64\bzimage.sha256"
"unRAIDServer-6.7.0-x86_64\bzmodules"
"unRAIDServer-6.7.0-x86_64\bzmodules.sha256"
"unRAIDServer-6.7.0-x86_64\bzroot"
"unRAIDServer-6.7.0-x86_64\bzroot.sha256"
"unRAIDServer-6.7.0-x86_64\bzroot-gui"
"unRAIDServer-6.7.0-x86_64\bzroot-gui.sha256"
"unRAIDServer-6.7.0-x86_64\changes.txt"
"unRAIDServer-6.7.0-x86_64\license.txt"
"unRAIDServer-6.7.0-x86_64\make_bootable.bat"
"unRAIDServer-6.7.0-x86_64\make_bootable_linux"
"unRAIDServer-6.7.0-x86_64\make_bootable_mac"
"unRAIDServer-6.7.0-x86_64\memtest"
"unRAIDServer-6.7.0-x86_64\config"
"unRAIDServer-6.7.0-x86_64\EFI-"
"unRAIDServer-6.7.0-x86_64\syslinux"
"unRAIDServer-6.7.0-x86_64\bzfirmware"
"unRAIDServer-6.7.0-x86_64\bzfirmware.sha256"

[itimpi]

What are you thinking looks wrong with that?    

[holoX]

Hi guys,

just wanted to let you know that I ran into the same issue. I'll try hi2hello fix tonight. 

Maybe a more common error? @limetech

[phileeny]

5 hours ago, itimpi said:

What are you thinking looks wrong with that?    

.sha256 file isn't in the older zip file.