6.12.8 Unraid completely unusable after USB failure and replacment. 8 hours into troubleshooting. Loads ubuntu fine.

[suchamoneypit]

My server is completely screwed, I am desperate for help or guidance. I have spent 8 hours straight trying to get this working. I started at 5:45pm and im writing this now as of 2am. It was also my home router, opnsense was running as a VM. I work from home and this it's literally taking me offline. I was planning to segregate the two but now Im paying the ultimate price for not getting to it sooner. Once this is (hopefully) resolved thats getting done ASAP.

 

I had a ryzen 2700x and 48GB RAM. I upgraded to a 5950x and 128GB. After rebooting after doing this, unraid failed to start at all. After hours, i realized the USB for unraid would not load even in windows. A dead USB after a reboot? super odd. I realized I had a super recent backup from unraid connect just 8hrs prior. Thought this was my saving grace and I would be back and running shortly. NOPE. Setup a new USB, tried to boot, and now I'm completely stuck. No matter what I do, dockers or VMs don't run and it can never get to GUI. It just boots to a black screen with a typing cursor in the top left.

 

After booting up, the initial few hundred lines of text from unraid show, but then for gui boots it goes to a blank screen with a cursor. For my BIOS, I have csm enabled, SVM enabled, legacy USB support enabled, fast boost and secure boot off. The system boots in ubuntu with the live installer/preview without any issues. I tried switching back to my old RAM. No matter what I do, the GUI will not come up. I get display output throughout the initial boot process. If I try to go to GUI from there, it goes to the blank screen with a cursor once I enter "slim". Same thing in GUI safemode as well. No dockers will launch or VMs. I tried launching a VM from CLI with "vrish start OPNsense" and got "failed to connect to hypervisor". "operation not supported. Cannot use direct socket mode if no URI is set". I've tried this with a fresh unraid install using the official tool both with a new download, and by using a ZIP. Exact same behavior. Im using a Samsung 128GB USB 3.0 flashdrive formatted with Fat32. 

 

You would have thought a flashdrive backup from unraid connect would have been a simple ordeal, it has honestly become a nightmare from me. I followed the exact steps in their "Manual method" for restoring the backup. Im literally going to take the day off of work tomorrow because my whole network is down and I cant get unraid working.

 

BIOS Settings:
SVM enabled (virtualization)
CSM enabled
legacy USB enabled
fast boot off

(I also tried with the BIOS defaults, with SVM off)

 

Main hardware:
ryzen 5950x CPU
Gigabyte Aorus Master B550, Bios F15 (latest is F16 with minor update, F10 supported 5950x)
Tried many RAM configs including one that was working prior

 

I'm booting with UEFI, my folder on the USB is "EFI". It won't boot if I try non UEFI. I was doing this before as well. 

 

And again, this system has posted a hundred times now at this point and live linux (ubuntu) loads and runs just fine. Unraid refuses to. If anyone knows whats going on I would be IMMENSELY thankful. 

 

 

 

 

EDIT: SOLUTION:

 

Something that really made troubleshooting a problem was my unraid server was also my router, with an OPNsense VM. The main problem that kicked this all off was my USB died during my hardware upgrade. Without unraid array started, I also had no router or network, meaning no WebGUI access. For some reason, the local GUI stopped working for me. I believe it was because I also lost my Nvidia Driver plugin, but I will note it did work before without that. In retrospect, the webGUI was likely up the entire time once I setup my backup USB, but I could not access it with no network as OPNsense VM was down.

 

Setting up an old secondary router allowed me to get my network online. I connected the unraid server to this router and got it on a network (manually setting correct IP scheme by editing network.cfg on the USB), and from there I was able to access the WebGUI, setup the new USB license, then get my array started again. From there is was standard setup getting everything back to working order. 

Edited February 29 by suchamoneypit

[suchamoneypit]

I got this diagnostics file from the CLI, when trying booting to nonGUI with a fresh install

tower-diagnostics-20240227-2117.zip

[suchamoneypit]

Diagnostics when ran using my backup USB files. Also the results of running the "df" command which should should if there was a flash mounting issue. 

 

 

 

 

I am now heading to bed, unsuccessful, at at 3am, 9 hours after I started what was supposed to be a standard hardware upgrade. Praying to the computer gods someone out there takes pity on me and knows enough to help me get my stuff back. I mean hell, if anyone directly leads me to or tells me the fix, $20 is yours. 

s-cartographer-diagnostics-20240227-2135.zip

Edited February 28 by suchamoneypit

[JorgeB]

Feb 27 21:16:46 [1707]: br0: soliciting a DHCP lease
Feb 27 21:16:51 [1707]: br0: probing for an IPv4LL address
Feb 27 21:16:55 [1707]: br0: using IPv4LL address 169.254.106.101

 

Server is failing to get an IP address from the DHCP server, usually your router, check router config and/or reset it to defaults.

 

 

[suchamoneypit]

4 hours ago, JorgeB said:

Feb 27 21:16:46 [1707]: br0: soliciting a DHCP lease
Feb 27 21:16:51 [1707]: br0: probing for an IPv4LL address
Feb 27 21:16:55 [1707]: br0: using IPv4LL address 169.254.106.101

 

Server is failing to get an IP address from the DHCP server, usually your router, check router config and/or reset it to defaults.

 

 

Server is my router. OPNsense runs as a VM which is down. It won't get an IP currently. No services will start and I can't get to webGUI to check anything. I'm am troubleshooting with a monitor hooked up to the server. 

Edited February 28 by suchamoneypit

[JorgeB]

So the current issue is you not being able to boot into GUI mode? You could give it a manual IP and access the GUI using a LAN connected computer/device.

[suchamoneypit]

22 minutes ago, JorgeB said:

So the current issue is you not being able to boot into GUI mode? You could give it a manual IP and access the GUI using a LAN connected computer/device.

Yes, it won't boot into GUI mode. And I assume because it's a new USB, if I boot into CLI, nothing works, no VMs or Dockers start. Because the array needs to be set back to its initial config and started. 

 

unRAID runs an OPNsense VM, so even if I manually assign an IP, there is no router for another computer on my network to connect to unRAID. 

 

GUI refuses to load when testing locally. I have a kbm and monitor hooked up right to the machine. Previously when troubleshooting network down situations when I was getting the VM working with OPNsense, loading the GUI locally was of no issue. If the GUI won't load locally, I can't imagine I'll get it up on the network. 

Edited February 28 by suchamoneypit

[ijuarez]

go and get a cheap router so you can get internet, work and plug into server and let it get an ip you can get to the webgui

[JorgeB]

26 minutes ago, suchamoneypit said:

unRAID runs an OPNsense VM, so even if I manually assign an IP, there is no router for another computer on my network to connect to unRAID.

This is not ideal for troubleshooting as you have found, GUI mode may not be working because the Nvidia driver is not installed, but not sure if it's possible to install it without GUI access, @ich777?

 

 

[trurl]

27 minutes ago, suchamoneypit said:

unRAID runs an OPNsense VM

Running Unraid as a VM is not officially supported, but there is a subforum where those users try to help each other.

https://forums.unraid.net/forum/46-virtualizing-unraid/

[suchamoneypit]

7 minutes ago, ijuarez said:

go and get a cheap router so you can get internet, work and plug into server and let it get an ip you can get to the webgui

I'll see if I can get that going and if I can get to webGUI

 

 

2 minutes ago, JorgeB said:

This is not ideal for troubleshooting as you have found, GUI mode may not be working because the Nvidia driver is not installed, but not sure if it's possible to install it without GUI access, @ich777?

 

 

I will say that even without the Nvidia plugin, it worked just fine with my 1060 before for basic display output 

[suchamoneypit]

Just now, trurl said:

Running Unraid as a VM is not officially supported, but there is a subforum where those users try to help each other.

https://forums.unraid.net/forum/46-virtualizing-unraid/

unRAID is not a VM, OPNsense (my home router), runs as a VM on unRAID itself. 

[trurl]

Sorry, reading comprehension letting me down.

[ijuarez]

the assumption is that you have your ISP modem directed connected to the unraid box when the OPNsense vm runs.

based on the diags you're not getting a ip to the unraid box.

So i presume you connected something else to the ISP modem to make sure you have internet or its handing out an ip?

 

 

[ich777]

19 minutes ago, JorgeB said:

but not sure if it's possible to install it without GUI access, @ich777?

@suchamoneypit remove the file:

/boot/config/plugins/nvidia-driver.plg

from your boot device and then boot the server again.

 

By doing that your server won‘t install the plugin on boot.

[JorgeB]

1 hour ago, ich777 said:

By doing that your server won‘t install the plugin on boot.

The driver appears to be installing in the syslog, but when you go to lspci.txt it's not installed:

 

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. GP106 [GeForce GTX 1060 6GB] [3842:6163]
07:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. GP106 High Definition Audio Controller [3842:6163]

 

Can you see in the diags why? I was thinking that not booting into GUI mode can be because there's no driver installed, at least it helped before in some cases.

[suchamoneypit]

1 hour ago, ijuarez said:

go and get a cheap router so you can get internet, work and plug into server and let it get an ip you can get to the webgui

I am still holding my breath but I got an older router working and with my old ISP I have yet to disconnect services with but still had an account with (switched ISPs a few months ago), but after doing this, and then manually editing my network.cfg, I have been able to get to the webGUI on that network. Currently slowly restoring things back as they way and verifying everything is still working at each step. This may have have been the key. As I suspected when I got in nothing was running because even before I could start the array it made me transfer my license to the new USB. I will provide an update soon but I'm feeling quite hopeful which has not been the case for nearly 10 hours of troubleshooting prior. 

[suchamoneypit]

11 minutes ago, JorgeB said:

The driver appears to be installing in the syslog, but when you go to lspci.txt it's not installed:

 

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. GP106 [GeForce GTX 1060 6GB] [3842:6163]
07:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. GP106 High Definition Audio Controller [3842:6163]

 

Can you see in the diags why? I was thinking that not booting into GUI mode can be because there's no driver installed, at least it helped before in some cases.

I will say when I checked my USB, the .plg file did not exist. I'm guessing it never downloaded because the server never had Internet access since the new flash drive

[JorgeB]

2 minutes ago, suchamoneypit said:

I will say when I checked my USB, the .plg file did not exist. I'm guessing it never downloaded because the server never had Internet access since the new flash drive

That makes sense, with the driver loaded the GUI boot option should work.

[ich777]

9 hours ago, suchamoneypit said:

Gigabyte Aorus Master B550, Bios F15 (latest is F16 with minor update, F10 supported 5950x)

Did you also change the motherboard?

Have you yet tried to delete the network.cfg from /config and reboot?

 

9 minutes ago, JorgeB said:

That makes sense, with the driver loaded the GUI boot option should work.

The driver is not installing and only the md5 sum exists on the flash device.

[suchamoneypit]

2 hours ago, ijuarez said:

go and get a cheap router so you can get internet, work and plug into server and let it get an ip you can get to the webgui

More to come regarding this, this seemed to be the key (I think you're gonna get the $20 bounty)

 

 

2 hours ago, ijuarez said:

the assumption is that you have your ISP modem directed connected to the unraid box when the OPNsense vm runs.

based on the diags you're not getting a ip to the unraid box.

So i presume you connected something else to the ISP modem to make sure you have internet or its handing out an ip?

 

 

Initially no, I had nothing else to hook up to the ISP modem so I had no internet or wifi. I had to do everything and troubleshooting from my phones data or hotspot data which was terribly slow. I do now have unraid connected to a router thats getting internet. I actually have two ISPs. one I had essentially in a dormant setup as I was going to disconnect that service as unraid and OPNsense run my new provider which is fiber. I hooked back up my old ISP's router so I have wifi and internet. 

 

 

 

Edited February 28 by suchamoneypit

[suchamoneypit]

36 minutes ago, ich777 said:

Did you also change the motherboard?

Have you yet tried to delete the network.cfg from /config and reboot?

 

The driver is not installing and only the md5 sum exists on the flash device.

 

49 minutes ago, JorgeB said:

That makes sense, with the driver loaded the GUI boot option should work.

 

 

No motherboard change, only CPU and RAM. 

 

Im going to post a better explanation later, im tried and have to work again now that I have internet. 

 

Long story short: my server is back and running. The only exception currently is that although my USB plugins folder has all my plugins, in unraid it shows zero installed plugins. The local GUI is still not working, but with zero plugins the nvidia driver one is not installed. 

 

 

I'm going to have to dig into that later unless in the meantime someone has a solution or answer to that. The server is running with the new CPU but old RAM. now that the flashdrive is working later today im also going to put back in the new RAM. 

[ich777]

@suchamoneypit gui-mode will only work if you install the Nvidia Driver plugin and reboot afterwards.

[suchamoneypit]

8 minutes ago, ich777 said:

@suchamoneypit gui-mode will only work if you install the Nvidia Driver plugin and reboot afterwards.

Do you know of a good way to restore my plugins? The folders are there, but unraid seems to not like it and lists errors for them all. 

 

And notably before I ever installed the nvidia driver plugin, the GUI worked just fine through my 1060. I only ever got the plugin once I wanted to do GPU transcoding with Plex/Emby. 

[ich777]

10 minutes ago, suchamoneypit said:

Do you know of a good way to restore my plugins? The folders are there, but unraid seems to not like it and lists errors for them all. 

Yes.

 

First of all, this happened because you had no active internet connection after you‘ve restored you USB boot device.

 

Go to /boot/config/plugins-error and move all plg files from there into /boot/config/plugins and reboot, this will reinstall all your plugins that are now in the errors tab, of course only if you have a active internet connection.