[SOLVED] Kernal Panic: VFS cannot open root device

[Unknownkwita]

Server

Unraid version 6.9.2 since release with Pro license

CPU: AMD Ryzen 5900x

GPU: Nvidia GTX 1080

Memory: 64GB 3200MHz DDR4

USB Key: Kingston 16gb USB 2.0

2x SilverStone ECS02 LSI 9211-8i SAS/SATA

various plugins loaded including dynamix.

 

Problem

After restarting my server to troubleshoot my vpn docker container, it failed to boot. It displays the error "Unable to mount root fs on unknown-block..." The full error is in the attached picture. No hardware changes have occured in 6+ months. I havent had an issue rebooting in the past. Diagnostic zip file from immediatly before the error is attached. My last USB backup is 6months old from when I upgraded  from 6.9.1 to 6.9.2, so I would prefer not to use that if possible. I have created a backup of the current USB key.

 

Trouble shooting performed

1. Attached a screen, keyboard and mouse and booted in safemode. Same error occurs.
2. Updated bios on my MSI x570 Creation. No change to the error.
3. After reading this thread and this thread which seemed to have a similair issue. I tried adding root=sda and/or root=sda1 to syslinux.cfg. The error persists but changes from "null" to which ever value I entered. All futher testing has these returned to there original state
4. I installed a fresh copy of unraid using the creation tool on a variety of USB including wiping the original usb key. I was not able to get any of these to boot, they dont even get to the unraid boot selection screen, the system just goes straight into the bios. I used the make_bootable.bat, run as admin, enter twice. Doesnt change anything. I dont think its the bios settings. Everything is configured to UEFI. all USB keys were recognized by the bios and selected as the boot location.
5. To test if my system can actually boot anything, I tried a windows 10 install media. It loaded without issue.
6. I copied my backup back onto the original USB key and run make_bootable.bat. The bios recognizes the usb key, boots, shows the unraid boot menu and then shows the original error.
7. I have a second test unraid server I use. I shut it down and inserted the USB key with the unraid that is providing the error. The same error appears. This suggests the issue is related to the USB key or the files on the USB key.
8. I tried booting my USB key from my test server in my main server, it wont boot. same as item 4. So maybe it is the PC hardware?

 

 

At this point I am not sure what the problem is. I am going to buy a new USB to test with, but not sure why my other USB keys werent booting. Based on some google-fu, it could be the memory? Any thoughts or suggestions?

 

 

ditto-diagnostics-20211109-1906.zip

[itimpi]

I would try downloading the zip file for the release and extracting all the bz* type files overwriting the ones on the flash.    This seems to quite often solve any issues reading them from the flash drive.

[Unknownkwita]

You good sir are MAGNIFICENT!

 

Replaced the bz* files and it booted straight in. Still confused what caused this issue but thank you so much. 😁

[itimpi]

14 minutes ago, Unknownkwita said:

You good sir are MAGNIFICENT!

 

Replaced the bz* files and it booted straight in. Still confused what caused this issue but thank you so much. 😁

Not sure why, but for some reason this seems to reasonably frequently be required after an upgrade to fix problems reading the newly written bz* type files off the flash drive.

[Raduzard]

Server

Unraid version 6.11.5
USB flash drive: Kingston DataTraveler Exodia 32 GB
HP ProLiant MicroServer N40L
BIOS: HP Version O41. Dated: 07/29/2011
CPU: AMD Turion™ II Neo N40L Dual-Core @ 1500 MHz
HVM: Enabled
IOMMU: Disabled
Cache: 256 KiB, 2 MB, 0 KiB
Memory: 8 GiB Other (max. installable capacity 8 GiB)
Network: bond0: fault-tolerance (active-backup), mtu 1500
Kernel: Linux 5.19.17-Unraid x86_64
OpenSSL: 1.1.1s

 

Plugins:

community.applications.plg
dynamix.unraid.net.plg
unassigned.devices.plg
unassigned.devices-plus.plg

 

Problem

I run into this problem very often (boot fails with kernel panic message). I'm new to Unraid and this is a new installation that is only 3 weeks old.

 

Temporary fix

I perform itimpi's solution (I overwrite the bz files with those from the zip file) and then it boots fine. However, the problem eventually happens again. I didn't make any changes nor add any plugins.

 

Observations

Performing a diff with Beyond Compare between the problematic USB flash drive and a good image (either the downloaded zip file or a newly created Unraid flash drive), I can see that some of the bz* files have changed (the combination of changed files varies). However, their corresponding *.sha256 checksum files don't change.

 

Another observation is that a huge amount of FSCK*.REC files get created in the root directory (as many as 576 files).

 

I'd like to know if I just have a bad USB flash drive and should I just replace it? I have a Kingston DataTraveler Exodia 32 GB which I thought were reliable.

 

Thanks.

[itimpi]

58 minutes ago, Raduzard said:

Another observation is that a huge amount of FSCK*.REC files get created in the root directory (as many as 576 files).

These files being created indicate corruption has been detected.

 

The fact you say the bz*type files are changing is a big worrying - that should never happen.       Whether it is a problematical USB drive or something else like the USB port is not clear.

[shuklaash]

On 11/9/2021 at 7:30 AM, itimpi said:

I would try downloading the zip file for the release and extracting all the bz* type files overwriting the ones on the flash.    This seems to quite often solve any issues reading them from the flash drive.

THANK YOU!!! I have been trying random things for a few hours and when I came across this post I tried it and it worked!

Edited January 21 by shuklaash