(Solved) Kernel Panic (Unexpected reschedule of offline CPU?)

[dingd0ng]

Hi

I have been running Unraid mostly without problems for about 18 months now, but after rebooting the server yesterday I can no longer access it.

I get to the blue screen where I can choose to boot Unraid with or without GUI or Safe Mode, and the first time this problem occurred I chose to boot using the GUI as I always do. I then got to "Loading bzroot", but it never progressed from there. I then restarted the server using the button on the chassis, as nothing was happening.

The next time I tried booting using Safe Mode, and this time i got beyond "loading bzroot", but after a few seconds I got a Kernel Panic. I have seen a few of these now after trying to boot several times, and they always have a warning regarding CPU, but I think they are somewhat different each time.

The last one I saw said among other things

"Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0). Unexpected reschedule of offline CPU#0! Warning: CPU: 2 PID: 1 at arch/x86/kernel/smp.c.128 native_smp_send_reschedule+0x1a/0x32. Modules linked in: CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.19.56-Unraid #1. Hardware name (name of my MB). RIP: 0010:native_smp_send_reschedule+0x1a/0x32."

 

I backed up the boot USB by copying the files to my mac (don't know if this means I'm missing some files backing up this way), made a new boot usb using the unraid tool and connected it to the server. This time I got straight into the Unraid GUI as usual, however since this was an unregistered copy of Unraid, I couldn't do very much. I could however see all my disks (although not connect to them as this was unregistered).

 

Doesn't this indicate that the problem with my server isn't hardware related? I don't think I could boot into the unregistered version if there was a hardware related problem, since the hardware is the same.

I haven't made any changes to the server lately except update a few plugins and dockers the normal way.

 

So how do I proceed to figure out the cause of this error? What files from the USB drive are needed for troubleshooting?

 

Could I safely just move the backup of the USB boot disk to another USB drive and boot from this? Seing as the only difference between booting from the current USB boot drive (which doesn't work) and when I booted from the unregistered usb drive (which worked) is the files on the drives and the drives themselves. So swapping USB boot drives wouldn't fix my problem unless the problem is the drive itself? And if the drive itself was faulty, would I even have been able to make a backup of it using my mac?

This leads me to belive the problem is something with the files on the drive. Moving them to another drive then would probably result in the same error, only also giving me an error about using the wrong drive?

 

Hopefully someone here can help me.

 

Unraid version: I think it is 6.7.2, since in the USB backup there is a file changes.txt which was updated june 2019 with Unraid 6.7.2.

CPU: Intel i3 3570k

Motherboard: Asus ROG Strix Z370-F

Cache drive: WD Black intern M.2 PCIe SSD 512 GB

Controller: LSI SAS 9211 8i (all array disks are connected to this)

Disks: Assorted, total of 8, mostly Seagate 10 TB, some smaller

 

Plugins (list from boot drive backup, assume this is all of them, haven't installed any for a long time):

- Ca Backup

- Ca Turbo

- Community Applications

- Custom Tab

- Dynamix 

- Dynamix Cache Dirs

- Dynamix SSD Trim

- Dynamix System Autofan

- System info

- System temp

- Fix Common Problems

- Nerdpack

- Preclear Disks

- Server Layout

- Statistics Sender

- Tips and Tweaks

- Unassigned Devices

- Unbalance

 

Dockers (I think this is all of them):

- Handbrake

- Duckdns

- Binhex-plexpass

- Binhex-plex

- Binhex-delugeVPN

- Tautulli

- MakeMKV

- MKVToolNix

- Binhex Sonarr

- Binhex Radarr

- Sabnzbd

 

Virtual Machines:

- Windows 10

 

 

 

[Edit: Following the steps suggested below seemed to work. Haven't had any problems for a week or so. Still don't know what the problem really was.]

Edited June 16, 2020 by dingd0ng
Marked as solved

[JorgeB]

Try recreating the flash drive: backup current one, re-doit using the USB tool, restore config folder from backup.

[dingd0ng]

So I should just do these steps?

1. Backup current USB boot drive (already done)

2. Use the same (?) USB boot drive to make a fresh boot drive using the Unraid tool

3. Move complete config folder from backup to the boot drive

4. Boot Unraid using 'new' boot drive

5. Success?

 

Is there anything I risk losing this way? Will it restore my array settings? All my dockers and plugins including settings?

[itimpi]

30 minutes ago, dingd0ng said:

So I should just do these steps?

1. Backup current USB boot drive (already done)

2. Use the same (?) USB boot drive to make a fresh boot drive using the Unraid tool

3. Move complete config folder from backup to the boot drive

4. Boot Unraid using 'new' boot drive

5. Success?

 

Is there anything I risk losing this way? Will it restore my array settings? All my dockers and plugins including settings?

Yes.

 

all your settings and licence key are stored in the config folder so if these files are all intact and not corrupted your system will come up exactly as it used to.    This set of steps will help if for any reason Unraid is having problems loading any of the bz* type files on the flash when booting.

[dingd0ng]

Thanks so far. Only one more question before giving it a go:

Before this error I was running Unraid 6.7.2 I think. When recreating the USB boot drive I can only choose 6.8.2 or 6.8.3. Is it a problem that the OS version doesn't match the new OS version? And if so, is it possible to make a new boot drive using the old OS 6.7.2?

[JorgeB]

Use the latest release, upgrade should be straight forward, but always good to read the release notes.