VMs not auto-starting after upgrade from 6.12.6 to 6.12.8

[harshl]

Hey all,

 

I have 3 VMs on one of my unraid servers and after an upgrade to 6.12.8, none of them will autostart after a reboot. Not sure what is going on or how to begin diagnosing it.

Dockers auto start as configured, the VMs, though configured to autostart, do not.

Manually starting the VMs after boot works perfectly fine.

 

Diagnostics attached. Let me know if you need other details, I don't even know what to provide for this.

 

Thanks for any guidance you can provide!

-Landon

rackserver-diagnostics-20240321-1948.zip

[harshl]

Oh! I just found this!

 

VM Autostart disabled due to vfio-bind error

 

That is clearly the issue. I will start looking at what might be wrong, but I don't believe any hardware has changed so I am not sure what is producing that error.

 

VFIO log looks fine to me?

Loading config from /boot/config/vfio-pci.cfg
BIND=0000:01:00.0|10de:1f06 0000:01:00.1|10de:10f9 0000:01:00.2|10de:1ada 0000:01:00.3|10de:1adb 0000:09:00.0|1b73:1100
---
Processing 0000:01:00.0 10de:1f06
---
Processing 0000:01:00.1 10de:10f9
---
Processing 0000:01:00.2 10de:1ada
---
Processing 0000:01:00.3 10de:1adb
---
Processing 0000:09:00.0 1b73:1100
---
Devices listed in /sys/bus/pci/drivers/vfio-pci:
vfio-pci binding complete

 

I don't have any references in the syslinux config like we used to do.

 

The only passthrough I even have is a spinner hard drive I am passing to a BlueIris server, which seems to be working perfectly.

 

No other hardware passthrough going on as far as I can see.

 

Thoughts?

-Landon

Edited March 22 by harshl
Additional detail provided.

[SimonF]

5 hours ago, harshl said:

Oh! I just found this!

 

VM Autostart disabled due to vfio-bind error

 

That is clearly the issue. I will start looking at what might be wrong, but I don't believe any hardware has changed so I am not sure what is producing that error.

 

VFIO log looks fine to me?

Loading config from /boot/config/vfio-pci.cfg
BIND=0000:01:00.0|10de:1f06 0000:01:00.1|10de:10f9 0000:01:00.2|10de:1ada 0000:01:00.3|10de:1adb 0000:09:00.0|1b73:1100
---
Processing 0000:01:00.0 10de:1f06
---
Processing 0000:01:00.1 10de:10f9
---
Processing 0000:01:00.2 10de:1ada
---
Processing 0000:01:00.3 10de:1adb
---
Processing 0000:09:00.0 1b73:1100
---
Devices listed in /sys/bus/pci/drivers/vfio-pci:
vfio-pci binding complete

 

I don't have any references in the syslinux config like we used to do.

 

The only passthrough I even have is a spinner hard drive I am passing to a BlueIris server, which seems to be working perfectly.

 

No other hardware passthrough going on as far as I can see.

 

Thoughts?

-Landon

Looks like you had a 2060 bound to vfio on 1:00.x which is no longer there. and a device at 09:00.0 if you don't need them delete the vfio file and next reboot you should be ok.

[harshl]

17 hours ago, SimonF said:

Looks like you had a 2060 bound to vfio on 1:00.x which is no longer there. and a device at 09:00.0 if you don't need them delete the vfio file and next reboot you should be ok.

Thank you @SimonF! I'm assuming I need to remove /boot/config/vfio-pci.cfg completely, or should I just empty the file and leave it? Any difference?

 

Interesting that those things are in there. I am not sure where they would have come from. If memory serves me, this may have been passed down from my desktop unraid setup to a dedicated server, but that was literally years ago. So something has changed in unraid as it relates to this file and autostarting VMs, that, or this file was magically placed on my flash drive somehow, which has been in this server without removal for years.

 

Anyway, I appreciate the guidance and I suspect this will fix my particular issue.

 

Thanks again,

-Landon

[SimonF]

6 hours ago, harshl said:

Thank you @SimonF! I'm assuming I need to remove /boot/config/vfio-pci.cfg completely, or should I just empty the file and leave it? Any difference?

 

Interesting that those things are in there. I am not sure where they would have come from. If memory serves me, this may have been passed down from my desktop unraid setup to a dedicated server, but that was literally years ago. So something has changed in unraid as it relates to this file and autostarting VMs, that, or this file was magically placed on my flash drive somehow, which has been in this server without removal for years.

 

Anyway, I appreciate the guidance and I suspect this will fix my particular issue.

 

Thanks again,

-Landon

If you are not using PCI passthru you can either leave blank or remove. Changes where made to stop autostart if vfio binding fails as could indicate PCI numbers have chaged due to hardware removal/additions etc. If the VM autostarts and now as the wrong device is allocated to the VM due to PCI number changes it could lookup or crash the host.

 

Before   

07:00.0 VGA Card(Passthru to VM)

08:00.0 Disk Controller(Unraid)

 

After

07:00.0 DIsk Controller(Unraid)

 

If the VM autostarts the diskcontroller would be allocated to the VM and all the Unraid disks would not been seen on the host as a simple example.

 

 

[harshl]

Thank you @SimonF for taking the time to respond and explain the changes that have indeed taken place in software.

 

I have some hardware work to do on the server in the coming days, so I will have the opportunity to test.

 

Thanks again!

-Landon