Upgrade to 6.8.0 stable went very bad - pfsense and dockers busted


Recommended Posts

I have had no issues upgrading between lots of releases in the past, so got up early thinking this would be as simple as the others.

This is on an AMD Ryzen 7 2700 8 core.

After the restart, the array came up - but that was as good as I got it.

Re pfsense VM : (I can ignore all my other VM's that are for work at this stage)

My autostart VM is pfsense - but when I vnc'ed to it, the last line was 'masks .......' and nothing more was happening - was not boting. 

The single CPU allocated to that VM was showing 100%.  All others no activity. 

pfsense is critical obviously, I have the original physical device I can start up but I would prefer to get this vm working on 6.8.0 

 

Re dockers :

The dashboard did not show any dockers as running, even the autostart ones.

The docker tab was just unresponsive - just the spinning lines.

The log suggests it was autostarting the dockers - but they are not visible in the dash or the dockers tab. 

I tried a new browser that had never run unraid console thinking it was a browser cache issue - no better.

 

I got the diagnostics downloaded (first time I did that ever).

After leaving it for a bit still no good.

Took the restore option (another first for me) and with tight but cheeks, waited.  

All came back on 6.7.2 and is working fine.

If any of you experts can help, that will be great.  I have users on my VM's etc in the day so I get one shot a day to do this and have to get up at 3AM.

tower-diagnostics-20200103-0406.zip

Link to comment

Something goes wrong in the passthrough of your ethernet controller.

Jan  3 04:03:45 Tower kernel: vfio-pci 0000:09:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
Jan  3 04:03:45 Tower kernel: vfio-pci 0000:09:00.1: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
Jan  3 04:03:45 Tower kernel: vfio-pci 0000:0b:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
Jan  3 04:03:45 Tower kernel: vfio-pci 0000:0b:00.1: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff

You may want to try to new BIND method, see the release notes of Unraid 6.7

 

Check also if a newer BIOS for your motherboard is available.

Edited by bonienl
Link to comment

Thanks for the assistance. 

QQ - Does this also relate to the dockers that are not able to be used in the dockers tabs, or visible on the dashboard (even though they start ini the logs).

 

Re the bind thing, the notes for 6.7 state 'Since it appears that the xen-pciback/pciback kernel options no longer work', but in my case, everything has been working fine on all the 6.7.x's.  

 

I currently have these IOMMU for the 4 port NIC :

IOMMU group 26:    [8086:10c9] 09:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 27:    [8086:10c9] 09:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 28:    [8086:10c9] 0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 29:    [8086:10c9] 0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

 

And only have this in the flash startup as per spaceinvaders excellent instructions that allowed me to then allocate the 4 NIC ports to the pfsense vm :

 

append vfio-pci.ids=8086:10c9 pcie_acs_override=downstream,multifunction initrd=/bzroot

 

This has all been working fine on all the 6.7.x releases.

 

With regards to the instructions on that 6.7.0 release notes, it states 'This is accomplished by specifying the PCI ID(s) of devices to bind to vfio-pci in the file 'config/vfio-pci.cfg' on the USB flash boot device'.  I do not currently have this file there.

 

So - 

 

do I need to create the file config/vfio-pci.cfg, then edit it to have :

 

BIND="09:00.0 09:00.1 0b:00.0 0b:00.1"

(note, the quotes seem to be needed for multiple devices after I researched this more)

 

and should I then remove the 'vfio-pci.ids=8086:10c9' from the current boot startup ?

 

Then set pfsense to NOT autostart, reboot my Unraid 6.7.2, and check the VM shows the 4 NIC's, then start it ? If it starts up, again re-try the upgrade to 6.8.0 ?

 

Thanks again in advance.

 

 

 

 

 

Link to comment

So.....

 

My first crack of this did not go so well.

After changes as above (but no bios as yet - one thing at a tme), then unraid boot, my PC's could not connect to it.

I assume the nic's were not reserved and unraid decided to use one of them or something.

As I had no monitor/keyboard on my unraid, I flipped out the USB boot stick and reversed the changes on my PC.

Booted back fine.

 

I have now added a monitor, keyboard and mouse and will attempt this change again another early morning.

Link to comment
  • 2 weeks later...

so... this new vfio bind method does not work.

You can see my issues, and others in this post.

So I am back to stage 1 with my unraid upgrade.  Maybe 6.8.1 will be better......

Have to give the users a week of stable operation before I try that again.

On the upside, I have installed my PCIE-X1 graphics card, freeing up the High Speed NVME port for future upgrade.

Link to comment
  • 2 weeks later...

So I tried 6.8.1 now (after the 6.8.0 issues, and after running around with the new vfio bind option which did not work for me - see the other post if interested).  I tried pulling plugins but same issue after reboot.

 

I have two issues :

 

As reported in this thread, the dockers tab just spins and spins, and the dashboard never shows them.  I do know thy have started and I can use them all (as most of mine start up with custom: br0 and hence their own IP address so I dont need to run them from the console).  

 

My pfsence vm issue as reported above.  The 4 nics are reserved, I mave the same iommu's as before the upgrade, but when I start the VM, I get this in the log : 'Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff' .    

Edited by vw-kombi
Link to comment

Another week gone.

I stop my vm from autostartng, so ignore that issue.  I can recreate from scratch and restore.

The dockers view either in dashboard or in the dockers app is what is stopping me from upgrading to 6.8.1.

 

Cant anyone give me something to try ?

 

Or should I just start again from scratch with a fresh install which will take me a few days of configuring/restoring, but its been over 2 weeks now so I could have done that a few times over.

 

Link to comment
9 minutes ago, vw-kombi said:

The dockers view either in dashboard or in the dockers app is what is stopping me from upgrading to 6.8.1.

I doubt it will make a difference, but you could try 6.8.2

 

Have you done the normal recreate docker image procedure? It's normally relatively painless, a few minutes of your time and some download time.

Link to comment

I guess I could do the recreate docker image procedure - but i don't know why I seem to be the only one with this issue.

I suspect that will need internet I believe, so I would have to get my pfsense VM working first - hence even more downtime.

Or I can dust off my physical router laptop machine first and move to that.

6.8.2 still not showing on my upgrade options - maybe it is RC still ?

Link to comment
  • 2 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.