Upgrade to 6.10 docker offline

[jmztaylor]

Upgraded to 6.10 yesterday and woke up this morning to docker and VMs offline.  Trying to reboot the server but stuck at Retry unmounting disk share.

tower-diagnostics-20220522-0934.zip

[jmztaylor]

After awhile the server finally rebooted and everything seems to be up and running now without issue. Would like to find out the issue if possible.

[itimpi]

You should enable the syslog server to get some log information that survives a reboot in case it happens again.

[jmztaylor]

2 minutes ago, itimpi said:

You should enable the syslog server to get some log information that survives a reboot in case it happens again.

diagnostics was ran before reboot.

[itimpi]

Your syslog is full of crashes on macvlan.   This can normally be fixed by upgrading to 6.10.1 and using iplan instead under docker.

 

later you start getting btrfs errors on the loop2 device which is the one holding the docker image file so this is corrupt and will need recreating.    Not sure if there are also problems at the btrfs level on the cache pool.

[JorgeB]

You should disable Intel VT-d ASAP, looks like the same issue that affects the HP Microserver Gen8, likely because they use the same NIC driver, more info below:

 

https://forums.unraid.net/topic/123620-unraid-os-version-6100-available/page/8/#comment-1129501

 

[jmztaylor]

44 minutes ago, itimpi said:

Your syslog is full of crashes on macvlan.   This can normally be fixed by upgrading to 6.10.1 and using iplan instead under docker.

 

later you start getting btrfs errors on the loop2 device which is the one holding the docker image file so this is corrupt and will need recreating.    Not sure if there are also problems at the btrfs level on the cache pool.

 

Where is the setting for iplan?  Also nuked the docker.img file and rebuilt from previous apps.  

 

38 minutes ago, JorgeB said:

You should disable Intel VT-d ASAP, looks like the same issue that affects the HP Microserver Gen8, likely because they use the same NIC driver, more info below:

 

https://forums.unraid.net/topic/123620-unraid-os-version-6100-available/page/8/#comment-1129501

 

 

I don't specifically have VT-d in my BIOS but I am assuming its just the virtualization technology setting.  Is there a bug tracker to follow this?  I am also assuming VMs will not work till this is corrected?

 

Edit: Also parity check is showing 2900 errors so far 3% into it.  Is there a way to find out where they are to nuke those files.  Array data does not concern me too much with loss. 

Edited May 22, 2022 by jmztaylor

[itimpi]

55 minutes ago, jmztaylor said:

Where is the setting for iplan? 

Under Settings->Docker with advanced view enabled (Docker Custom Network Type).   I think the array needs to be stopped to change it.

 

[jmztaylor]

1 minute ago, itimpi said:

Under Settings->Focker with advanced view enabled (Docker Custom Network Type).   I think the array needs to be stopped to change it.

 

Ok missed the advanced view switch.  Here is latest diags with 6.10.1 installed and iplan enabled.  

tower-diagnostics-20220522-1149.zip

Edited May 22, 2022 by jmztaylor

[JorgeB]

1 hour ago, jmztaylor said:

I don't specifically have VT-d in my BIOS

It might be called a different thing, like Intel Virtualization technology or similar, alternatively add intel_iommu=off to syslinux.cfg append line, in either case check after booting that IOMMU is really disable, click on system info, top right of the GUI, you should not use the server until that's done.

 

 

[jmztaylor]

Just now, JorgeB said:

It might be called a different thing, like Intel Virtualization technology or similar, alternatively add intel_iommu=off to syslinux.cfg append line, in either case check after booting that IOMMU is really disable, click on system info, top right of the GUI, you should not use the server until that's done.

 

 

 

Yeah I disabled Intel Virtualization Technology.  Assumed it was the same thing

[JorgeB]

It was still enable in the last diags posted.

[jmztaylor]

1 minute ago, JorgeB said:

It was still enable in the last diags posted.

 

It for sure is disabled and VMs are failing to start due to virtualization turned off.  I have added that line to syslinux to be sure and rebooted.

tower-diagnostics-20220522-1220.zip

[JorgeB]

Now it's disable, likely there are two different settings in the BIOS, the one you disabled is for VT-x, you can re-enable that and still run VMs as long an no hardware if being passed through (for that you need VT-d).

[jmztaylor]

1 minute ago, JorgeB said:

Now it's disable, likely there are two different settings in the BIOS, the one you disabled is for VT-x, you can re-enable that and still run VMs as long an no hardware if being passed through (for that you need VT-d).

 

What is the solution for the parity sync errors.  If this issue causes corruption, what should be trusted?

[JorgeB]

You now should also run a correcting scrub on the pool.

[jmztaylor]

1 minute ago, JorgeB said:

You now should also run a correcting scrub on the pool.

 

Is there documentation for that?

[JorgeB]

Just now, jmztaylor said:

If this issue causes corruption, what should be trusted?

Without pre-existing checksums for the files not much you can do other that correct parity, any files that are corrupt in the pool will be listed in the syslog after the scrub.

[JorgeB]

1 minute ago, jmztaylor said:

Is there documentation for that?

 

Click on cache on the main page then scroll down until the scrub section.

[jmztaylor]

1 minute ago, JorgeB said:

 

Click on cache on the main page then scroll down until the scrub section.

 

Didn't think my cache pool had an issue.  Parity was showing sync errors on the array. 

[jmztaylor]

FWIW, I ran the scrub and this is what shows

 

Edit: Ran repair right after and all corrupted files are plex phototranscoder files.   So no big deal with those.

 

tower-diagnostics-20220522-1332.zip

Edited May 22, 2022 by jmztaylor

[jmztaylor]

So the cache pool is returning no more errors but parity check on array is returning errors.  Last run there was zero.  So between the upgrade and this issue some corruption was introduced.  What should be believed?  That parity is still good and needs to do corrections or restart with blank parity and trust the data is good?

[JorgeB]

You should run a correcting check now, you can then run a non correcting one after a couple of days to confirm no more issues.

[jmztaylor]

7 hours ago, JorgeB said:

You should run a correcting check now, you can then run a non correcting one after a couple of days to confirm no more issues.

 

Is there a bug tracker for the iommu issue?

[JorgeB]

Not for now, but I'll post an update in the announcements thread when there's new info, while this isn't an Unraid issue I know that LT is looking at it.