Multiple Issues VT-d/Docker Service/Unmountable Cache drive

[SgtHobNob]

Hello, sorry I'm a total novice with Unraid and related technologies, I've been running it without issue for a few years and not really done much tinkering since I set it up.

 

Today I updated from version 6.9.2 to 6.10.2, upon updating I was no longer able to connect, my NIC is blacklisted. To counter this I disabled VT-d and rebooted which allowed me to access Unraid again. I didn't think it would be a big deal as I don't run any VMs, however the Docker Service failed to start. I downgraded back to 6.9.2 and re enabled VT-d, the Docker Service still failed to load along with Libvirt Service.

 

Also, my cache drive is unmountable, I'm not sure if this is related but it's lower on my list of priorities as I need to get docker up and running ASAP.

 

Your help would be appreciated, right now I'm at a total loss and have already spent hours trying to get everything running again.

 

Thanks,

 

 

cookie-diagnostics-20220604-2120.zip

[trurl]

Did you make any changes since cache became unmountable?

[SgtHobNob]

No changes, the only thing I've done is start a parity check which I think I did before I pulled the diagnostics.

[JorgeB]

There are some recovery options here, but based on the error not very optimistic, cache fs might be lost.

 

Do you mind confirming the sequence of events, my suspicion so far has been that the NIC is connected to this issue, since all 5 or 6 models affected found so far use one, also believe that just having the NIC is not sufficient to have issues, since for example Dell servers with the same NIC appear to be unaffected, but by your description it might not be, so please confirm if this is correct:

 

-You updated from v6.9.2 directly to v6.10.2, never used any earlier v6.10 release

-After updating to v6.10.2 you didn't have GUI access, because the NIC was blacklisted due to vt-d being enable, but the array would have still started since autostart is enable

-At this point you did a clean shutdown/reboot and disabled vt-d, i.e., you didn't unblacklist the NIC and never used v6.10.2 with vt-d and the NIC enable, this is the most important question, including if the shutdown was clean or not

-After disabling vt-d you noticed the issues and downgraded back to v6.9.2

 

Is all of the above correct?

 

 

[SgtHobNob]

Hi JorgeB,

 

-You updated from v6.9.2 directly to v6.10.2, never used any earlier v6.10 release

• I cannot be 100% certain it was 6.9.2, I used the GUI to downgrade and it was the only option available, so I could have incorrectly assumed.

-After updating to v6.10.2 you didn't have GUI access, because the NIC was blacklisted due to vt-d being enable, but the array would have still started since autostart is enable

• Yes, I assume the array did start, I did do graceful shutdowns each time.

-At this point you did a clean shutdown/reboot and disabled vt-d, i.e., you didn't unblacklist the NIC and never used v6.10.2 with vt-d and the NIC enable, this is the most important question, including if the shutdown was clean or not

• I did not unblacklist the NIC, I'm not sure how to do that. Clean shutdowns each time.

-After disabling vt-d you noticed the issues and downgraded back to v6.9.2

• Yes

Apologies if I've misunderstood anything. I'm fairly confident the cache drive wasn't really used by anything, I don't think there's any data on it that I'd be concerned about. Unless of course Docker has been using it, but from what I've noticed historically that the cache drive wasn't used, it's just a spare SSD and thought I'd try it.

 

 

Thank you,

[JorgeB]

Thanks for the answers.

 

24 minutes ago, SgtHobNob said:

I cannot be 100% certain it was 6.9.2

 

If you used the GUI to go back to the previous release it would mean you were indeed on v6.9.2

 

As for the current issue, both the docker image and appdata are on disk1, there's a chance some appdata was also on cache, delete and re-create the docker image and hopefully your dockers will be restored.

 

You can still try the cache recovery options linked above, if they don't work you'll need to format it.

 

Also note that as long as you leave vt-d disabled it should be safe to upgrade back to v6.10 after if you want.

 

 

[JorgeB]

9 hours ago, SgtHobNob said:

I did not unblacklist the NIC, I'm not sure how to do that. Clean shutdowns each time.

One thing I forgot to ask, did you by any change download the diags or even just the syslog from when you were initially running v6.10.2 with vt-d on?

[SgtHobNob]

Unfortunately no I didn't take a copy of the diagnostics, lessons learnt there I think, If I'm ever doing anything major with unraid in the future then I will take copies of the diagnostics at various points.

 

I've been trying the recovery options you suggested, these commands do mean much to me so I'll paste in the commands and responses. I tried with the array both started and not (not sure if this makes a difference), also not sure if I needed to put the 1 at the end as the drive is just labelled sdd, so I tried both.

 

root@Cookie:~# mount -o usebackuproot,ro /dev/sdd1 /x

mount: /x: wrong fs type, bad option, bad superblock on /dev/sdd1, missing codepage or helper program, or other error.

root@Cookie:~# mount -o usebackuproot,ro /dev/sdd /x

mount: /x: wrong fs type, bad option, bad superblock on /dev/sdd, missing codepage or helper program, or other error.

 

root@Cookie:~# mount -o degraded,usebackuproot,ro /dev/sdd1 /x

mount: /x: wrong fs type, bad option, bad superblock on /dev/sdd1, missing codepage or helper program, or other error.

 

Before I move onto the next stage, I just wanted to check I'd done everything right (which I'm not confident!)

 

Quote

v6.10-rc1 and newer use:

mount -o rescue=all,ro /dev/sdX1 /x

For a single device: replace X with actual device, don't forget the 1 in the end, e.g., /dev/sdf1

 

Side note, referencing the guide, for idiots like my it may be worth using something other than X as I was originally changing the "/x" instead of the "/sdX1", it's obvious now looking back, but at the time I missed it.

 

Thanks again,

 

[JorgeB]

22 minutes ago, SgtHobNob said:

Before I move onto the next stage, I just wanted to check I'd done everything right (which I'm not confident!)

Commands are correct, unfortunately and because of the error the fs is showing, I already suspected that the recovery options won't really help in this case.

 

24 minutes ago, SgtHobNob said:

Side note, referencing the guide, for idiots like my it may be worth using something other than X as I was originally changing the "/x" instead of the "/sdX1", it's obvious now looking back, but at the time I missed it.

 

Thanks, I'll change the /x to /temp to avoid confusion.