Array has 2 failed devices after unclean shutdown

[Ulrich_W]

Hi, I need some help please

I shut down unraid as usual, made some changes boot config for fan control and started again.

After startup I got a "unclean shutdown detected" message and two drives of the array failed.

So I followed the "Troubleshooting" hint that at the end guided me to dead page linked as "checking and reparing"!?

https://docs.unraid.net/unraid-os/manual/troubleshooting/storage-management.md/#checking-a-file-system

 

After mounting the two failed disks again and reboot, one of it shows now the option to format.

The xfs array is four disks with two parity disks. One of the parity and one of the data disks failed.

But I guess it is not a good idea just to replace this two disk by new ones?

 

So I hope someone can support me, thanks in advance

 

3unetnas-diagnostics-20240402-1331.zip 3unetnas-syslog-20240402-1137.zip

[JorgeB]

The SATA controller is being passed-through to the Windows 10 Programming VM, so when the VM starts Unraid loses those disks, correct that, reboot and post new diags after array start.

[Ulrich_W]

It seems I cannot prevent the VMs from autostart. Also editing the VMs was not possible. So I disabled VMs via VM Manager and rebooted again.

Not sure if this helped?

3unetnas-diagnostics-20240402-1709.zip 3unetnas-syslog-20240402-1510.zip

[JorgeB]

Check filesystem on disk1, run it without -n, if it asks for -L use it.

[Ulrich_W]

Ok, this worked. But I am not sure how to proceed now?

disk1 is emulated and has a lost+found folder with the recovered data but with broken folders and links.

 

disk1_log.txt

[JorgeB]

If there are a lot of broken files on lost+found you can see it the actual disk mounts with UD and contents look better, note that it needs to be done with the array stopped, and after unassigning the disk.

[Ulrich_W]

Yes, content on disk1 is fully there and available without any errors.

But how can I get it back in the array again?

[JorgeB]

Instead of rebuild you can do a new config and re-sync parity - Tools - New config.

 

Any doubts ask.

[Ulrich_W]

Hello JorgeB,

new config and re-sync was fine, many thanks so far 😉

 

But enabling VMs in VM Manager went into a disaster again.  

After two times reboot still none of the array disks were available => first diagnosis files

After removing both graphics cards and rebooting, both data disks were ok, but both parity disks disabled. => second diagnosis files

 

Now I have done new config and re-sync parity running again, but I guess this will again not solve my problem!?

 

I was running the first graphics card for months now and added the second one three weeks ago without any problems.

Also rebooting several times and autostart VMs was no problem so far.

Is there a chance to correct the VM settings? 

 

3unetnas-diagnostics-20240404-1158.zip 3unetnas-syslog-20240404-1005.zip 3unetnas-diagnostics-20240404-1214.zip 3unetnas-syslog-20240404-1015.zip

[JorgeB]

13 minutes ago, Ulrich_W said:

But enabling VMs in VM Manager went into a disaster again

Well that was expect, if the passed-through controller was not corrected, @SimonFplease remind what is the current easiest way to prevent a VM from auto-starting, so he can edit the config.

 

[itimpi]

23 minutes ago, JorgeB said:

Well that was expect, if the passed-through controller was not corrected, @SimonFplease remind what is the current easiest way to prevent a VM from auto-starting, so he can edit the config.

 

I wonder if there is a good case for changing things so docker containers and/or VMs are not auto-started if you select Safe Boot.   I would think the fact you have chosen Safe Boot means you are troubleshooting so it seems a good idea to me but there may be a downside I have not thought of.

[JorgeB]

Try this, it may be the easiest way for now:

 

https://forums.unraid.net/topic/154505-unraid-crashes-when-vm-manager-services-start/?do=findComment&comment=1373412

 

[JorgeB]

1 hour ago, itimpi said:

I wonder if there is a good case for changing things so docker containers and/or VMs are not auto-started if you select Safe Boot.

I think that's a good idea if possible, don't remember if you already discussed this option.

[itimpi]

38 minutes ago, JorgeB said:

don't remember if you already discussed this option.

My memory being what is I do not either 😊

[Ulrich_W]

Ok, I could stop now VMs in safe mode and reboot without issues, so thanks for the hints.

 

But I have no graphics card in now and no PCI passthrough binding selected!

 

What to do next? Putting in my first graphics card again and rebuild a new VM?

Or do I have to fix anything else first?

 

 

[JorgeB]

The GPU likely changed IDs, this can happen if you add or remove any card, just re-select the correct GPU, instead of:

 

-device '{"driver":"vfio-pci","host":"0000:0b:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0","romfile":"/mnt/user/isos/vbios/GT730_vbios.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:0b:00.1","id":"hostdev1","bus":"pci.5","addr":"0x0"}' \

 

you need to pass-through these devices:

 

0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 730] [10de:1287] (rev a1)
    Subsystem: ASUSTeK Computer Inc. GK208B [GeForce GT 730] [1043:884a]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidia_drm, nvidia
0a:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
    Subsystem: ASUSTeK Computer Inc. GK208 HDMI/DP Audio Controller [1043:884a]
    Kernel driver in use: vfio-pci

 

so replace 0b:00.0/1 with 0a:00.0/1.

[Ulrich_W]

sorry, you are too fast for me ...

 

where can I change it, in the VM config?

As I have removed graphics cards I cannot see the config anymore, only VNC!?

 

Do I have to put back cards first?

 

[JorgeB]

Yes, the GPU must be installed, you then select it in the VM config, click the VM and then edit.

[Ulrich_W]

Ok, up and running again as before only parity-sync need to finish, thank you very much again 😉👍

 

Is there a way to prevent such disaster?

Do I need to change something in the shutdown timing?

 

Maybe unckecking the VM Autostart before changing something in boot config or the system at all is always a good idea!

[JorgeB]

10 minutes ago, Ulrich_W said:

Maybe unckecking the VM Autostart before changing something in boot config or the system at all is always a good idea!

Yep, recommend doing that.

[Ulrich_W]

Hello again

I don't know if it has something to do with my previous problem and I should open a new topic?

But something now disables autostart of the array? It does not show any notification why.

 

I finished sync, updated to 6.12.10, fixed all issues with "fix common problems" and still not working?

But I cannot figure out what else could be wrong, can anyone help please?

3unetnas-diagnostics-20240405-0850.zip 3unetnas-syslog-20240405-0650.zip

[Ulrich_W]

ok, sorry for my silly question and forget it

I forgot to check the autostart again for some reason it was disabled 🤪