Graceful and forced shutdown fail - "Unexpected EOF while looking for matching `"

[LumpyCustard]

My unraid server used to run on a 3600X/B450 combo about a month ago and I experienced an error where both graceful AND forced shutdown failed and the server would just infinitely hang on terminal. I can type into terminal and interact with Unraid but it just can't shutdown. My only option is to power off the system by holding the power button.

 

I sold off my parts and moved to a 12700/Z690 combo and I am still experiencing the same problem. When I connect to my KVM i see the following

 

Waiting up to 90 seconds for graceful shutdown...

[...]

Forcing shutdown...

Starting diagnostics collection... sh: -c: line 1: unexpected EOF while looking for matching `"

 

Diagnostics attached. I created them immediately after my server booted up.devoraid-diagnostics-20230306-2008.zip

 

Can someone point me toward what is causing shutdown to hang?

 

Thanks.

 

Edited March 6, 2023 by LumpyCustard

[JorgeB]

Do you have the same issue if you first manually shutdown the VM(s)?

[Squid]

What are the names of the VMs you're running.  Any special characters or oddities on the naming?

[LumpyCustard]

Since moving to an Intel based server and utilising QSV/NVENC i opted to retire my Server 2016 VM. I haven't done anything funky with IOMMU groups either.

 

When i WAS using a VM it was simply named "Windows Server 2016"

 

I currently have no VM's in Unraid and have disabled the VM service.

Edited March 8, 2023 by LumpyCustard

[Squid]

When running the diagnostics from the GUI, does the popup hang?  What does it show when it hangs?

[LumpyCustard]

No, when i boot my server after failing to force shutdown it SOMETIMES says improper shutdown, other times it doesn't.

 

When i generate diagnostics it completes in about 10 seconds.

 

Once the array has started though the diagnostics take a long while to complete as it seems to be going through thousands of files in my Urbackup directory.

 

Edited March 8, 2023 by LumpyCustard

[Squid]

7 minutes ago, LumpyCustard said:

as it seems to be going through thousands of files in my Urbackup directory.

Why do you think that?

[LumpyCustard]

I posted a screenshot above. I have generated diagnostics dozens of times and i've never seen it do this before. Urbackup is a docker container i use to handle backups for my personal PC and a friend of mine who backs up over the internet. I've never seen it go through tens of thousands of files to generate a diagnostic before.

 

It's been generating a diagnostic for the last 5 minutes now and it's still going.

 

edit: 15 minutes now and it's still going lol.

Edited March 8, 2023 by LumpyCustard

[LumpyCustard]

OK i had to stop the diagnostic, after 20+ minutes my browser had reached 32GB of memory usage and things were getting weird. Why is the diagnostic tool killing itself by going through my nvme cache / urbackup folder?

[Squid]

You have mover logging enabled (Settings, Schedules, Mover Settings) so everything that gets moved gets logged, and obviously a ton of lines that urbackup is logging and then mover logs it, and it all needs to get anonymized

[LumpyCustard]

I did enable mover logging some weeks back for testing purposes, it has now been disabled. This wouldn't be the cause of the shutdown issue would it?

[Squid]

Doubtful

[LumpyCustard]

Back to square one then

[JorgeB]

And do the diagnostics work if you get them on the console?