[SOLVED] Unraid 6.12.10 on Proxmox 8.2.2 instability - rebooting every few minutes

[gvns]

Hi

I recently deployed a new Proxmox server. I am intending to use Unraid as a VM to manage the storage. I have the VM created and booting from a USB, however, I noted today that it will randomly restart every few minutes. Despite my troubleshooting efforts I cannot pin down why.
Here is a screenshot of the task history of the VM in Proxmox:

 

Attached are my diagnostics. Is there anything in there that sticks out as problematic? Thanks in advance for any guidance.

gbyte-diagnostics-20240505-2030.zip

Edited May 10 by gvns
adding screenshot, tag

[gvns]

Thanks for approving, @trurl. 

Further to my issue here, I have noticed the reboot occurs every 5 minutes. I also noted it occurs even in Safe Mode, so I can rule out a plugin causing this.

 

My suspicion is this is virtual hardware related, though I cannot identify which specific piece of VM configuration is causing this. I have a few other VMs & LXCs that are all running stably, though they are Debian-based and not Slackware that I think Unraid is based on.

I am a bit of a loss here as to what the cause is, particularly as I have not been able to get syslog writing to my array so every time I peek at the logs I can only see the boot up logs.
 

I have the server stopped for now, to not waste power/drive life. Though, I am determined to get this to work!

Edited May 6 by Gareth Evans

[gvns]

I am highly tempted to bail on the virtualized setup and go back to Unraid on bare metal here. Though any insight on troubleshooting before I do that would be highly appreciated!

[BRiT]

18 hours ago, gvns said:

Further to my issue here, I have noticed the reboot occurs every 5 minutes. I also noted it occurs even in Safe Mode, so I can rule out a plugin causing this.

 

Sounds like WatchDog timers.

 

Is it only the unraid VM which reboots or is it the entire host, including ProxMox rebooting?

[gvns]

3 hours ago, BRiT said:

 

Sounds like WatchDog timers.

 

Is it only the unraid VM which reboots or is it the entire host, including ProxMox rebooting?

The Proxmox host and other VMs/LXCs remain stable.

 

I am not familiar with watchdog timers, what are those? A brief Google gets me "A watchdog timer is a component of a computer system that helps it recover from malfunctions. During normal operation, software will periodically reset the watchdog timer to prevent it from elapsing, or "timing out". If, due to a software or hardware problem, the watchdog timer is not reset, it will elapse and generate a system reset, or in some cases a system interrupt. This mechanism ensures that if your program or system hangs, it will automatically reboot after a certain amount of time."

 

... sounds pretty darn likely! How would I dig further on this? 
 

[tjb_altf4]

Can you post a screenshot of the proxmox hardware config page for the VM?

 

What kind of hardware is Proxmox running on?

 

 

[gvns]

@tjb_altf4certainly - thanks for lending a hand.

 

This is what I have configured in proxmox for my unraid machine:

 

I just booted it back up, and it reboots almost exactly 5 minutes after first boot. That certainly aligns with the theory it is a preset timer, like a watchdog timer, being tripped.

Edited May 7 by gvns

[tjb_altf4]

1 hour ago, gvns said:

@tjb_altf4certainly - thanks for lending a hand.

 

This is what I have configured in proxmox for my unraid machine:
 

I just booted it back up, and it reboots almost exactly 5 minutes after first boot. That certainly aligns with the theory it is a preset timer, like a watchdog timer, being tripped.

Try changing "Processors" to the following

[gvns]

1 hour ago, tjb_altf4 said:

Try changing "Processors" to the following

Gave this a shot, unfortunately I see the same behaviour. These are my current hardware settings:

 

We can see it reboot 5 minutes & 19 seconds after startup:

 

Here also are my VM Options:

 

 

I also just booted Unraid back up on bare metal without Proxmox in the mix and it ran perfectly stable. That certainly points to my Proxmox config being the problem, though I am not sure what about it. These are all default settings.

 

I have just jumped back into Unraid on Proxmox - and it reboots after exactly 5 minutes and 19 seconds again. I am not sure what there is specifically about 319 seconds, but apparently it is my lucky/unlucky number.

 

I am going to shut it down for the night, sleep on it and decide what to do tomorrow. My Unraid trial license ends tomorrow so that puts a definitive timeline on things. Right now, I am leaning towards dropping Proxmox and using Unraid as my hypervisor. I had initially favoured Proxmox for the additional functionality around snapshotting etc though I think if I have to choose between the Unraid array or Proxmox VM functionality, I would go with Unraid.

 

[tjb_altf4]

Apologies you can leave cpu type as "host", but otherwise rules out that being the issue.

 

Most of your settings are the same as mine... only other suggestion is to ensure "Ballooning Device" is unchecked in Proxmox vm memory options, possibly pump this up to 6 or 8GB if you have the spare memory especially if you are running more than pure storage on it e.g. docker.

 

5 min is not very long, are you actually getting into Unraid at any point?

Does the console show the boot process finishing?

 

 

Edited May 7 by tjb_altf4

[gvns]

7 hours ago, tjb_altf4 said:

Apologies you can leave cpu type as "host", but otherwise rules out that being the issue.

 

Most of your settings are the same as mine... only other suggestion is to ensure "Ballooning Device" is unchecked in Proxmox vm memory options, possibly pump this up to 6 or 8GB if you have the spare memory especially if you are running more than pure storage on it e.g. docker.

 

5 min is not very long, are you actually getting into Unraid at any point?

Does the console show the boot process finishing?

 

 

Yup, I get into Unraid - it typically boots in about 2 minutes. I have it set to auto start on Proxmox startup.

 

I can absolutely bump it up to 8 GB of RAM, I have 32 on the machine - I was keeping it in reserve for VMs on the Proxmox hypervisor.

[gvns]

I bumped up my Unraid VM to 8 GB of RAM, no change. I also noted that the Ballooning Device was enabled, I disabled with no change in behaviour - it still reboots consistently.

 

What is odd is that I noted that the 5:19 countdown timer seems to start from first boot. So, if I boot up and then manually reset and start back up, I will still see a reboot at exactly 5:19 after that initial first boot. That reinforces to me that it is something to do with Proxmox that seems to interact poorly with Unraid in a VM. 

 

Have not yet made a decision on my final build though I think I have exhausted my troubleshooting approach here, unless I figure out a better logging solution and find some good clues. 

[tjb_altf4]

I did find this interesting thread on Proxmox forums, does this sound applicable to your setup?

Quote

It's often because of third-party monitoring software that is installed (and forgotten) by the user, that does not register the running state of the VMs and restarts them. I kid you not; this happens regularly on this forum.

https://forum.proxmox.com/threads/vm-stop-starts-every-5-minutes.136678/

[gvns]

@tjb_altf4 you solved it!!!

 

I feel like an idiot - it was indeed monitor-all that was causing this problem. Evidently, because I do not have a qemu agent running, Monitor-all assumed this was a dead machine and power cycled it. After removing it (via the same install script and using the remove menu option), my Unraid VM is now running stably.

 

I had done some brief googling and did not find this as a result. Well done on your searching, and I really appreciate your going the extra mile after I had essentially resigned to giving up on Proxmox. I feel especially silly knowing I brought this problem on myself! 

 

A fair warning to others, do not deploy scripts on your environment without understanding the impact.

 

Thanks again Reptar @tjb_altf4, you have been extremely helpful. I owe you one.

[tjb_altf4]

6 hours ago, gvns said:

@tjb_altf4 you solved it!!!

 

I feel like an idiot - it was indeed monitor-all that was causing this problem. Evidently, because I do not have a qemu agent running, Monitor-all assumed this was a dead machine and power cycled it. After removing it (via the same install script and using the remove menu option), my Unraid VM is now running stably.

 

I had done some brief googling and did not find this as a result. Well done on your searching, and I really appreciate your going the extra mile after I had essentially resigned to giving up on Proxmox. I feel especially silly knowing I brought this problem on myself! 

 

A fair warning to others, do not deploy scripts on your environment without understanding the impact.

 

Thanks again Reptar @tjb_altf4, you have been extremely helpful. I owe you one.

Great to hear!
Also if you need qemu agent, it is already installed on unraid, but you need to activate it like so:

 

[gvns]

4 hours ago, tjb_altf4 said:

Great to hear!
Also if you need qemu agent, it is already installed on unraid, but you need to activate it like so:

 

 

I had actually tried this prior to removing Monitor-All, without success. I do not think I specifically need Monitor-All, so I will just go without it until I see the need. I think I am going to use Uptime Kuma instead.