Server "stuck" every few seconds

[pappaq]

Hey, my server is currently having the issue of "getting stuck" every few seconds. Three CPU cores go up to 100%, file transfers are stuck and continue after the CPU usage return back to normal. This repeats every few seconds. This has an impact on other services as well but data transfer is impacted the most. top returns high shfs usage on one core.

 

Could someone have a look at my logs and help me? Thanks in advance!

 

 

dringenet-ms-diagnostics-20230419-1717.zip

[JorgeB]

Does stopping the VM help?

[pappaq]

I did not try that, yet. The light in the flat is controlled by it and I would like to check it, if nobody else is home. In the meantime I've observed, that the high CPU usage is on random cores, not only on the core which are pinned to the VM. May that be an indication against your idea?

 

[JorgeB]

Possibly, still worth trying.

[pappaq]

When the CPU spikes occure, the wa value peaks as well:

I'm going to try your suggestion later. Thanks in advance. IF it is the VM what would be the next step? Just to lay out a plan to keep downtimes to a minimum!

[pappaq]

Unfortunatly nothing changed after shutting down the VM.

The wa is still high and random cores are up to 100% load.

Here are my diagnostics after shutting down the VM.

 

Do you guys have an idea?

dringenet-ms-diagnostics-20230424-1652.zip

[JorgeB]

Boot in safe mode with VM and docker services disabled, then start enabling one by one to see if you find the culprit.

[pappaq]

Yeah, thought so. Gonna do that tomorrow evening. Thanks for you feedback!

[pappaq]

My problem has returned, worse than ever. The system is pretty much locked up when the iowait issue occures.

Seems like memory writeback is the issue. Is there something I can do? I've found many threads about it but never a solution for it.

 

Here some screenshots from my netdata and the diagnostics.

dringenet-ms-diagnostics-20231021-1038.zip

Edited October 21, 2023 by pappaq

[JorgeB]

If possible use disk shares (or exclusive shares) and see if that helps.

[pappaq]

You mean like this? 

 

[pappaq]

The behavior occures all the time not only when copying files and I can't pin down what it is.

[JorgeB]

I meant if the the docker system and containeres are using /mnt/user use /mnt/disk or exclusive shares when possible.

[pappaq]

Thank you. Will take some time to change everything. Will report back here.

[pappaq]

I've update unraid to 6.12.4, changed all possible shares to exclusive shares and updates everything I could. The same problem persists.

I've even changed the transcoding of emby to the RAM.

 

It seems like it is randomly doing memory writeback and that's causing the iowait to lock up the system. But I can't pin down what is causing this.

 

Any ideas?

dringenet-ms-diagnostics-20231021-1605.zip

Edited October 21, 2023 by pappaq

[pappaq]

Currently moving all data from my btrfs caches to convert them to zfs...lets see if this helps.

[JorgeB]

If it doesn't see if you can identify if it's a specific container causing that.

[pappaq]

I did start dockers one by one and it seems like it only occurred when  transcoding from emby was done. But not everytime. When zfs is not the solution I'm going to build up my emby installation from scratch.

[pappaq]

On 10/21/2023 at 7:10 PM, pappaq said:

Currently moving all data from my btrfs caches to convert them to zfs...lets see if this helps.

After moving away from btrfs for the caches there is nearly no iowait. The performance is so much better, it's absurd. Thank you JorgeB for your support!