Massive performance issues since 6.10-RC3

[D.Romeleitis]

Hello,

 

on my Unraid server everything was working fine when I used version 6.10-RC2 or lower.

Since RC3 the whole GUI is running slow. First I thought it will be fixed with next RC or the stable.

Today I gave Version 6.10.2 a try. But still the same. The GUI is very slow. Docker containers starts only partially and slow.

When I downgrade to 6.10-RC2 everything is back at the old speed. Any idea? What was changed?

 

Hardware : 

Gigabyte W480M VISION W

11th Gen Intel® Core™ I5-11600 @ 2.80GHz

32 GB RAM

Cache : WD Black SN750 NVMe SSD

Parity : Seagate EXOS 18TB

Data : 1 x Seagate EXOS 18TB, 5 x Seagate Ironwolf 8 TB

 

Installed Plugins :

CA Backup / Restore Appdata
CA Config Editor
Community Applications
disklocation
Docker Folder
Dynamix Active Streams
Dynamix Auto Fan Control
Dynamix Cache Directories
Dynamix Date Time
Dynamix Local Master
Dynamix S3 Sleep
Dynamix Schedules
Dynamix SSD TRIM
Dynamix System Information
Dynamix System Statistics
Dynamix System Temperature
File Activity
GPU Statistics
GUI Search
Intel GPU TOP
Intel GVT-g
Libvirt Hotplug USB
My Servers
Nerd Tools
Open Files
Parity Check Tuning
Theme Engine
Tips and Tweaks
Unassigned Devices
Unassigned Devices Plus
Unassigned Devices Preclear
unBALANCE
unraid.usbip
USBIP Gui
User Scripts
VM Backup
 

Docker container with autostart :

AdGuard-Home
diskover
Docker-WebUI
duplicati
EmbyServer
JDownloader2
paperless-ng
PhotoPrism
Photoview
elasticsearch
mariadb
mysql
redis

 

Kind regards,

 

Daniel

 

Edited May 29, 2022 by D.Romeleitis

[JorgeB]

Please post the diagnostics with v6.10.2, hopefully they will give some clues.

[D.Romeleitis]

Diagnostics is running since 10 minutes. Still waiting.

During that time I found another issue. Tried to open console to get sytem information. But console window has the error : 502 Bad Gateway

[D.Romeleitis]

20 Minutes now. Still :

 

From all dockers only AdGuard-Home seems to be started. All others are still stopped.

The server is started 50 minutes after the update to Version 6.10.2.

 

Processor usage is higher than 6.10-rc2. There I had under 10%.

Now 2 cores are continuesly at 100%:

 

Edited May 29, 2022 by D.Romeleitis

[D.Romeleitis]

One hour later the downloading diagnostics popup still haven't changed.

Worst thing : I can't even access my shares from other computers... That is really terrible...

[JorgeB]

If the diags don't download after a couple of minutes they won't, try getting at least the syslog for now:

 

cp /var/log/syslog /boot/syslog.txt

 

[D.Romeleitis]

I wasn't able to start terminal because of "502 Bad Gateway"

SSH on Windows Commandline is giving only "connect to host 192.168.0.5 port 22: Connection refused.

Now I get "500 Internal Server Error" on GUI.

I will now try to restart Unraid without plugins.

[D.Romeleitis]

Before the restart I was able to copy syslog.txt directly on servers terminal.

Now I restarted in Safe mode. Even there I get a "502 Bad Gateway" when I try to start a terminal window in the Web UI.

I started the array. GUI is more performant than before. The docker tab is still very slow showing the containers...

I have this window since minutes :

syslog.txt

 

I tried to get the diagnostic information, but still the same window like before...

Edited May 28, 2022 by D.Romeleitis

[D.Romeleitis]

Does anybody have an idea? The system is useless at the moment. Dockers are not starting. GUI is completely unstable/not performant/not reachable. The Array is not reachable. I only see the option to go back to 9.10-RC2, but then can't ever update the system anymore... 

[JorgeB]

There are some NVMe related errors, if you can can move it to a different PCIe/M.2 slot to see if they go away, but main issue are likely all the nginx errors, hopefully someone else will help with those, for now did you try booting in safe mode just to rule out any plugin issues?

[D.Romeleitis]

Result of top command...

[D.Romeleitis]

55 minutes ago, JorgeB said:

There are some NVMe related errors, if you can can move it to a different PCIe/M.2 slot to see if they go away, but main issue are likely all the nginx errors, hopefully someone else will help with those, for now did you try booting in safe mode just to rule out any plugin issues?

System is running in safe mode since yesterday. GUI is responding better than in normal mode, but e.g. on Tools -> Update OS the Status of versions remains in "checking". Docker tab is not showing anything. Dockers in main page shows that all dockers are stopped. Array is started, but not reachable over network.

 

I don't think that there is an issue with the NVMe. It is running normal with version 9.10-RC2 and below. 

Edited May 29, 2022 by D.Romeleitis

[JorgeB]

27 minutes ago, D.Romeleitis said:

I don't think that there is an issue with the NVMe.

It's not a problem with the NVMe, but it's generating correctable PCIe errors, unlikely they are related to the issues but you should still to get rid of them, if it's not possible at least suppress the errors to stop spamming the log.

 

[D.Romeleitis]

Because of rumors in the family I went back to 6.10-RC2. Everything is running smooth as before. Dockers are up. GUI is running smooth.

Please try to find a solution. As it is now I can't ever upgrade to newer versions...

Something must have change from 6.10-RC2 to 6.10-RC3.

I reported it : https://forums.unraid.net/bug-reports/prereleases/unraid-os-version-6100-rc3-available-r1760/page/7/?tab=comments#comment-17701

 

Attached the diagnostics (which also are no problem on that version): unraid-server-diagnostics-20220529-1438.zip

Edited May 29, 2022 by D.Romeleitis

[JorgeB]

I would suggest creating a new bug report in the stable releases section, and don't forget the diagnostics.

[D.Romeleitis]

14 minutes ago, JorgeB said:

I would suggest creating a new bug report in the stable releases section, and don't forget the diagnostics.

Opened one under https://forums.unraid.net/bug-reports/stable-releases/6102-massive-performance-issues-no-array-access-no-dockers-r1961/