[6.12.4/5/6] Major GUI delays after upgrade (specially dashboard, docker and apps tab)

We will incorporate these changes for upcoming Unraid version 6.12.7 (and 6.13 of course).

I have done a similar fix for VMs, which may experience the same thing.

 

It would be nice if more users can confirm this fix works for them.

 

Thanks everyone for participating and bringing in some useful information and help to uncover this situation.

Though it is still a mystery to me why some people have slow write access to their flash device, something to keep an eye on.

 

1 hour ago, bonienl said:

Though it is still a mystery to me why some people have slow write access to their flash device

Downgrading fixed those issues. Were there big changes between .3 and .4 in the linux base system? I did not checked any diags from affecting users: is dmesg/kernel log saying something?

54 minutes ago, KluthR said:

Were there big changes between .3 and .4 in the linux base system?

 

In Unraid 6.12.4 we introduced a fix for macvlan. To make this work seemlessly for users the system automatically updates configuration files and other relevant settings when bridging gets enabled or disabled.

This introduces the config files updates on the flash device.

 

I meant kernel side.

Every release comes with the latest 6.1.x kernel version available at the time.

I don't remember the exact 6.1.x kernel versions for Unraid 6.12.3 and 6.12.4.

 

1 hour ago, KluthR said:

Downgrading fixed those issues. Were there big changes between .3 and .4 in the linux base system? I did not checked any diags from affecting users: is dmesg/kernel log saying something?

 

The kernel version is listed in the release notes for each Unraid version: https://docs.unraid.net/unraid-os/release-notes/6.12.6/

 

But the kernel version is not what fixed the issue when downgrading. It is that the older release didn't have this webgui code:

9 minutes ago, bonienl said:

 

In Unraid 6.12.4 we introduced a fix for macvlan. To make this work seemlessly for users the system automatically updates configuration files and other relevant settings when bridging gets enabled or disabled.

This introduces the config files updates on the flash device.

 

 

15 minutes ago, ljm42 said:

But the kernel version is not what fixed the issue when downgrading. It is that the older release didn't have this webgui code:

The main reason is slow flash drive writes on .4+ which get back to normal at .3. Not any webgui issue at all.

 

@CiscoCoreX correct me if Iam wrong but the „echo 1 > /boot/test“ command delays on .4 onwards and is immediately done for .3?

 

correct me if Iam wrong, but the root cause is not fixed at all. The webgui introduced a workaround now which is perfectly fine.

Edited December 13, 2023 by KluthR

My interpretation of https://forums.unraid.net/bug-reports/stable-releases/612456-major-gui-delays-after-upgrade-specially-dashboard-docker-and-apps-tab-r2660/page/2/?tab=comments#comment-26836 is that CiscoCoreX has two servers (one on 6.12.3 and one on 6.12.6) that both take about 6 seconds to run this command:

time echo 1 > /boot/test.xml

 

Not sure why they are so slow (maybe they are slow flash drives?) but as long as the webgui limits writes to the flash drive it sounds like the slowness doesn't cause noticeable issues.

It feels like this is a hardware issue with those specific flash drives and not a kernel issue.
 

In order to prove whether a linux kernel update caused the slowness, we would need to see the results of that command from the same flash drive on the some server under two different versions of Unraid.

 

FWIW, on my 6.12.6 system the results are quite fast:

# time echo 1 > /boot/test.xml

real    0m0.132s
user    0m0.000s
sys     0m0.001s

 

14 minutes ago, ljm42 said:

Not sure why they are so slow (maybe they are slow flash drives?)

But its only happening since .4

 

14 minutes ago, ljm42 said:

as long as the webgui limits writes to the flash drive it sounds like the slowness doesn't cause noticeable issues.

Indeed! I agree but now we know there is something. And it doenst feel 100% right because there is nothing obvious which could caused it.

 

The initial issue could be closed, yes, but myself is interested whats the root cause. But those IO things are out of my knowledge if dmesg is not giving any obvious informations.

Just now, KluthR said:

But its only happening since .4

 

Where are you seeing that?  in https://forums.unraid.net/bug-reports/stable-releases/612456-major-gui-delays-after-upgrade-specially-dashboard-docker-and-apps-tab-r2660/page/2/?tab=comments#comment-26836 , they say running that command is slow on their 6.12.3 server as well

Indeed. Ignored that. I still remember that @CiscoCoreX mentioned clearly a normal operation after downgrading.

 

To clarify all currently affected users should make tests (which is not doable on prod).

 

I still making some tests with @CiscoCoreX in background. Maybe we just observe the  behavior for now. But it seemed weird that users explicitly reported this after .4. As I understood, the exec‘s were executed several times below .4 as well, right? 🤔

Edited December 13, 2023 by KluthR

3 minutes ago, KluthR said:

As I understood, the exec‘s were executed several times below .4 as well, right?

 

Bonienl said the exec sed commands were added in 6.12.4 to address the macvlan issues: https://forums.unraid.net/bug-reports/stable-releases/612456-major-gui-delays-after-upgrade-specially-dashboard-docker-and-apps-tab-r2660/page/3/?tab=comments#comment-26861 
 

Yes. You are right. Mixed that up.

 

Hi,

Just to be clear, Yes I have to servers. My primary one was the first server i noticed the slowness. When I upgraded it to  v.6.12.4. Then I downgraded to v.6.12.3. Back to normal.

Then I tested my backup server, That one was also in version 6.12.3 upgraded to 6.12.4 then the problem starts, slow pages.
I just kept upgrading the backup server all the way to this release 6.12.6 and still the same.

But, booth computers are using the same USB pen-drive Kingston DataTravler 3.0 - 8GB

This is from my primary server running v.6.12.3 (same USB Pen-Drive as the one in my backup server)

 

 

Runnig command: time echo 1  > /boot/test.xml

On my primary server (6.12.3)

 

My backup server on version 6.12.6 (now without Bonienl's fix) Slow pages again.

 

And backup server with Bonienl's fix: Fast page load. Fast as in v.6.12.3
But the write speed is the same

Edited December 13, 2023 by CiscoCoreX

To add a datapoint to this thread, I decided to roll back to 6.11.5 and wait for a solution to this problem.   Running the shell commands from @CiscoCoreX here's what I see when writing to the flash drive:

 

 

The one thing in common is that I also have a Kingston DataTraveler 3.0 8GB pen drive.  My other unraid server is running 6.12.6 but has a Kingston DataTraveler 2.0 pen drive and does not exhibit the slow GUI problem.  I'm wondering if there is an underlying USB linux driver problem with the Kingston DataTraveler 3.0 series that is only manifesting in 6.12.6.

 

Thoughts anyone?

Edited December 14, 2023 by bwnautilus

33 minutes ago, bwnautilus said:

I'm wondering if there is an underlying USB linux driver problem with the Kingston DataTraveler 3.0 series that is only manifesting in 6.12.6.

According to what you posted the slow writes also happen with v6.11.5, so if it's a kernel issue with those flash drives it's not new.

can anyone please provide with 6.12.3 i am not able to find it anywhere and i have lots of problems with my server, i have already lost enough data trying to fix this

1 hour ago, Danuel said:

i have already lost enough data trying to fix this

This issue doesn't affect any data, do you have a slow flash drive? You can check by posting the output of:

 

time echo 1 > /boot/test.xml

 

up to 1 min in response, this eventually will crash my server

4 minutes ago, Danuel said:

up to 1 min in response, this eventually will crash my server

Please can you include diagnostics for your post

i am trying to fix this since Thursday

 

looking into hardware thinking that might be, hardware related, reinstall data remake pools (which cause me data loss)

now i have to stay alert that my server can go down at any time, putting all my services offline of which some people depend of it.

i think Unraid lost its track, feels like i cant relay on this product

is hard to troubleshoot if is a software problem or hardware

im starting loosing trust in this

 

wanted to try the a different USB thinking that might be the USB flash drive, but surprise key is not working, cant import, go figure, sad

 

i have looked trough the logs  and i cant see anything out of ordinary, i doubt someone else will 

seven-diagnostics-20231231-2110.zip

Edited December 31, 2023 by Danuel

47 minutes ago, Danuel said:

i am trying to fix this since Thursday

 

looking into hardware thinking that might be, hardware related, reinstall data remake pools (which cause me data loss)

now i have to stay alert that my server can go down at any time, putting all my services offline of which some people depend of it.

i think Unraid lost its track, feels like i cant relay on this product

is hard to troubleshoot if is a software problem or hardware

im starting loosing trust in this

 

wanted to try the a different USB thinking that might be the USB flash drive, but surprise key is not working, cant import, go figure, sad

 

i have looked trough the logs  and i cant see anything out of ordinary, i doubt someone else will 

seven-diagnostics-20231231-2110.zip

Nothing of note in logs so far. Are you logging to an external syslog server also?

yes on a different share, because i did not want to stress the USB drive for that, but those are logs that appear in system log, 

 

i thought you might see some hardware problems or something that i cant see, like i mention i am now in my third day with this and all started when i wanted to upgrade

immich container then all go downhill

 

but still when a crash happen does not show why, only i see forced reboot and then how is coming back online

 

syslog-10.10.10.6.log

Edited December 31, 2023 by Danuel

why so many networks ? is this normal ?

i know that br0 is my ethernet port and wg0 is VPN but i don't know rest of them, anyone can help me out ?

 

> We will incorporate these changes for upcoming Unraid version 6.12.7

 

If you are having the issues discussed in this thread, with certain pages loading slowly due to the flash drive writing slowly, please consider testing 6.12.7-rc2 and confirming the issues are resolved.  Thanks!