• [6.12.4/5/6] Major GUI delays after upgrade (specially dashboard, docker and apps tab)


    CiscoCoreX
    • Solved Urgent

    Hi, after upgrading from 6.12.3 to 6.12.4 my Webui has been very slow. Special on docker tab and dashboard.

    Takes almost 12 sec before the page is visible. I can see my CPU are wokring MUCH more when i try to open DOCKER tab or the Dashboard.
    Never seen a behaver like this before. I tried to stop Array, but Dashboard is still slow 12 seconds to open it, normally it tooks 2 seconds.
    Tried to remove some plugins, but still the same

    EDIT 1:
    After removing more plugins and multiple reboots. The issue is till there.

     

    EDIT 2: I decide to revert back to 6.12.3. And guess what, it working normally again, fast like it always was :)
    So can someone please explain me why I had this kind of issue with 6.12.4 ?

    EDIT 3: Tested v.6.12.5 RC 1 and the latest version released yesterday, v.6.12.5. It's still the same.
    Adding the diagnostics from my backup server (Galactica)
     

    pegasus-diagnostics-20230901-1450.zip

    galactica-diagnostics-20231128-2220.zip

    • Like 1
    • Upvote 2



    User Feedback

    Recommended Comments



    We will incorporate these changes for upcoming Unraid version 6.12.7 (and 6.13 of course).

    I have done a similar fix for VMs, which may experience the same thing.

     

    It would be nice if more users can confirm this fix works for them.

     

    Thanks everyone for participating and bringing in some useful information and help to uncover this situation.

    Though it is still a mystery to me why some people have slow write access to their flash device, something to keep an eye on.

     

    • Like 1
    • Upvote 1
    Link to comment
    1 hour ago, bonienl said:

    Though it is still a mystery to me why some people have slow write access to their flash device

    Downgrading fixed those issues. Were there big changes between .3 and .4 in the linux base system? I did not checked any diags from affecting users: is dmesg/kernel log saying something?

    Link to comment
    54 minutes ago, KluthR said:

    Were there big changes between .3 and .4 in the linux base system?

     

    In Unraid 6.12.4 we introduced a fix for macvlan. To make this work seemlessly for users the system automatically updates configuration files and other relevant settings when bridging gets enabled or disabled.

    This introduces the config files updates on the flash device.

     

    Link to comment

    Every release comes with the latest 6.1.x kernel version available at the time.

    I don't remember the exact 6.1.x kernel versions for Unraid 6.12.3 and 6.12.4.

     

    Link to comment
    1 hour ago, KluthR said:

    Downgrading fixed those issues. Were there big changes between .3 and .4 in the linux base system? I did not checked any diags from affecting users: is dmesg/kernel log saying something?

     

    The kernel version is listed in the release notes for each Unraid version: https://docs.unraid.net/unraid-os/release-notes/6.12.6/

     

    But the kernel version is not what fixed the issue when downgrading. It is that the older release didn't have this webgui code:

    9 minutes ago, bonienl said:

     

    In Unraid 6.12.4 we introduced a fix for macvlan. To make this work seemlessly for users the system automatically updates configuration files and other relevant settings when bridging gets enabled or disabled.

    This introduces the config files updates on the flash device.

     

     

    Link to comment
    15 minutes ago, ljm42 said:

    But the kernel version is not what fixed the issue when downgrading. It is that the older release didn't have this webgui code:

    The main reason is slow flash drive writes on .4+ which get back to normal at .3. Not any webgui issue at all.

     

    @CiscoCoreX correct me if Iam wrong but the „echo 1 > /boot/test“ command delays on .4 onwards and is immediately done for .3?

     

    correct me if Iam wrong, but the root cause is not fixed at all. The webgui introduced a workaround now which is perfectly fine.

    Edited by KluthR
    Link to comment

    My interpretation of https://forums.unraid.net/bug-reports/stable-releases/612456-major-gui-delays-after-upgrade-specially-dashboard-docker-and-apps-tab-r2660/page/2/?tab=comments#comment-26836 is that CiscoCoreX has two servers (one on 6.12.3 and one on 6.12.6) that both take about 6 seconds to run this command:

    time echo 1 > /boot/test.xml

     

    Not sure why they are so slow (maybe they are slow flash drives?) but as long as the webgui limits writes to the flash drive it sounds like the slowness doesn't cause noticeable issues.


    It feels like this is a hardware issue with those specific flash drives and not a kernel issue.
     

    In order to prove whether a linux kernel update caused the slowness, we would need to see the results of that command from the same flash drive on the some server under two different versions of Unraid.

     

    FWIW, on my 6.12.6 system the results are quite fast:

    # time echo 1 > /boot/test.xml
    
    real    0m0.132s
    user    0m0.000s
    sys     0m0.001s

     

    Link to comment
    14 minutes ago, ljm42 said:

    Not sure why they are so slow (maybe they are slow flash drives?)

    But its only happening since .4

     

    14 minutes ago, ljm42 said:

    as long as the webgui limits writes to the flash drive it sounds like the slowness doesn't cause noticeable issues.

    Indeed! I agree but now we know there is something. And it doenst feel 100% right because there is nothing obvious which could caused it.

     

    The initial issue could be closed, yes, but myself is interested whats the root cause. But those IO things are out of my knowledge if dmesg is not giving any obvious informations.

    Link to comment

    Indeed. Ignored that. I still remember that @CiscoCoreX mentioned clearly a normal operation after downgrading.

     

    To clarify all currently affected users should make tests (which is not doable on prod).

     

    I still making some tests with @CiscoCoreX in background. Maybe we just observe the  behavior for now. But it seemed weird that users explicitly reported this after .4. As I understood, the exec‘s were executed several times below .4 as well, right? 🤔

    Edited by KluthR
    • Like 1
    Link to comment

    Hi,

    Just to be clear, Yes I have to servers. My primary one was the first server i noticed the slowness. When I upgraded it to  v.6.12.4. Then I downgraded to v.6.12.3. Back to normal.

    Then I tested my backup server, That one was also in version 6.12.3 upgraded to 6.12.4 then the problem starts, slow pages.
    I just kept upgrading the backup server all the way to this release 6.12.6 and still the same.

    But, booth computers are using the same USB pen-drive Kingston DataTravler 3.0 - 8GB

    image.thumb.png.d9db5e8a057084817c54cc328fc65243.png


    This is from my primary server running v.6.12.3 (same USB Pen-Drive as the one in my backup server)

    image.thumb.png.bb7d183eba27a695b384b817e874d420.png

     

     

    Runnig command: time echo 1  > /boot/test.xml

    On my primary server (6.12.3)
    image.png.c92d80433596719f9a45fad782773a06.png

     

    My backup server on version 6.12.6 (now without Bonienl's fix) Slow pages again.

    image.png.0f6187e4c7534dca36647b3ff2b056ee.png

     

    And backup server with Bonienl's fix: Fast page load. Fast as in v.6.12.3
    But the write speed is the same
    image.png.fe64749da26b1a3cc68a87ebaf116d55.png

    Edited by CiscoCoreX
    Link to comment

    To add a datapoint to this thread, I decided to roll back to 6.11.5 and wait for a solution to this problem.   Running the shell commands from @CiscoCoreX here's what I see when writing to the flash drive:

     

    958297204_2023-12-1411_54_52-root@MediaTower__bash--login(MediaTower).thumb.png.17c40d17b65907eb7521c626c4179bbc.png

     

    The one thing in common is that I also have a Kingston DataTraveler 3.0 8GB pen drive.  My other unraid server is running 6.12.6 but has a Kingston DataTraveler 2.0 pen drive and does not exhibit the slow GUI problem.  I'm wondering if there is an underlying USB linux driver problem with the Kingston DataTraveler 3.0 series that is only manifesting in 6.12.6.

     

    Thoughts anyone?

    Edited by bwnautilus
    Link to comment
    33 minutes ago, bwnautilus said:

    I'm wondering if there is an underlying USB linux driver problem with the Kingston DataTraveler 3.0 series that is only manifesting in 6.12.6.

    According to what you posted the slow writes also happen with v6.11.5, so if it's a kernel issue with those flash drives it's not new.

    Link to comment

    can anyone please provide with 6.12.3 i am not able to find it anywhere and i have lots of problems with my server, i have already lost enough data trying to fix this

    Link to comment
    1 hour ago, Danuel said:

    i have already lost enough data trying to fix this

    This issue doesn't affect any data, do you have a slow flash drive? You can check by posting the output of:

     

    time echo 1 > /boot/test.xml

     

    Link to comment

    i am trying to fix this since Thursday

     

    looking into hardware thinking that might be, hardware related, reinstall data remake pools (which cause me data loss)

    now i have to stay alert that my server can go down at any time, putting all my services offline of which some people depend of it.

    i think Unraid lost its track, feels like i cant relay on this product

    is hard to troubleshoot if is a software problem or hardware

    im starting loosing trust in this

     

    wanted to try the a different USB thinking that might be the USB flash drive, but surprise key is not working, cant import, go figure, sad

     

    i have looked trough the logs  and i cant see anything out of ordinary, i doubt someone else will 

    seven-diagnostics-20231231-2110.zip

    Edited by Danuel
    Link to comment
    47 minutes ago, Danuel said:

    i am trying to fix this since Thursday

     

    looking into hardware thinking that might be, hardware related, reinstall data remake pools (which cause me data loss)

    now i have to stay alert that my server can go down at any time, putting all my services offline of which some people depend of it.

    i think Unraid lost its track, feels like i cant relay on this product

    is hard to troubleshoot if is a software problem or hardware

    im starting loosing trust in this

     

    wanted to try the a different USB thinking that might be the USB flash drive, but surprise key is not working, cant import, go figure, sad

     

    i have looked trough the logs  and i cant see anything out of ordinary, i doubt someone else will 

    seven-diagnostics-20231231-2110.zip

    Nothing of note in logs so far. Are you logging to an external syslog server also?

    Link to comment

    yes on a different share, because i did not want to stress the USB drive for that, but those are logs that appear in system log, 

     

    i thought you might see some hardware problems or something that i cant see, like i mention i am now in my third day with this and all started when i wanted to upgrade

    immich container then all go downhill

     

    but still when a crash happen does not show why, only i see forced reboot and then how is coming back online

     

    syslog-10.10.10.6.log

    Edited by Danuel
    Link to comment

    why so many networks ? is this normal ?

    i know that br0 is my ethernet port and wg0 is VPN but i don't know rest of them, anyone can help me out ?

     

    whatisthis.png

    Link to comment

    > We will incorporate these changes for upcoming Unraid version 6.12.7

     

    If you are having the issues discussed in this thread, with certain pages loading slowly due to the flash drive writing slowly, please consider testing 6.12.7-rc2 and confirming the issues are resolved.  Thanks!

     

     

    • Like 1
    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.