[6.12.8] No WebGUI - /run full

Post diagnostics

 

https://forums.unraid.net/bug-reports/stable-releases/report-guidelines-r68/

 

I added to the first post

You have a large number of plugins, and for some reason checking plugin version is blocked or something, so can't easily tell if any are out-of-date. Updating all plugins was mentioned in the release notes.

 

In fact, booting in SAFE mode is mentioned in the Report Guidelines I linked above.

 

And, you have a lot of stuff in the extra folder on your flash drive, probably put there by NerdTools.

 

Not sure if extra folder will be loaded if you boot in SAFE mode or not.

Instead of using your router as your DNS servers, setup DNS #1 and #2 with a dedicated DNS server.  Use 208.67.222.222 and 208.67.220.220 (OpenDNS servers) 

  

29 minutes ago, trurl said:

You have a large number of plugins, and for some reason checking plugin version is blocked or something, so can't easily tell if any are out-of-date. Updating all plugins was mentioned in the release notes.

 

In fact, booting in SAFE mode is mentioned in the Report Guidelines I linked above.

 

And, you have a lot of stuff in the extra folder on your flash drive, probably put there by NerdTools.

 

Not sure if extra folder will be loaded if you boot in SAFE mode or not.

 

OK, first of all, thank you for the analysis.
But I would like to mention again that I cannot install the current update either. That means I flash the USB stick with version 6.12.8 and then don't do anything (no restore settings, just the delivery state) and get the same problem.
So it can't be from a plugin or extra etc., but I can re-run diagnostics with this status after the actual backup..

Edited February 18 by Amane

24 minutes ago, Squid said:

Instead of using your router as your DNS servers, setup DNS #1 and #2 with a dedicated DNS server.  Use 208.67.222.222 and 208.67.220.220 (OpenDNS servers) 

Thanks, I have internet now, even after reboots. The DNS works, but the web-server/webUI/nginx whatever was not started. Sorry for the confusion in my description

So everything works though if you boot in one of the NON GUI modes and you access via another computer? (this is how 99.9% of the times you would manage the system.  Fundamentally, GUI mode is there if your daily driver you use is a VM on the Unraid Server itself.

 

Curious if when booting via the non-gui mode after you log in at the prompt and then enter in the command 

slim

if the GUI comes up.

12 minutes ago, Squid said:

So everything works though if you boot in one of the NON GUI modes and you access via another computer? (this is how 99.9% of the times you would manage the system.  Fundamentally, GUI mode is there if your daily driver you use is a VM on the Unraid Server itself.

 

Curious if when booting via the non-gui mode after you log in at the prompt and then enter in the command 

slim

if the GUI comes up.

No, no, that would be nice, nothing works, that's why I'm in the GUI. ^^ When I start normally, I cannot access Unraid because the web server is not running. I would just use the CLI, but unfortunately then I have an English keyboard layout. But that's another topic

Edited February 18 by Amane

Seems to be related to this issue

https://forums.unraid.net/topic/154069-services-not-starting-after-61218-upgrade-pid-files-cannot-be-created/

To combat any kind of confusion, I have the Dignostics here again.

 

What have i do for it:
I installed Unraid using USB Creator, connected it to the server and started it (no restore any config).

No web server wants to start. Internet works, i can go to any page but no unraid webGUI.

The new update doesn't want to work on my server.... ☹️

unraid-diagnostics-20240217-1946.zip

8 hours ago, Mainfrezzer said:

Seems to be related to this issue

https://forums.unraid.net/topic/154069-services-not-starting-after-61218-upgrade-pid-files-cannot-be-created/

Yes, my /run/ is 100% full, that's not normal right? I use a Intenso-USB with 8GB.

 

Filesystem    Size    Used    Avail    Use%    Mounted on

tmpfs           32M     32M         0    100%    /run

 

i will made a unraid on a another stick (32G) but the Tool wan't finish the filesystem syncing process.. (yes i have clean the stick)

Spoiler

 

Edited February 18 by Amane

the /run/udev/data fills up:
run_udev_data.log

This may be a bug, though for now it's only affecting a couple of users.

I have a workaround for my, my System is now running!

 

Please fix this bug in a future version.. I think it affects users who own many devices. In the log you can see that every core, slot etc. makes an entry.

My system with 128 threads, 1TB RAM (8 sticks), 3 graphics cards and many hard drives in the raid etc. produce a lot of output: (33MB or more)

 

If you want to check whether this is the problem, you can use "df -h" to show you the size and load of the file system.

Quote

root@unraid:~# df -h

Filesystem    Size    Used    Avail    Use%    Mounted on

tmpfs           32M     32M         0    100%    /run

 

The workaround for this error is a entry in the /boot/config/go file:

It resizes the mount of /run on startup, restarts dbus and ensures that rc.elogind is started

#increase /run size
/bin/mount -t tmpfs tmpfs /run -o remount,size=128M

#start dbus
/usr/bin/dbus-uuidgen --ensure
/usr/bin/dbus-daemon --system

#start elogind
/etc/rc.d/rc.elogind start

 

Reboot and check the file system again. Nginx (and the system) should now boot normally

Quote

root@unraid:~# df -h

Filesystem    Size    Used    Avail    Use%    Mounted on

tmpfs          128M     33M     96M     26%    /run

 

 

*(I update the workaround) I found this post after my workarount:

 

Edited February 28 by Amane
* Check the Post and thread from the link

3 hours ago, Amane said:

the /run/udev/data fills up:
run_udev_data.log

I think the abnormal allocation under /run/udev/data is affecting everyone, it just doesn't fill the 32MB for people with lower core counts and lower pci-e lanes + associated devices.

Please fix the allocation issue, must be a bug introduced with 6.12.8 as it wasn't happening in 6.12.6, or give us a way to increase the size of /run that is not a line in the go file as there are services failing before that runs and it's messy and potentially unstable to fix it that way.

Atm only 2 users reported it, but i bet since the release was on Friday, many high core count and pci-e lanes users did not upgrade yet, for sure there are many that I know of on Discord with similar systems which are gonna have the same issue.

TLDR: if you're one of those users, do not upgrade yet

Edited February 18 by bubbl3

@Amaneand @bubbl3can you confirm if the /run used size keeps growing or it's stable after start?

 

 

9 minutes ago, JorgeB said:

@Amaneand @bubbl3can you confirm if the /run used size keeps growing or it's stable after start?

Looks stable at around 34MB, it only increases (few KB, nothing much) if i plug in a new device, like usb storage, etc. or if I run more docker containers.

Edited February 18 by bubbl3

1 hour ago, JorgeB said:

@Amaneand @bubbl3can you confirm if the /run used size keeps growing or it's stable after start?

 

 

For me too it leveled off at around 33 MB. I also manually performed "/etc/rc.d/rc.elogind restart" without anything happening. I will monitor this and report back after a while.
I don't think it will increase dramatically.

Wanted to post this yestherday but was having issue merging my account

 

How many core and thread do you have on your machine?

 

Have been cratching my head on a new install also with the same issue.  CHanged network card, reflashing multiple USD stick with the flasher and manually to no avail.  on shutting down was seing this error message about NGINX not running.  This morning was thinking it could be a gui thing crashingand stated deativating core.  I have a 7742 with 64 core and 128 thread deativating SMT in BIOS made things work, so could try that.  I was actulally in the forum to see if someone else had seen similar issues.

Edited February 18 by AppleJon

@AppleJon disabling SMT indeed solved the issue, but seems to be a mitigation as 75% usage of /run is still abnormal:

 

/run/udev/data still taking way too much space:

 

Now wondering why other Threadripper/EPYC users with similar core counts are not affected.

 

The /run usage appears to have increased by a lot from 6.12.6 to v6.12.8, for example for my 4 core test server, it went from 2% to 17% (of the default 32M), unclear for now if this is a kernel change or some other introduced bug, LT is looking into it.

 

Most/all users will have the increased usage but it won't cause any issues since 32M is still enough, I think for now only users with 64 core CPUs (128 threads) or more will be affected, those should for now just increase /run size until LT fixes the issue, either by increasing the size by default, or better yet, try to find out what is causing the >10x increase in used size.

15 minutes ago, JorgeB said:

The /run usage appears to have increased by a lot from 6.12.6 to v6.12.8, for example for my 4 core test server, it went from 2% to 17% (of the default 32M), unclear for now if this is a kernel change or some other introduced bug, LT is looking into it.

 

Most/all users will have the increased usage but it won't cause any issues since 32M is still enough, I think for now only users with 64 core CPUs (128 threads) or more will be affected, those should for now just increase /run size until LT fixes the issue, either by increasing the size by default, or better yet, try to find out what is causing the >10x increase in used size.

 

The puzzling thing is that according to the conversation we were having yesterday this is not affecting everyone with 64 cores / 128 threads CPUs, which makes me think the high core count is just one of the factors.

Edited February 19 by bubbl3

3 minutes ago, bubbl3 said:

The puzzling thing is that according to the conversation we were having yesterday this is not affecting everyone with 64 cores / 128 threads CPUs

That is strange, and the user had SMT enabled for sure? It would be good to see the diags to confirm.

 

All 3 affected cases I've sound so far have a 128 thread CPU, RAM appears to not play a part, since I've seen 1TB and 128GB, the remaining devices may, but the thread count still seems the main issue.

8 minutes ago, JorgeB said:

That is strange, and the user had SMT enabled for sure? It would be good to see the diags to confirm.

 

All 3 affected cases I've sound so far have a 128 thread CPU, RAM appears to not play a part, since I've seen 1TB and 128GB, the remaining devices may, but the thread count still seems the main issue.

That's what I'm wondering, will ask Synd to share his diagnostics later when he's online.

Can someone with a 64+ core system the output from:

grep -c ^processor /proc/cpuinfo