Upgraded two servers to latest release and no longer have WebGui access - page will not load.

[DieFalse]

I have SSH and direct server connected GUI / Terminal access.  The web pages will not load on either.  The only change was upgrade to the latest stable release.

 

I am beggining this ticket to see if its a known issue I may have missed in my search or if any generic steps are available - IE: how to check the default host webserver status etc?

 

Diagnostics pending

Edited April 26, 2021 by fmp4m

[ljm42]

Are you able to get the diagnostics?

 

The first thing I would check is to confirm that the files were stored correctly on the flash. From SSH or a console window run these commands:

 

head -n 1 /boot/changes.txt
sha256sum /boot/bz*

 

For 6.9.2, the output should match this:

 

root@Tower:~# head -n 1 /boot/changes.txt
## Version 6.9.2 2021-04-07
root@Tower:/boot# sha256sum /boot/bz*
7216239d48d9f276c65fd1bce5c80d513beadde63f125bbb48b97228f4e3db1c  /boot/bzfirmware
debc904556b518fc6ea2bf7c679b86d8b99ad978b321fad361c25d829ecb7460  /boot/bzfirmware.sha256
1a7dd82250acf93b711633bbf854cc90a03465bb32c3cec4d56a0355cfc10096  /boot/bzimage
b9098fd8dc1f1e3fa594a54864a1e0ede7c2d41d750564e8168b2ab406c3ec3f  /boot/bzimage.sha256
75be3470b4536272062f4673ef21726da1d54b7bde5e264254e5df77c87c40a0  /boot/bzmodules
9de395254b24ddb1c52c2d9f22e613567ef61659dab837777f41c25ae0bafa5b  /boot/bzmodules.sha256
7692d002882cc96760d5f1a98b23e4c8872f6b8d2233bfcdec7e6331802b0cf1  /boot/bzroot
9fa3228cebfdd48eb5d78f44a1272231e9d1e0944b54e08c18f2aa315b8e148f  /boot/bzroot-gui
52f7f3e9118f8b96db00ea8cbe795baf48bddb6ed2be08cf54af81e66ff17ab6  /boot/bzroot-gui.sha256
12ce4274dcb3f3422c1e0f9fcc37bc3f0aef9c834a19c25da06a21c2ce52303f  /boot/bzroot.sha256

 

 

[DieFalse]

Thanks;

 

I am having trouble pulling the diagnostics even with SCP.   I will keep trying and once I can gain physical access again tonight will copy them manually also.

 

CURL IP of host results in a time out also.

 

root@GSA:/mnt/user/www# sha256sum /boot/bz*
7216239d48d9f276c65fd1bce5c80d513beadde63f125bbb48b97228f4e3db1c  /boot/bzfirmware
debc904556b518fc6ea2bf7c679b86d8b99ad978b321fad361c25d829ecb7460  /boot/bzfirmware.sha256
1a7dd82250acf93b711633bbf854cc90a03465bb32c3cec4d56a0355cfc10096  /boot/bzimage
b9098fd8dc1f1e3fa594a54864a1e0ede7c2d41d750564e8168b2ab406c3ec3f  /boot/bzimage.sha256
75be3470b4536272062f4673ef21726da1d54b7bde5e264254e5df77c87c40a0  /boot/bzmodules
9de395254b24ddb1c52c2d9f22e613567ef61659dab837777f41c25ae0bafa5b  /boot/bzmodules.sha256
7692d002882cc96760d5f1a98b23e4c8872f6b8d2233bfcdec7e6331802b0cf1  /boot/bzroot
9fa3228cebfdd48eb5d78f44a1272231e9d1e0944b54e08c18f2aa315b8e148f  /boot/bzroot-gui
52f7f3e9118f8b96db00ea8cbe795baf48bddb6ed2be08cf54af81e66ff17ab6  /boot/bzroot-gui.sha256
12ce4274dcb3f3422c1e0f9fcc37bc3f0aef9c834a19c25da06a21c2ce52303f  /boot/bzroot.sha256
root@GSA:/mnt/user/www# head -n 1 /boot/changes.txt                                                                                ## Version 6.9.2 2021-04-07

 

[ljm42]

The sha256 of the bz files looks good, so that should rule out issues related to invalid files on the flash drive.

 

From the command line you should be able to type "diagnostics" and that will write the zip to the flash drive, where you can copy it to your local system.

 

If that is failing for some reason, please upload the /var/log/syslog as a starting point.

[DieFalse]

Ok - so I have made some progress -   I can access the webgui from another device on the network.   It seems my primary PC is blocked from accessing the WebGui.... Is there an Fail2Ban or similar item that would blacklist my connection to ports 80/443?

[itimpi]

52 minutes ago, fmp4m said:

Ok - so I have made some progress -   I can access the webgui from another device on the network.   It seems my primary PC is blocked from accessing the WebGui.... Is there an Fail2Ban or similar item that would blacklist my connection to ports 80/443?

Not unless you have explicitly set something up  

 

seems more likely it is something on the PC in question - although what it might be I have no idea.

[SimonF]

This was added in 6.9.2

 

Failed Login Restrictions
For webGUI login, you now get 3 login attempts per IP address before a 15-minute cool-off is enforced. Further, the timestamp of the last three failed login attempts per IP address are stored in files located in /var/log/pwfail/<ip-address>.

Note: this only applies to webGUI login, not ssh or telnet.

 

[DieFalse]

17 hours ago, SimonF said:

This was added in 6.9.2

 

 

 

I was thinking it was in the release notes.   Im wondering if for some reason it is blacklisting my desktop.   Where can I check the Fail2Ban config / what is blocked or add any whitelist (IE My home subnet)

[trurl]

Since it is in RAM like the rest of the OS just rebooting should clear the blacklist

[SimonF]

33 minutes ago, fmp4m said:

Where can I check the Fail2Ban config

Not sure where the config is, but its producing the following, just did a test on one of my test boxes,

 

root@computenode:~# ls /var/log/pwfail/
root@computenode:~# ls /var/log/pwfail/
192.168.1.28
root@computenode:~# cat  /var/log/pwfail/192.168.1.28 
1619624206
1619624210
1619624215
1619624219
1619624228
root@computenode:~# rm -r  /var/log/pwfail/192.168.1.28 
root@computenode:~# cat  /var/log/pwfail/192.168.1.28 
cat: /var/log/pwfail/192.168.1.28: No such file or directory
root@computenode:~# ls /var/log/pwfail/
root@computenode:~# 

Removing the IP address file allowed me to log in.

Edited April 28, 2021 by SimonF

[ljm42]

 

On 4/27/2021 at 12:13 PM, fmp4m said:

Ok - so I have made some progress -   I can access the webgui from another device on the network.   It seems my primary PC is blocked from accessing the WebGui.... Is there an Fail2Ban or similar item that would blacklist my connection to ports 80/443?

 

What exactly do you see when you try to access the webgui from the problematic computer? 

 

The Failed Login Protection that was added to 6.9.2:

https://wiki.unraid.net/Manual/Release_Notes/Unraid_OS_6.9.2#Failed_Login_Restrictions

does not block your access completely like Fail2Ban. It will still display the login page, it just won't process your login.

[DieFalse]

I get a timeout error - it never loads.  

 

SSH / NFS works - however file transfers themselves do not.  They timeout.    I rolled back one of the two servers and its still experiencing the issue.   I can access the server all ways except webgui on my primary pc.

 

Wireshark pcap didn't reveal anything. 

 

-------

 

 

Strangely enough - today I can access and login to both servers webgui ---- with NOTHING changed

 

Diag from when it wasn't working 

gsa-diagnostics-20210426-1109.zip

Edited April 30, 2021 by fmp4m

[ljm42]

Not sure if this is the cause, but it looks like there are some hardware memory issues:

 

Apr 26 07:01:21 GSA kernel: mce: [Hardware Error]: Machine check events logged
### [PREVIOUS LINE REPEATED 2 TIMES] ###
Apr 26 07:07:57 GSA kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Apr 26 07:07:57 GSA kernel: EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 10: 8c000050000800c1
Apr 26 07:07:57 GSA kernel: EDAC sbridge MC0: TSC 34a72b44769827 
Apr 26 07:07:57 GSA kernel: EDAC sbridge MC0: ADDR 1875bdc000 
Apr 26 07:07:57 GSA kernel: EDAC sbridge MC0: MISC 122120002000208c 
Apr 26 07:07:57 GSA kernel: EDAC sbridge MC0: PROCESSOR 0:306e4 TIME 1619438877 SOCKET 0 APIC 0
Apr 26 07:07:57 GSA kernel: EDAC MC0: 1 CE memory scrubbing error on CPU_SrcID#0_Ha#0_Chan#1_DIMM#0 or CPU_SrcID#0_Ha#0_Chan#1_DIMM#1 or CPU_SrcID#0_Ha#0_Chan#1_DIMM#2 (channel:1 page:0x1875bdc offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:0 ha:0 channel_mask:2 rank:255)
Apr 26 07:33:35 GSA kernel: mce: [Hardware Error]: Machine check events logged
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Apr 26 07:49:10 GSA kernel: mce_notify_irq: 2 callbacks suppressed
Apr 26 07:49:10 GSA kernel: mce: [Hardware Error]: Machine check events logged
Apr 26 07:49:10 GSA kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Apr 26 07:49:10 GSA kernel: EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 10: 8c000050000800c1
Apr 26 07:49:10 GSA kernel: EDAC sbridge MC0: TSC 34ad3d92ba5f28 
Apr 26 07:49:10 GSA kernel: EDAC sbridge MC0: ADDR 1875bdc000 
Apr 26 07:49:10 GSA kernel: EDAC sbridge MC0: MISC 122120002000208c 
Apr 26 07:49:10 GSA kernel: EDAC sbridge MC0: PROCESSOR 0:306e4 TIME 1619441350 SOCKET 0 APIC 0
Apr 26 07:49:10 GSA kernel: EDAC MC0: 1 CE memory scrubbing error on CPU_SrcID#0_Ha#0_Chan#1_DIMM#0 or CPU_SrcID#0_Ha#0_Chan#1_DIMM#1 or CPU_SrcID#0_Ha#0_Chan#1_DIMM#2 (channel:1 page:0x1875bdc offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:0 ha:0 channel_mask:2 rank:255)

 

[DieFalse]

The hardware memory issue only exists in the secondary server - the issue existed in both servers.   

 

The memory issue has been present for the last 6 version releases. 

[trurl]

39 minutes ago, fmp4m said:

memory issue has been present for the last 6 version releases

Why aren't you fixing it?

[DieFalse]

2 hours ago, trurl said:

Why aren't you fixing it?

Secondary server - hard to access - non-critical error.