webgui throwing 500 error

[tetrapod]

I can no longer reach my servers Unraid GUI. It worked just fine a couple of days ago.

 

 

If I click [Remote access] I'm directed to the URL 

https://www.xxxxxxxxxxxxxxxxxxxxxxxxxx.unraid.net:<port>/Dashboard

where I get a 500 Internal server error from nginx

 

I have a wireguard tunnel active from where I can remotly reach the GUI of active containers via this URL format

http://<server LAN IP>:<port>/

 

I can also reach services like SMB via the wireguard tunnel, but not the Unraid GUI like I could before via

https://<server LAN IP>/Dashboard

I get the same 500 Internal server error from nginx

 

What can be wrong?

I do at the moment have access to a shell via a connected PiKVM, but do not want to do an unessasary restart and what to find the root of the problem. The server is geografically located in another country and I'm terrified to do something stupid.

[ljm42]

Access the command line of the server via PiKVM and type:
  diagnostics
Note where it tells you it put the file on your flash drive.

 

Then download that file from your flash drive and upload it here.

[tetrapod]

Thank you

treebeard-diagnostics-20221201-1029.zip

[ljm42]

At the command line, type:

/etc/rc.d/rc.nginx reload

If that brings back the webgui, great! Otherwise you'll need to type this to reboot:

reboot

So the system is throwing call traces:

Nov 24 16:48:34 treebeard kernel: WARNING: CPU: 15 PID: 0 at net/netfilter/nf_conntrack_core.c:1205 __nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Nov 24 16:48:34 treebeard kernel: Modules linked in: xt_connmark xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_mark nvidia_uvm(PO) xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 igb i2c_algo_bit r8169 realtek nvidia_drm(PO) nvidia_modeset(PO) edac_mce_amd edac_core nvidia(PO) kvm_amd wmi_bmof mxm_wmi drm_kms_helper kvm drm btusb btrtl btbcm crct10dif_pclmul btintel crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd bluetooth mpt3sas cryptd sr_mod rapl

Do you have "Docker custom network type" set to macvlan? You might try changing that to ipvlan (on Settings -> Docker Settings). I'm not super familiar with this but there are other discussions about it here, or maybe someone else can chime in.

In general, I'd recommend upgrading from Unraid 6.11.1 to 6.11.5 but I know that can be scary if you are remote.

 

The only other thing I see are some potential attacks:

Nov 11 15:06:33 treebeard nginx: 2022/11/11 15:06:33 [crit] 5800#5800: *65198 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 43.128.225.197, server: 0.0.0.0:443

Nov 12 10:47:18 treebeard kernel: TCP: request_sock_TCP: Possible SYN flooding on port 22922. Sending cookies.  Check SNMP counters.

I'm guessing you have My Servers Remote Access enabled? If script kiddies have found your server I would change your Remote Access port to a different, high random number. Something between 1024 and 64,000. I don't know if this could be causing issues but I would change the port out an abundance of caution.

 

You should also switch from the old unraid.net certificate to the new myunraid.net certificate, it has a different style of url that is more private. You can do this on the Settings -> Management Access page, I forget the exact wording but it is something like "upgrade certificate"

 

And optionally, you can ask your ISP for a new IP address if you think someone is targeting you specifically.

 

Oh, and it won't affect anything we've talked about here but please upgrade your My Servers plugin  
https://forums.unraid.net/topic/112073-my-servers-releases-and-announcements/#comment-1196555 

 

[tetrapod]

Thank you so much for being thorow in your reply. I'll will have a look tomorow and tell you here how it goes. 

[tetrapod]

23 hours ago, ljm42 said:

/etc/rc.d/rc.nginx reload

Did restart normally, but didn't do any difference for unraid GUI access

 

23 hours ago, ljm42 said:

reboot

Yes, that did start the reboot command, but I lost the prompt and all services seemed to be still open. I could see the disk-activity lamp plinking through the PiKVM interface

I waited an hour and no difference:

 

...so used ssh in and did another reboot which did reboot the server, but triggered the parity-check 😐

 

After this I did manage to get the GUI, but not through the 

https://www.xxxxxxxxxxxxxxxxxxxxxxxxxx.unraid.net:<port>

 

I had to go through

http://<WAN IP>:<port>/

 

Now I upgraded to 6.11.5 which went well, but still the behavior from above. I changed to the new myunraid certificate and that helped. Now I could get the GUI from my servers page. Still couldn't get VPN access through my WireGuard tunnels, but after a while I realized I had to update the the tunnels after the URL changed.

 

On 12/1/2022 at 5:19 PM, ljm42 said:

Do you have "Docker custom network type" set to macvlan?

Yes, I do. I'm reading up on this and will m a y b e change it. Need to breathe first a couple of days. My old man heart is not built for remote updates

 

On 12/1/2022 at 5:19 PM, ljm42 said:

Oh, and it won't affect anything we've talked about here but please upgrade your My Servers plugin

It was the latest. Got uppdated automagically somewhere along the way.

 

Thank you for the quick help and have a nice weekend.