HI all,

 

any idea why I could be getting 500 internal server error from unraid GUI nginx?

No, changes or updates to the server done.

Also, docker containers which are using br0 are of course having the same problem.

 

Diagnostics attached. unraid-diagnostics-20240821-0949.zip

 

After the reboot it worked for 30 minutes or so. But then error got back.

 

Also tried to restart nginx and php, so it helped for a while, but then it went back to the original error.

After that made a second diagnostics unraid-diagnostics-20240821-1745.zip

 

Did few packet captures basically it reflect logs as well.

 

Any ideas please?

unraid-diagnostics-20240821-0949.zip unraid-diagnostics-20240821-1745.zip

Log is full of these:

 

Aug 21 15:16:01 unraid nginx: 2024/08/21 15:16:01 [error] 10231#10231: *2609 limiting requests, excess: 20.445 by zone "authlimit", client: 192.168.1.8, server: , request: "GET /login HTTP/2.0", host: "192.168.1.200", referrer: "https://192.168.1.200/Dashboard"
Aug 21 15:16:05 unraid nginx: 2024/08/21 15:16:05 [error] 10231#10231: *2609 limiting requests, excess: 20.422 by zone "authlimit", client: 192.168.1.8, server: , request: "GET /login HTTP/2.0", host: "192.168.1.200", referrer: "https://192.168.1.200/Dashboard"
Aug 21 15:16:12 unraid nginx: 2024/08/21 15:16:12 [error] 10231#10231: *2609 limiting requests, excess: 20.894 by zone "authlimit", client: 192.168.1.8, server: , request: "GET /login HTTP/2.0", host: "192.168.1.200", referrer: "https://192.168.1.200/Dashboard"
Aug 21 15:16:16 unraid nginx: 2024/08/21 15:16:16 [error] 10231#10231: *2609 limiting requests, excess: 20.882 by zone "authlimit", client: 192.168.1.8, server: , request: "GET /login HTTP/2.0", host: "192.168.1.200", referrer: "https://192.168.1.200/Dashboard"

 

Quote

client: 192.168.1.8

 

I assume this is your PC? If yes try leaving the browser windows closed.

Yes, 192.168.1.8 is the PC from where I am trying to reach the GUI.

Also I tried multiple browsers and multiple machines, multiple OS, including win10, win7 and android.

While testing it from other system the browser in this case firefox 129.0.2 on system 192.168.1.8 (win10) was closed.

Edited August 21, 20241 yr by matew

In that case try booting is safe mode to rule out any plugin issues.

Well, honestly I thought it would be plugin or a docker problem.

Tried to /etc/rc.d/rc.nginx restart and /etc/rc.d/rc.php-fpm restart,  then installed dynamix safe mode plugin to disable all plugins and so far so good and stable GUI.

The reason why I have not booted into proper safe mode is that the parity check is still running, because apparently I did not rebooted unraid properly from the cli.

Edited August 21, 20241 yr by matew

Seems that the root caue of these symptoms was caused by frigate 0.14 container, running on br0, causing nginx to misbehave.

Re-installing frigate container and using bridge for it fixed the issue with nginx.

Now to find the reason why.

So, it was not a solution. It still keeps re-appearing.

 

Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: [mmhub] page fault (src_id:0 ring:24 vmid:4 pasid:32771, for process ffmpeg pid 15489 thread ffmpeg:cs0 pid 15658)
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:   in page starting at address 0x00008001072fb000 from client 0x12 (VMC)
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: MMVM_L2_PROTECTION_FAULT_STATUS:0x00403831
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      Faulty UTCL2 client ID: VCN (0x1c)
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      MORE_FAULTS: 0x1
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      WALKER_ERROR: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      PERMISSION_FAULTS: 0x3
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      MAPPING_ERROR: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      RW: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: [mmhub] page fault (src_id:0 ring:24 vmid:4 pasid:32771, for process ffmpeg pid 15489 thread ffmpeg:cs0 pid 15658)
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:   in page starting at address 0x00008001072fb000 from client 0x12 (VMC)
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: MMVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      Faulty UTCL2 client ID: MP0 (0x0)
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      MORE_FAULTS: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      WALKER_ERROR: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      MAPPING_ERROR: 0x0
Aug 30 21:20:29 unraid kernel: amdgpu 0000:0e:00.0: amdgpu:      RW: 0x0
Aug 30 21:20:39 unraid kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring vcn_dec_0 timeout, signaled seq=18111395, emitted seq=18111397
Aug 30 21:20:39 unraid kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process ffmpeg pid 15489 thread ffmpeg:cs0 pid 15658
Aug 30 21:20:39 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: GPU reset begin!
Aug 30 21:20:39 unraid kernel: [drm] Register(0) [mmUVD_POWER_STATUS] failed to reach value 0x00000001 != 0x00000002
Aug 30 21:20:40 unraid kernel: [drm] Register(0) [mmUVD_RBC_RB_RPTR] failed to reach value 0x00000360 != 0x00000320
Aug 30 21:20:40 unraid kernel: [drm] Register(0) [mmUVD_POWER_STATUS] failed to reach value 0x00000001 != 0x00000002
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: free PSP TMR buffer
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: MODE2 reset
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: GPU reset succeeded, trying to resume
Aug 30 21:20:40 unraid kernel: [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
Aug 30 21:20:40 unraid kernel: [drm] PSP is resuming...
Aug 30 21:20:40 unraid kernel: [drm] reserve 0xa00000 from 0xf41e000000 for PSP TMR
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: RAS: optional ras ta ucode is not available
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: RAP: optional rap ta ucode is not available
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: SMU is resuming...
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: smu driver if version = 0x00000004, smu fw if version = 0x00000005, smu fw program = 0, smu fw version = 0x00544fde (84.79.222)
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: SMU driver if version not matched
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: SMU is resumed successfully!
Aug 30 21:20:40 unraid kernel: [drm] DMUB hardware initialized: version=0x05000C00
Aug 30 21:20:40 unraid kernel: [drm] kiq ring mec 2 pipe 1 q 0
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring vcn_dec_0 test failed (-110)
Aug 30 21:20:40 unraid kernel: [drm:amdgpu_device_ip_resume_phase2 [amdgpu]] *ERROR* resume of IP block <vcn_v3_0> failed -110
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: GPU reset(1) failed
Aug 30 21:20:40 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: GPU reset end with ret = -110
Aug 30 21:20:40 unraid kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* GPU Recovery Failed: -110
Aug 30 21:20:41 unraid kernel: [drm] Register(0) [mmUVD_POWER_STATUS] failed to reach value 0x00000001 != 0x00000002
Aug 30 21:20:41 unraid kernel: [drm] Register(0) [mmUVD_RBC_RB_RPTR] failed to reach value 0x00000010 != 0x00000000
Aug 30 21:20:41 unraid kernel: [drm] Register(0) [mmUVD_POWER_STATUS] failed to reach value 0x00000001 != 0x00000002
Aug 30 21:20:43 unraid kernel: [drm] Fence fallback timer expired on ring sdma0
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Aug 30 21:20:51 unraid kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring vcn_dec_0 timeout, signaled seq=18111397, emitted seq=18111397
Aug 30 21:20:51 unraid kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process ffmpeg pid 15489 thread ffmpeg:cs0 pid 15658
Aug 30 21:20:51 unraid kernel: amdgpu 0000:0e:00.0: amdgpu: GPU reset begin!
Aug 30 21:23:39 unraid php-fpm[12845]: [WARNING] [pool www] server reached max_children setting (50), consider raising it