6.9.2 locks up on HP DL380 Gen9 with P840 in HBA mode


Recommended Posts

Today I performed a clean reboot, and the lockup happened just after 14 minutes of uptime. My ssh session, sftp session were locked up, so I went to the web ui and saw the CPU utilization panel details weren't moving. I had to press the power button which started a graceful shutdown, however it got stuck at this screen below. I gave it about 10 minutes before I just unplugged the server. 

 

Things to note: Today I rebooted because I added 2 new disks. 1 for a storage pool, and 1 to expand the array. However this type of lockup has happened previously when I wasn't expanding the disks. 

 

What's interesting is the SFTP attempted to re-connect well after the runlevel 0 happened. 

 

Attached are the syslog reports. 

 

image.png.afd3bbee861aca6c85e546840874948f.png

graylog-search-result-relative-0.csv

Link to comment

I do see some nginx issues, but those are outside my wheelhouse:

 

2021-04-27T11:21:19.000Z,"unraid nginx: 2021/04/27 11:21:19 [error] 11092#11092: *4947 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.13, server: , request: ""POST /webGui/include/Boot.php HTTP/1.1"", upstream: ""fastcgi://unix:/var/run/php5-fpm.sock"", host: ""unraid:7000"", referrer: ""http://unraid:7000/Dashboard"""
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *4809 open socket #25 left in connection 10
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *5276 open socket #19 left in connection 9
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *5280 open socket #31 left in connection 13
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *3413 open socket #24 left in connection 8
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *3407 open socket #26 left in connection 18
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: aborting

 

Make sure you only have one browser windows open in the GUI, hopefully somebody else has more ideas.

Link to comment

 

1 hour ago, JorgeB said:

I do see some nginx issues, but those are outside my wheelhouse:

 




2021-04-27T11:21:19.000Z,"unraid nginx: 2021/04/27 11:21:19 [error] 11092#11092: *4947 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.13, server: , request: ""POST /webGui/include/Boot.php HTTP/1.1"", upstream: ""fastcgi://unix:/var/run/php5-fpm.sock"", host: ""unraid:7000"", referrer: ""http://unraid:7000/Dashboard"""
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *4809 open socket #25 left in connection 10
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *5276 open socket #19 left in connection 9
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *5280 open socket #31 left in connection 13
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *3413 open socket #24 left in connection 8
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: *3407 open socket #26 left in connection 18
2021-04-27T11:21:19.000Z,unraid nginx: 2021/04/27 11:21:19 [alert] 11092#11092: aborting

 

Make sure you only have one browser windows open in the GUI, hopefully somebody else has more ideas.

 

It's happened again after about an hour. My docker containers appear to still be working. 

 

So 3 times today :(

 

Yep, only 1 browser open. I'll be happy if it's just an nginx issue. How do I restart the web ui?

 

Edit: ok not all containers are working, but some are. Could it be a disk issue? 

 

Edit again: I see this in the syslog right before the nginx timeouts occur. 

Apr 27 12:39:49 unraid kernel: sd 7:0:4:0: Power-on or device reset occurred

 

Edited by drsprite
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.