I've had quite a bit of trouble accessing my server since adding a dedicated router. Most problems seem to be user error but recently I was unable to access my server at all. I posted about this and wasn't able to find an answer. Ultimately I had to power the server down with the power button on my tower... Everything was OK after the unclean shutdown and I was able to access my server again.

 

Today I started getting error 500 and wasn't really able to ssh in either. I finally loaded ssh and was able to get diagnostics and reboot. It was another unclean shutdown but I can access the server now. I've attached logs and am really hoping someone can tell me what happened so I can prevent this issue from repeating.
 

EDIT: I tried looking through the logs myself and it looks like I'm having the same problem that a lot of others have experienced. This is the error I'm referring to, repeating over and over (there might be more, I don't really know what I'm looking for):

 

12:04:50 TrueSource php-fpm[8840]: [WARNING] [pool www] child 5041 exited on signal 9 (SIGKILL) after 12.901833 seconds from start

 

I found a bunch of threads related to this but no fixes.

 

It also looks like there is something wrong with disk 7. I'm running a SMART short self-test and I have no idea when it's going to finish. I've attached the existing SMART report which should have relevant info. Pretty sure that this drive needs to be replaced but I'd like to know if it's also contributing to my access issues.

truesource-diagnostics-20231207-1149.zip truesource-diagnostics-20231207-1148.zip

truesource-smart-20231207-1539.zip

Edited December 7, 20232 yr by Drogon

Looking at the diags ending in 1149, I see some issues with disks that somebody else might be able to help with:

 

Dec  5 22:21:57 TrueSource kernel: ata6.00: exception Emask 0x0 SAct 0x80ff8001 SErr 0x0 action 0x0
Dec  5 22:21:57 TrueSource kernel: ata6.00: irq_stat 0x40000008
Dec  5 22:21:57 TrueSource kernel: ata6.00: failed command: READ FPDMA QUEUED
Dec  5 22:21:57 TrueSource kernel: ata6.00: cmd 60/00:78:c8:4c:23/04:00:cf:01:00/40 tag 15 ncq dma 524288 in
Dec  5 22:21:57 TrueSource kernel:         res 41/40:00:40:4d:23/00:00:cf:01:00/00 Emask 0x409 (media error) <F>
Dec  5 22:21:57 TrueSource kernel: ata6.00: status: { DRDY ERR }
Dec  5 22:21:57 TrueSource kernel: ata6.00: error: { UNC }
Dec  5 22:21:57 TrueSource kernel: ata6.00: configured for UDMA/133
Dec  5 22:21:57 TrueSource kernel: sd 6:0:0:0: [sds] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=4s
Dec  5 22:21:57 TrueSource kernel: sd 6:0:0:0: [sds] tag#15 Sense Key : 0x3 [current] 
Dec  5 22:21:57 TrueSource kernel: sd 6:0:0:0: [sds] tag#15 ASC=0x11 ASCQ=0x4 
Dec  5 22:21:57 TrueSource kernel: sd 6:0:0:0: [sds] tag#15 CDB: opcode=0x88 88 00 00 00 00 01 cf 23 4c c8 00 00 04 00 00 00
Dec  5 22:21:57 TrueSource kernel: I/O error, dev sds, sector 7770164424 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 2
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164360
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164368
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164376
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164384
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164392
...

Dec  6 02:48:39 TrueSource kernel: sd 13:0:1:0: [sdc] tag#1192 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Dec  6 02:48:39 TrueSource kernel: sd 13:0:1:0: [sdc] tag#1192 Sense Key : 0x5 [current] 
Dec  6 02:48:39 TrueSource kernel: sd 13:0:1:0: [sdc] tag#1192 ASC=0x21 ASCQ=0x0 
Dec  6 02:48:39 TrueSource kernel: sd 13:0:1:0: [sdc] tag#1192 CDB: opcode=0x42 42 00 00 00 00 00 00 00 18 00
Dec  6 02:48:39 TrueSource kernel: critical target error, dev sdc, sector 1950420992 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2
Dec  6 02:48:39 TrueSource kernel: BTRFS warning (device sdt1): failed to trim 1 device(s), last error -121

Shortly after that there are a bunch of errors from php-fpm and nginx, which led to the error 500's you were seeing. I am not sure if a disk issue can cause these php/nginx errors

 

I do see this system has a Realtek RTL8168h/8111h network card, I'd suggest trying the alternate drivers for that nic:

https://forums.unraid.net/topic/141349-plugin-realtek-r8125-r8168-and-r81526-drivers/

 

 

Thanks! I didn't know they needed to be changed. I'll try that and see how it goes.

10 hours ago, ljm42 said:

Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164376
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164384
Dec  5 22:21:57 TrueSource kernel: md: disk7 read error, sector=7770164392

These are logged as a disk problem, and SMART shows some warning signs, but they can be intermittent, suggest running an extended SMART test.

4 hours ago, JorgeB said:

These are logged as a disk problem, and SMART shows some warning signs, but they can be intermittent, suggest running an extended SMART test.

 

Thanks. Any idea what's going on with these errors?

 

12:04:50 TrueSource php-fpm[8840]: [WARNING] [pool www] child 5041 exited on signal 9 (SIGKILL) after 12.901833 seconds from start

Try booting in safe mode and/or closing any browser windows open to the GUI, only open when you need to use it then close again.

When you say open to the GUI, do you mean the unraid GUI specifically? Would leaving something like a Homarr tab open contribute to the problem?

Edited December 8, 20232 yr by Drogon

6 minutes ago, Drogon said:

do you mean the unraid GUI specifically?

Yes.

 

6 minutes ago, Drogon said:

Would leaving something like a Homarr tab open contribute to the problem?

Not sure.