[SOLVED] Server crashing every 3-4 days

[cheww]

Hello!,

 

I'm having an issue with my server its crashing every 3-4 days. Becomes unresponsive and unreachable.

 

II have noticed that i am getting ATA-errors, i'm not sure if that's was causing that issue. The server usually crashes after midnight when its doing nothing. I originally thought it was the backup it was doing of the docker apps which happens after midnight. I have disabled the backups and it still crashed.

 

I have replaced all the Sata cables and the sas to sata cables with name brand but it still happened.

 

A little history i have these 10tb seagate hdd's that a few people here had issues with and the LSI cards. Hard drives would go missing or just error out. I had to put all my 10 tb hdd's onto the motherboard Sata ports and all my 3tb wd hdd are using the LSI card. i have 2 cache drives using an internal PCI sata card and i have 1 nvme sata drive aslo using it as an incoming data drive for download coming into the server.

 

any advice would be most welcome. I think my next step is to try safe mode. Could some one please look at my diags to see if there is any issue?

 

For the Syslog it crashed some time between Jan 24 00:00:37 to Jan 24 15:29:47 which is when i came home from work and powered it off using the power button.

 

Cheers and many thanks.

 

gauntlet-diagnostics-20240124-2032.zip syslog-192.168.4.12.log

Edited February 1 by cheww
Responce solved issue

[JorgeB]

The ATA errors are likely unrelated, but you still should try and fix them, it will slow down performance and spam the log, try replacing cables for both affected disks.

 

Jan 24 15:31:16 Gauntlet kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jan 24 15:31:16 Gauntlet kernel: ? _raw_spin_unlock+0x14/0x29
Jan 24 15:31:16 Gauntlet kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

[cheww]

On 1/24/2024 at 10:41 PM, JorgeB said:

The ATA errors are likely unrelated, but you still should try and fix them, it will slow down performance and spam the log, try replacing cables for both affected disks.

 

Jan 24 15:31:16 Gauntlet kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jan 24 15:31:16 Gauntlet kernel: ? _raw_spin_unlock+0x14/0x29
Jan 24 15:31:16 Gauntlet kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

 

Switching to ipvlan sold the crashing issue. Which is most important!! Thank you.

 

Replacing the cables did not fix the ATA errors. Need to find out which hard drives this issue is affecting or may be my MB sata ports or LSI card issue.

 

But thanks for helping with the main issue.

 

 

 

[cheww]

Just another update - I have sorted out the ATA errors.

 

The cause was not enough power to the back plane. I have a 24 bay hdd case with a back plane that has 24 sata ports and 6 molex power connectors.

 

i had 2 lanes from the Power supply going to all 6 power ports. I disconnected 1 from each 3 ( 3 control 12 hdd and 3 control the other 12 hdd) power ports and added another lane of power so now its getting 3 lanes of power from the power supply.

 

The ata errors have stopped completely i have started a parity check and its been going for 6 hours and not one error.

 

Hope this helps someone ever experiencing this issue.