Server is unresponsive, won't mount drives, log is full and other misc issues.

[ZVeguillaCotto]

As per the title.

 

I have a new build that has started having errors and crashes recently. I'm currently remote and suspect a bad HBA or connection as per googling.

 

Thankfully I can still access the glitchy GUI and can remote reboot via a smart plug.

 

Can anybody help me pinpoint and correct the issue?

 

Thanks.

 

 

root@HL15:~# df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           128M  128M     0 100% /var/log
root@HL15:~# du -sm /var/log/*
1       /var/log/Xorg.0.log
1       /var/log/apcupsd.events
0       /var/log/btmp
0       /var/log/cron
0       /var/log/debug
1       /var/log/dmesg
0       /var/log/faillog
0       /var/log/lastlog
0       /var/log/libvirt
0       /var/log/maillog
0       /var/log/mcelog
0       /var/log/messages
0       /var/log/nfsd
1       /var/log/nginx
0       /var/log/packages
1       /var/log/pkgtools
0       /var/log/plugins
0       /var/log/preclear
0       /var/log/pwfail
0       /var/log/removed_packages
0       /var/log/removed_scripts
0       /var/log/removed_uninstall_scripts
0       /var/log/sa
1       /var/log/samba
0       /var/log/scripts
0       /var/log/secure
0       /var/log/setup
0       /var/log/spooler
0       /var/log/swtpm
128     /var/log/syslog
1       /var/log/tailscale-utils.log
1       /var/log/tailscale.log
0       /var/log/vfio-pci
0       /var/log/vfio-pci-errors
1       /var/log/wtmp

hl15-syslog-20240303-0145.zip

Edited March 3 by ZVeguillaCotto

[itimpi]

The syslog shows you getting read errors on multiple drives and this is eventually filling up the space allowed for logging.   What do they share than can cause this (e.g. power, HBA, etc).

[ZVeguillaCotto]

7 hours ago, itimpi said:

The syslog shows you getting read errors on multiple drives and this is eventually filling up the space allowed for logging.   What do they share than can cause this (e.g. power, HBA, etc).

Power via 1 PSU port to 4 molex (backplane) , HBA 9300-16i 1 cable per 4drive group (0-3, 4-7, 8-11 , 12-14).