syslog killing my unraid

[sensia255]

Hello everyone,

My first post here; I was really hoping I don't need it as I have tested Unraid for about a month. Very recently, as in last three days, I am noticing my syslog becoming huge to the point the system becomes inaccessible. Doing a df -h, I am seeing /var/log is completely full and syslog is taking up that entire 128MB. It seems, hell is breaking loose with my network. Here is an excerpt;

"Feb 23 11:37:05 Tower kernel: alx 0000:02:00.0 eth0: fatal interrupt 0x4001607, resetting" 

Tower is my hostname. I am not sure what is going on here or how to troubleshoot. alx module indicates it has something to do with my networking but this has not happened until I failed to install an Ubuntu VM. I have deleted all of that failed installation but now the server is in an unstable state because there is no predictable way to tell when syslog may fill up.

Can't upload my syslog since it is 128MB in size. Can't get the fresh one, since the system went down again.

Need to add that my setup has one 14tb parity drive and a 14tb and 8tb backup drive. No caching drive involved in this install.

Also uploaded a truncated version of syslog.

Truncated_syslogtxt.txt

 

Now added zipped syslog when problem started and a fresh syslog, just after reboot from firmware upgraded to 6.12.8.

problematic_old_6.12.6-syslog.zip Clean_after_reboot_syslog-20240224-0315.zip

Edited February 24 by sensia255
Added problematic and fresh syslog

[trurl]

26 minutes ago, sensia255 said:

Can't upload my syslog

Please zip it, text compresses very well. 

 

Then after we have that, you can reboot and

 

Attach Diagnostics to your NEXT post in this thread 

[sensia255]

Thanks! uploaded the files in OP.

[Squid]

11:37 2/23 something happened with your network adapter.   Diagnostics would also help, but you're going to need to reboot and if it's an add-on NIC card, reseat it in the slot

 

[sensia255]

Diagnostics is attached. The NIC is integrated into Z97M motherboard.

Is there a way to move the syslog to a different location before the system dies? I just rebooted and lost the old syslog predictably.

tower-diagnostics-20240224-1145.zip

[itimpi]

The syslog in the diagnostics is the RAM version that starts afresh every time the system is booted.  You should enable the syslog server (probably with the option to Mirror to Flash set) to get a syslog that survives a reboot so we can see what leads up to a crash.  The mirror to flash option is the easiest to set up (and if used the file is then automatically included in any diagnostics), but if you are worried about excessive wear on the flash drive you can put your server's address into the remote server field. 

[sensia255]

As of now, I have enabled the syslog server. I think this set option will be sufficient; townfall2 is in my shared array. Remote syslog server is additionally set to my unraid server's IP.

Unfortunately, this solution does not work when I have to manually restart the server. 

Edited February 24 by sensia255

[trurl]

syslog server will not stop you from filling log space

 

Don't know why that NIC is spamming syslog. Can you disable it in BIOS and install another?

[sensia255]

Well, I guess there is no way to know other than building a new array. No, I don't have access to a new NIC. I mean a new NIC is half the cost of this old PC. 

Is there any option to flashback my system to an older date? I am certain all was good not too long ago.

Edited February 25 by sensia255

[sensia255]

It seems there were some bug back in 2016. I am not sure if that was worked out into latest Unraid. Anyway, I reinstalled the 6.12.8 in a new usb drive to start afresh and the problem came back quickly.

https://bugzilla.kernel.org/show_bug.cgi?id=70761 Old error thread with Atheros alx driver

And entry from my ballooning syslog;

Feb 23 11:37:18 Tower kernel: alx 0000:02:00.0 eth0: fatal interrupt 0x4001607, resetting

[JorgeB]

Please post new diagnostics showing the log spam.

[sensia255]

7 hours ago, JorgeB said:

Please post new diagnostics showing the log spam.

There is nothing new to add; the logfile (problematic_old_6.12.6.....) in my OP  still represents the current problem. I can reproduce this problem with 6.12.8, mounted on a fresh USB startup disk.

[JorgeB]

15 minutes ago, sensia255 said:

There is nothing new to add

Sorry, the last diags you posted don't have the error, I missed the first ones.

 

From what I can find this is a kernel error with that driver, some report that using jumbo frames helps, did you try that? If not try setting the MTU to 9000.

 

If that doesn't help I'm afraid there won't be much LT can do, unless it's fixed in the Linux kernel and merged upstream, then the fix will be available in a future Unraid release that includes that kernel.

[sensia255]

Ok; I will try changing the MTU to 9000. The search results are very few related to this specific problem. Worst case, I need to disable the onboard LAN controller and use a different NIC. However, I am still very very upset that nothing happened in my first month of the trial period.

[sensia255]

So, adding further productive update. I disabled collecting syslog and the array is going strong for a day. Will see if some shenanigans come back after installing a VM.

[trurl]

1 hour ago, sensia255 said:

disabled collecting syslog

 

On 2/24/2024 at 5:38 PM, trurl said:

syslog server will not stop you from filling log space

 

If you mean you disabled syslog server, syslog server has nothing to do with filling log space. It just saves a copy of the syslog elsewhere so it will survive reboot. syslog is still collected in /var/log, which is in RAM like the rest of the OS.