ITs been a while this type of sudden unreasoned symptom doesn't occur for quite some time and recently it happen again. 

 

Ive try to create log on one of my network folder and hope it long something before the server respond.

Ive force reboot the server and generate a diagnostic after the server boots up.

Not sure Guru here manage to pinpoint anything.

 

ts-p500-diagnostics-20250403-1609.zip syslog-10.10.2.10.log.1

There are multiple call traces logged at the end, but difficult to say if they are hardware or software related, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers. 

4 hours ago, JorgeB said:

There are multiple call traces logged at the end, but difficult to say if they are hardware or software related, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers. 

Hi @JorgeB Thanks for your respond. The thing is it don't have a pattern the last crash if not mistaken are nearly a year back.

During the last crash I've done Memtest and Hardware stress test also pass. I don't have VMs just dockers and the server runs as purely NAS and Media application only.

If it happens once a year, it's going to very difficult to diagnose.

Yes last year it happened quite frequently I remember when I bought 2 12tb Seagate Exon and it had a similar crash while doing preclear. But after completed the two drives preclear the system never had such crash until recently.

 

according to those call tracer anything you might think off?

 

Another update I had a fren which had a similar system, same motherboard, same processor, different memory amount, different drives and nvme.

He told me he had similar crash intermittently as well sometime the system crash and reboot itself, sometime everything just not responding can’t remote and force reboot just like my cases. He mention this symptom start from version 6.12 onward like mine. 

Not really, but server rebooting by itself is almost always a hardware issue.

On 4/4/2025 at 3:00 PM, JorgeB said:

Not really, but server rebooting by itself is almost always a hardware issue.

But how to troubleshoot as it dont crash at the same situation so i dont know how to simulate and replicate the issue.

I dont overclock the hardware aswell. What type of info should i provide.

 

A good place to start is typically with the RAM, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM, but like mentioned if the server crashes once per year, it's going to be difficult to test, PSU is another good suspect, if you have another PC, swap PSUs and retest.

On 4/12/2025 at 2:51 PM, JorgeB said:

A good place to start is typically with the RAM, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM, but like mentioned if the server crashes once per year, it's going to be difficult to test, PSU is another good suspect, if you have another PC, swap PSUs and retest.

Noted, Will try that out. i had other PSU and may swap out the RAM. 

But i've run memtest before and come out pass will that's enough on the RAM portion?

19 hours ago, feins said:

But i've run memtest before and come out pass will that's enough on the RAM portion?

Memtest is only definitive if it finds errors, so this can be a better test.

On 4/14/2025 at 3:11 PM, JorgeB said:

Memtest is only definitive if it finds errors, so this can be a better test.

Hi @JorgeB its been a while, yesterday my server crashed and as usual a hard reboot and the server back online.

I got some error on the logs. and i attach here together with the diagnostic.

My last message received from my Telegram at 4.40am and i found the server crashed and cannot login or SSH at 12:40pm where i initial a hard reboot.

Hope the logs find something on your site.

ts-p500-diagnostics-20251229-1822.zip syslog-10.10.2.10.log

There are multiple call traces logged, also filesystem corruption on disk3:

Dec 29 03:14:57 TS-P500 kernel: XFS (md3p1): Metadata corruption detected at xfs_dinode_verify+0x250/0x6f0, inode 0x2c1 dinode

Dec 29 03:14:57 TS-P500 kernel: XFS (md3p1): Unmount and run xfs_repair

After that there are dozens of out-of-memory events logged; they appear to be related to VMAs and task structures using much more RAM than normal.

Recomend booting the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.

3 minutes ago, JorgeB said:

There are multiple call traces logged, also filesystem corruption on disk3:

Dec 29 03:14:57 TS-P500 kernel: XFS (md3p1): Metadata corruption detected at xfs_dinode_verify+0x250/0x6f0, inode 0x2c1 dinode

Dec 29 03:14:57 TS-P500 kernel: XFS (md3p1): Unmount and run xfs_repair

After that there are dozens of out-of-memory events logged; they appear to be related to VMAs and task structures using much more RAM than normal.

Recomend booting the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.

Oooh i dont have VMs just Docker and Plugins. and this attach screen shot is the only Docker that being used.

Did it indicate what's been utilized more RAM than normal?

Ive reboot the server in normal mode since yesterday and seems stable till now.

The logs now looks pretty quiets aswell.

On the Disk3 filesystem corruption can it be recover?

syslog-10.10.2.10.log

8 minutes ago, feins said:

On the Disk3 filesystem corruption can it be recover?

Check filesystem