Husker_N7242C

I wasn't able to pin-down a hardware fault at first. I ran memtest86 with 3 passes on each of the modules 1 at a time then for 2 days straight with all 8 modules and got 0 errors. I replaced the HX850 PSU with a new RM850X single rail and checked that my total power draw is within spec (it is easily). I booted again and had a freeze within a day. Syslog now shows memory errors (as below). Unfortunately though doesn't show which module (I understand why it doesn't). So I probably have a memory issue, but if memtest86 can't find a fault I'm not sure how to work out which module is bad. I can't really affort to replace 64GB of ECC memory TBH. ANy ideas on how to narrow this down. Apr 29 07:29:50 NAS mcelog: Fallback Socket memory error count 32533 exceeded threshold: 2274769153 in 24h Apr 29 07:29:50 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:50 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:50 NAS mcelog: Too many trigger children running already Apr 29 07:29:50 NAS mcelog: Fallback Socket memory error count 17062 exceeded threshold: 2274704085 in 24h Apr 29 07:29:50 NAS mcelog: Fallback Socket memory error count 32533 exceeded threshold: 2274736619 in 24h Apr 29 07:29:50 NAS mcelog: Location: SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:50 NAS mcelog: Location: SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:50 NAS mcelog: Cannot collect child 9515: No child processes Apr 29 07:29:51 NAS mcelog: Fallback Socket memory error count 32112 exceeded threshold: 2274801266 in 24h Apr 29 07:29:51 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:51 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:51 NAS mcelog: Fallback Socket memory error count 18505 exceeded threshold: 2274819772 in 24h Apr 29 07:29:51 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:51 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:51 NAS mcelog: Fallback Socket memory error count 18505 exceeded threshold: 2274838278 in 24h Apr 29 07:29:51 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:51 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:51 NAS mcelog: Too many trigger children running already Apr 29 07:29:51 NAS mcelog: Fallback Socket memory error count 18505 exceeded threshold: 2274819772 in 24h Apr 29 07:29:51 NAS mcelog: Fallback Socket memory error count 32112 exceeded threshold: 2274801266 in 24h Apr 29 07:29:51 NAS mcelog: Location: SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:51 NAS mcelog: Location: SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:52 NAS kernel: mce_notify_irq: 120 callbacks suppressed Apr 29 07:29:52 NAS kernel: mce: [Hardware Error]: Machine check events logged Apr 29 07:29:52 NAS kernel: mce: [Hardware Error]: Machine check events logged Apr 29 07:29:52 NAS mcelog: Fallback Socket memory error count 18025 exceeded threshold: 2274856304 in 24h Apr 29 07:29:52 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:52 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:52 NAS mcelog: Fallback Socket memory error count 12193 exceeded threshold: 2274868498 in 24h Apr 29 07:29:52 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:52 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:52 NAS mcelog: Fallback Socket memory error count 12193 exceeded threshold: 2274880692 in 24h Apr 29 07:29:52 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:52 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:52 NAS mcelog: Too many trigger children running already Apr 29 07:29:52 NAS mcelog: Fallback Socket memory error count 18025 exceeded threshold: 2274856304 in 24h Apr 29 07:29:52 NAS mcelog: Fallback Socket memory error count 12193 exceeded threshold: 2274868498 in 24h Apr 29 07:29:52 NAS mcelog: Location: SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:52 NAS mcelog: Location: SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:53 NAS mcelog: Fallback Socket memory error count 31707 exceeded threshold: 2274912400 in 24h Apr 29 07:29:53 NAS mcelog: Location SOCKET:0 CHANNEL:? DIMM:? [] Apr 29 07:29:53 NAS mcelog: Running trigger `socket-memory-error-trigger' (reporter: sockdb_fallback) Apr 29 07:29:53 NAS mcelog: Fallback Socket memory error count 23544 exceeded threshold: 2274935945 in 24h

Thanks again. I've disabled EPC and low current spinup on all drives (8 of which are the model mentioned in the guide and another is a 3TB which had both features enabled). This is currently the best value NAS Seagate drive in Australia (and has been for some time now). Is there a way to build the SeaChest executable files into UNRAID and have them automatically check and disable these features on all Seagate drives? when adding a new drive? It seems like this would be a relatively simple script to embed as a bug-fix? (I know nothing about this stuff really, just seems silly to expect the users to have to fix this bug for themselves and repeat everytime they add a drive). Regarding the crashing, I've ordered a new PSU (haven't tested but it is due anyway) and will run a memory test and go from there. Are there any hardware diagnostic tools that you would suggest to check the LSI controllers and the motherboard?? Anywhere else that helpful logs might be hiding?

Thanks JorgeB, I enabled this and I have PM'd you the log from after today's lock-up with a bit more explanation. Thanks again for helping!

Hi everyone, I'm at a bit of a loss - every couple of days the server goes offline and is not responsive (including GUI and webGUI). Sometimes a disk is randomly disabled but once added back again, often doesn't get disabled again, but another disk gets disabled a few days later. I've replaced all SATA and power cables and swapped the drives around to try to find a pattern, but can't see one. Someone suggested that 8TB Seagate Ironwolf drives have an issue with spinning down, so I disabled spin-down which didn't help. I've tried running it with all dockers disabled and just one VM running (no hardware passthrough) and still get hangs. I'm not sure if diagnostics is any help as I think it clears each hang but I've attached just in case. Any suggestions of what to try next?? Much appreciated guys! Hardware basics: AsRock x79 Extreme 11 with E5-2670 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06) Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) 32GB DDR3 1333MHz ECC Memory (4x 8GB) 7x 8TB Seagate Ironwolf ST8000VN04-2M2101 (Parity+6 data) 2x 3TB Seagate Barracuda ST3000DM007-1WY10G (data) 2x 2TB Seagate NAS ST2000VN000-1HJ164 (data) 1x 2TB Seagate Barracuda ST2000DM006-2DM164 (data) Plus 3x Cache pools with 7 SSDs total (mix of Samsung and Crucial/Micron) diagnostics-20220417-1647.zip