Parity Disk on new tower fails

[AceBoyden]

Parity Disk on new tower fails

when doing the parity check on my new tower it gets to a few percent then aborts, saying i have a hardware issue, im not sure what it is, i've tried multiple different drives yet nothing works. can someone who knows a bit more then me have a look?

bigboy-diagnostics-20230224-0824.zip

[JorgeB]

Diags are after a reboot so we cannot see what happened, disk looks healthy, replace/swap cables and try again, if there are more errors post new diags before rebooting.

[AceBoyden]

2 hours ago, JorgeB said:

Diags are after a reboot so we cannot see what happened, disk looks healthy, replace/swap cables and try again, if there are more errors post new diags before rebooting.

I replaced all sata cables and power cables and it still gives me a error. i can run all disk just fine and healthy in an array, but if i create a new config where i have one of the disks in parity it will run a parity check for about 10 minutes then turn off the array and say hardware error, so not fully sure if there is a setting i should change

[JorgeB]

You need to re-sync parity, to do that unassign parity, start array, stop array, re-assign parity, start array to begin parity sync.

[AceBoyden]

1 minute ago, JorgeB said:

You need to re-sync parity, to do that unassign parity, start array, stop array, re-assign parity, start array to begin parity sync.

i've done that aswell but it still gives me the error, i can try again tonight to see if it does it full since estimated time is around 8 hours

[itimpi]

You will need to provide diagnostics that are taken after the error occurs.    If the whole system is crashing enable the syslog server to get logs that survive a reboot

[JorgeB]

3 hours ago, JorgeB said:

if there are more errors post new diags before rebooting.

 

[AceBoyden]

i just got an error now saying "/var/log is getting full (currently 100 % used)" after starting the parity sync after the reboot

bigboy-diagnostics-20230224-1312.zip

[JorgeB]

No parity check going on on those diags.

[AceBoyden]

Just now, JorgeB said:

No parity check going on on those diags.

is there somewhere else they are stored?

[JorgeB]

Nope.

[AceBoyden]

ok i'll try again now but in the meantime i found the error that was filling up my sys log, seems like my gpu is the fault with the vm perhaps?

Feb 24 13:04:31 BigBoy kernel: vfio-pci 0000:07:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]
Feb 24 13:04:31 BigBoy kernel: vfio-pci 0000:07:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]
Feb 24 13:04:31 BigBoy kernel: vfio-pci 0000:07:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]
Feb 24 13:04:31 BigBoy kernel: vfio-pci 0000:07:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]
Feb 24 13:04:31 BigBoy kernel: vfio-pci 0000:07:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]
Feb 24 13:04:31 BigBoy kernel: vfio-pci 0000:07:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]

 

[AceBoyden]

Started it now so we'll see how it goes again

[AceBoyden]

ok so it got to 9.8% before the array stopped

bigboy-syslog-20230224-1406.zip bigboy-diagnostics-20230224-1505.zip

[JorgeB]

Diags are after rebooting.

[AceBoyden]

36 minutes ago, JorgeB said:

Diags are after rebooting.

heres new one after reboot

bigboy-diagnostics-20230224-1652.zip

[AceBoyden]

not sure why but after that reboot the array turns off around 10 minutes being on and i need to turn them on, also auto start keeps turning back to off from on

[AceBoyden]

*sorry, it reboots after around 10 minutes of being on, not stopping array alone. not sure why it keeps rebooting after being on for around 10 minutes, i fear it might be something to do with the cache drive and how i set it up in appdata but im no genius 

[itimpi]

5 minutes ago, AceBoyden said:

*sorry, it reboots after around 10 minutes of being on, not stopping array alone. not sure why it keeps rebooting after being on for around 10 minutes, i fear it might be something to do with the cache drive and how i set it up in appdata but im no genius 

If you are getting the system unexpectedly rebooting than I would think the most likely culprits are either a thermal problem or a PSU issue.

[AceBoyden]

Just now, itimpi said:

If you are getting the system unexpectedly rebooting than I would think the most likely culprits are either a thermal problem or a PSU issue.

it seems to only reboot after the rust server on pterodactly starts, i moved that first to start (rebooted immidetly), moved it last (rebooted after some time) and now i deleted that one and it has not rebooted yet, started array in maintance now to see if that works for parity sync, but i still keep getting "machine check events - Your server has detected hardware errors" but not sure where to find whats wrong on where  

[JorgeB]

Those suggest a hardware problem, at first I understood that parity was getting disabled, not that the server was crashing, I see you are overclocking the RAM, so start here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

[AceBoyden]

On 2/24/2023 at 5:29 PM, JorgeB said:

Those suggest a hardware problem, at first I understood that parity was getting disabled, not that the server was crashing, I see you are overclocking the RAM, so start here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

I disabled overclocking on the ram and set it to "auto" in bios, same as set the "Power Supply Idle Control" to "typical current idle" in bios as well. i changed the ram to check (removed one by one and did a new set aswell) yet still have the same issues, it appears it dosn't happen if array is started in maintanance mode (this only lasted 11 hours as that was the time the parity sync took, then i startet array normally). it went fine for a few hours then saw the server was down again (not fully down but array had been stopped and auto start was set to "no" again). it appears what happends is that the OS itself restarts but not the system, i was in my server room while the crash happened and saw on my monitor in there that the os was restarting but not the system itself. i tried checking the drives and unplugging 1 by 1 there aswell and tested everyone alone, still same problem occoured. i can have no vm's or containers running and it will still happend, none of my friends or collegues can find any errors physical or in the diognostic logs i sent them before and after the crashes. any other suggestions? (also changed cpu to the same type just a different one i had spare and problem is still there) could it be my usb stick i have unraid on? everything should be fine there aswell

[AceBoyden]

not sure if relevant but the "crashed" has happened 19 times in a span of 1 hour 

 

 

edit: disabled c sate in bios now so i'll see how it goes tomorrow if that helped any way

edit edit: after checking sys logs after disabling c state im not getting an error im not fully sure what means, if it is the cpu or the ram..

 

Quote

Feb 25 23:37:34 BigBoy kernel: mdcmd (36): check correct Feb 25 23:37:34 BigBoy kernel: md: recovery thread: check P ... Feb 25 23:37:37 BigBoy kernel: nvidia-modeset: WARNING: GPU:0: Unable to read EDID for display device DVI-D-0 Feb 25 23:37:39 BigBoy kernel: traps: xdg-desktop-por[20026] trap int3 ip:14afc24a1ca7 sp:7ffda76f2590 error:0 in libglib-2.0.so.0.6600.8[14afc2465000+88000] Feb 25 23:37:39 BigBoy kernel: traps: xdg-desktop-por[19950] trap int3 ip:14fb1992bca7 sp:7fffd2027c80 error:0 in libglib-2.0.so.0.6600.8[14fb198ef000+88000] Feb 25 23:37:51 BigBoy kernel: nvidia-modeset: WARNING: GPU:0: Unable to read EDID for display device DVI-D-0 Feb 25 23:39:04 BigBoy kernel: mce: Uncorrected hardware memory error in user-access at 135690e640 Feb 25 23:39:04 BigBoy kernel: kernel BUG at arch/x86/kernel/cpu/mce/core.c:1582! Feb 25 23:39:04 BigBoy kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI Feb 25 23:39:04 BigBoy kernel: CPU: 17 PID: 19512 Comm: Job.Worker 8 Tainted: P M O 5.19.17-Unraid #2 Feb 25 23:39:04 BigBoy kernel: Call Trace: Feb 25 23:39:04 BigBoy kernel: mce: [Hardware Error]: Machine check events logged Feb 25 23:39:04 BigBoy kernel: [Hardware Error]: Uncorrected, software restartable error. Feb 25 23:39:04 BigBoy kernel: [Hardware Error]: CPU:17 (19:21:2) MC0_STATUS[-|UE|MiscV|AddrV|-|-|-|-|Poison|-]: 0xbc00080001010135 Feb 25 23:39:04 BigBoy kernel: [Hardware Error]: Error Addr: 0x000000135690e640 Feb 25 23:39:04 BigBoy kernel: [Hardware Error]: IPID: 0x001000b000000000 Feb 25 23:39:04 BigBoy kernel: [Hardware Error]: Load Store Unit Ext. Error Code: 1, An ECC error or L2 poison was detected on a data cache read by a load. Feb 25 23:39:04 BigBoy kernel: [Hardware Error]: cache level: L1, tx: DATA, mem-tx: DRD Feb 25 23:39:13 BigBoy root: Fix Common Problems: Error: Machine Check Events detected on your server Feb 25 23:39:13 BigBoy root: mcelog: ERROR: AMD Processor family 25: mcelog does not support this processor. Please use the edac_mce_amd module instead.

 

bigboy-diagnostics-20230225-2341.zip bigboy-syslog-20230225-2241.zip

Edited February 25, 2023 by AceBoyden

[trurl]

Try booting in SAFE mode

[JorgeB]

Hardware errors are still being logged suggesting a hardware problem.