Vonslappy

@JorgeB That was very helpful. DIMM D1 was the culprit. Seems to be operating without MCE events now. Thank you again.

Thank you. I'll get to work figuring out how to read the SEL. Much appreciated.

Hi, Started getting MCE event errors this week while I was out. Looks like 12 of them kicked off in the past few days. It looks as if this is a memory error, but I'd like to ask the more educated minds here for assistance diagnosing the issue. Syslog events below: May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 0 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e1ff2e7406c May 9 04:40:07 bender root: MISC 200800c030001086 ADDR 7fb6fb00 May 9 04:40:07 bender root: TIME 1652062226 Sun May 8 19:10:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS dc004c0001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 1 May 9 04:40:07 bender root: CPU 0 BANK 18 TSC 95e1ff2e7406c May 9 04:40:07 bender root: MISC 900008888000086 ADDR 6a534780 May 9 04:40:07 bender root: TIME 1652062226 Sun May 8 19:10:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER MS_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory scrubbing error May 9 04:40:07 bender root: MemCtrl: Corrected patrol scrub error May 9 04:40:07 bender root: STATUS cc000680000800c2 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 2 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e2002d06c74 May 9 04:40:07 bender root: MISC 200800c017e01086 ADDR 14eb4cf740 May 9 04:40:07 bender root: TIME 1652062226 Sun May 8 19:10:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS 9c00004001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 3 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e2c3f56e9e0 May 9 04:40:07 bender root: MISC 200800c017e01086 ADDR 17eea271c0 May 9 04:40:07 bender root: TIME 1652062255 Sun May 8 19:10:55 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS 9c00004001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 4 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e2c3fa144bc May 9 04:40:07 bender root: MISC 200800c037e01086 ADDR 17ee44cf80 May 9 04:40:07 bender root: TIME 1652062255 Sun May 8 19:10:55 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS 9c00004001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 5 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e9d369d158c May 9 04:40:07 bender root: MISC 200800c037e01086 ADDR 7fbc8f00 May 9 04:40:07 bender root: TIME 1652062525 Sun May 8 19:15:25 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS dc002d4001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 6 May 9 04:40:07 bender root: CPU 0 BANK 18 TSC 95e9d369d158c May 9 04:40:07 bender root: MISC 900088800880086 ADDR 7fa8b5c0 May 9 04:40:07 bender root: TIME 1652062525 Sun May 8 19:15:25 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER MS_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory scrubbing error May 9 04:40:07 bender root: MemCtrl: Corrected patrol scrub error May 9 04:40:07 bender root: STATUS cc000100000800c2 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 7 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e9d4fcc7a4c May 9 04:40:07 bender root: MISC 200800c010001086 ADDR 7faef7c0 May 9 04:40:07 bender root: TIME 1652062525 Sun May 8 19:15:25 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS dc0000c001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 8 May 9 04:40:07 bender root: CPU 0 BANK 18 TSC 95e9d4fcc7a4c May 9 04:40:07 bender root: MISC 908480088008086 ADDR 7facf900 May 9 04:40:07 bender root: TIME 1652062525 Sun May 8 19:15:25 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER MS_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory scrubbing error May 9 04:40:07 bender root: MemCtrl: Corrected patrol scrub error May 9 04:40:07 bender root: STATUS 8c000040000800c2 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 9 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e9da53ca5c0 May 9 04:40:07 bender root: MISC 200800c037e01086 ADDR 7fbb93c0 May 9 04:40:07 bender root: TIME 1652062526 Sun May 8 19:15:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS 9c00004001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 10 May 9 04:40:07 bender root: CPU 0 BANK 18 TSC 95e9da53ca5c0 May 9 04:40:07 bender root: MISC 908408080808086 ADDR 7fbb9380 May 9 04:40:07 bender root: TIME 1652062526 Sun May 8 19:15:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER MS_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory scrubbing error May 9 04:40:07 bender root: MemCtrl: Corrected patrol scrub error May 9 04:40:07 bender root: STATUS 8c000040000800c2 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 11 May 9 04:40:07 bender root: CPU 0 BANK 8 TSC 95e9da67b0bf8 May 9 04:40:07 bender root: MISC 200800c030001086 ADDR 7fbf6b40 May 9 04:40:07 bender root: TIME 1652062526 Sun May 8 19:15:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: Error enabled May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory read error May 9 04:40:07 bender root: M2M: MscodDataRdErr May 9 04:40:07 bender root: STATUS dc00008001010092 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: Hardware event. This is not a software error. May 9 04:40:07 bender root: MCE 12 May 9 04:40:07 bender root: CPU 0 BANK 18 TSC 95e9da67b0bf8 May 9 04:40:07 bender root: MISC 908488008080086 ADDR 7fbbc3c0 May 9 04:40:07 bender root: TIME 1652062526 Sun May 8 19:15:26 2022 May 9 04:40:07 bender root: MCG status: May 9 04:40:07 bender root: MCi status: May 9 04:40:07 bender root: Error overflow May 9 04:40:07 bender root: Corrected error May 9 04:40:07 bender root: MCi_MISC register valid May 9 04:40:07 bender root: MCi_ADDR register valid May 9 04:40:07 bender root: MCA: MEMORY CONTROLLER MS_CHANNEL2_ERR May 9 04:40:07 bender root: Transaction: Memory scrubbing error May 9 04:40:07 bender root: MemCtrl: Corrected patrol scrub error May 9 04:40:07 bender root: STATUS cc000080000800c2 MCGSTATUS 0 May 9 04:40:07 bender root: MCGCAP f000c14 APICID 0 SOCKETID 0 May 9 04:40:07 bender root: PPIN 9f191297c7e1c70f May 9 04:40:07 bender root: MICROCODE 2006a0a May 9 04:40:07 bender root: CPUID Vendor Intel Family 6 Model 85 May 9 04:40:07 bender root: mcelog: warning: 8 bytes ignored in each record May 9 04:40:07 bender root: mcelog: consider an update May 10 00:00:01 bender Plugin Auto Update: Checking for available plugin updates May 10 00:00:03 bender Plugin Auto Update: community.applications.plg version 2022.05.08 does not meet age requirements to update May 10 00:00:04 bender Plugin Auto Update: Checking for language updates May 10 00:00:04 bender Plugin Auto Update: Community Applications Plugin Auto Update finished May 10 00:00:58 bender root: /var/lib/docker: 22 GiB (23588225024 bytes) trimmed on /dev/loop2 May 10 00:00:58 bender root: /mnt/cache: 80.8 GiB (86716022784 bytes) trimmed on /dev/sdc1 May 10 03:40:16 bender crond[2493]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null May 10 04:40:01 bender root: Fix Common Problems Version 2022.04.14 May 10 04:40:01 bender root: Fix Common Problems: Warning: Docker Application jellyfin has an update available for it May 10 04:40:06 bender root: Fix Common Problems: Error: Machine Check Events detected on your server May 10 04:40:06 bender root: mcelog: warning: 8 bytes ignored in each record May 10 04:40:06 bender root: mcelog: consider an update

Makes sense. I disabled docker, and ran the appdata cleanup plugin. All reported well. After reboot: all deleted appdata files are back. New diags attached. Thanks for your assistance. bender-diagnostics-20210720-2000.zip

Hi everyone. I had to do a bunch of work to resolve some physical damage to my server (i dropped it. Butterfingers), and while i now have it back up and running, I'm starting to see some crazy. 1. /var/log is at 100 percent after trying to clean up appdata for unused dockers 2. every time I touch the old plex appdata directory, dockers croak, and things start going read-only. I suspect the log capacity is directly related to the plex issue, but I can't seem to resolve that one. As soon as I try to drop the plex directories, they come back and everything hangs. Any thoughts? What have I wrecked this time? Thanks for your help. Diags pre-reboot attached. bender-diagnostics-20210720-1624.zip

An update. Finally got everything cobbled back together, and have discovered that one of the data drives that I thought was good is not. Unraid hangs at boot until I disconnect it. So, the "busted" list is: 1 parity drive 2 data drives I still have the original parity drive and 2 data drives behaving properly. Do you think I have any hope of getting everything back with this many drives out of commission?

Oh, sorry -- Parity drive is parity information. I overused the word data. The 12TB has only been used as a parity drive. That's good news. Thank you. I will report back when the new case arrives and is built out. Wish me luck.

I made a gigantic mistake last week by tripping and dropping my Unraid server while moving. Two drives had SATA and power ports break in the process (lots of other stuff did as well, but I care about the data). The remaining pins are attached but bent pretty badly. I'm assuming I won't be able to rig these things so they'll power up and be readable again. They may, but I'm going to assume the worst. The two drives included my brand new 12tb second parity drive, and a data drive I had added to the array, but limited it to surveillance camera storage only. I'm pretty sure I care about the data on the 12TB Parity drive, but I couldn't care less about the surveillance footage on the data drive. I DO have a primary parity drive, and as of now, it is as large as the largest disc on the system. I'm assuming it will still be good. So, I have a few parts incoming, and I'm planning to rebuild. IF I start the array up missing a data drive and a parity drive, is it reasonable to assume I can remove the two damaged drives, rebuild parity, and move on? And yes, I feel like a first-class knucklehead for dropping a perfectly good server.

Aw crap. Just saw this. If I re-disable docker, and root around, will a re-enable bring things back to order? If not, it seems fairly trivial to delete and recreate the docker.img, so I can do that as well. Thank you.

Thanks, again, Jorge. Everything seems to be running better now.

Shoot. Got it. New diags posted here, taken after starting array. Thanks for investing the time in this. V bender-diagnostics-20210328-0958.zip

New diags posted here. Update: After rebooting the error seems to have cleared. Naturally, I don't trust this to fix itself, but it's interesting. Also, flash drive has now been replaced, if that matters. Thanks. bender-diagnostics-20210327-0742.zip

I would agree. Thanks, Jorge. I'd have done a lot of damage trying to resolve this without your assistance. New diags posted. bender-diagnostics-20210326-1156.zip

Crud. It's back. "Unable to write to cacheDrive mounted read-only or completely full." Is it time for a scrub?

Up and running again. Thank you @JorgeB