July 30, 2025Jul 30 Greetings folks. A very normal-seeming drive upgrade is now teetering on a possible catastrophe.At a quick glance, this is how the drives have been restructured, with the 20tb drive being added and the 8tb drive being retired.beforemidnowparity16tb 120tb 120tb 1disk 18tb 18tb 116tb 1disk 212tb 112tb 112tb 1disk 316tb 216tb 216tb 2No other system changes were performed, and prior to this activity ("before") my server had been reliably functional with no issues going back now almost 2 years. The parity rebuild on the new 20tb drive ("mid") was completed happily and with no issues. My old parity drive has been slotted in place of the 8tb retired drive, and that drive's data rebuild ("now") is repeatedly crashing my system at around the 20% completion mark. Obviously, I am very concerned about my data because I have disks spinning without further failure protection.I have run an extended SMART test (attached) on the 16tb drive that I am trying to rebuild. All drives passed short tests. MEMTEST also passed, though I am not sure I consider that exceptionally helpful.As per similar threads, I have attempted to set up flash drive syslog mirroring, and will attach that along with a diagnostics zip. My primary issue and reason I am posting here is that I feel I must not understand how syslogs get generated, because the logs I am attaching do not seem to contain any events near to the time of the crash. I see RAM errors being corrected and an L3 cache warning, both of which I am willing to investigate deeper if someone more knowledgeable here believes that is warranted.For example, "syslog-previous" was generated a little after midnight this morning which is when I believe the crash occurred, and it does not contain any events after 9pm.For this reason (my ignorance) I am attaching a handful of different syslogs and diagnostics. The others are timestamped except for that most recent log and the one currently running on my active system with its STOPPED array.I am mortified to continue troubleshooting since I do not understand why my system is failing. Further drive failures would be catastrophic.Additional supporting info:Motherboard: Asrock Rack X470D4UProcessor: AMD Ryzen 5 1600RAM: Kensington 2x Kingston KSM26ED8/16ME Server Premier - DDR4-16 GB - DIMM 288-pin - 2666 MHz / PC4-21300 - CL19-1.2 V - unbuffered - ECCGPU: GTX 1660 SUPERtower-diagnostics-20250729-2227.zip tower-syslog-previous-20250730-1616.zip syslog syslog-previous tower-diagnostics-20250730-1217.zip tower-smart-20250730-0953.zip Edited July 30, 2025Jul 30 by seredin formatting
July 30, 2025Jul 30 Community Expert Assuming the syslog-previous covers the crash, it looks more like a hardware issue, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.
July 30, 2025Jul 30 Author 5 minutes ago, JorgeB said:Assuming the syslog-previous covers the crash, it looks more like a hardware issue, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.Yes "syslog-previous" should be covering the crash, though the log has no records in it for about 3 hours before the actual crash. When you say "Hardware issue" is there something specific you're seeing to lead you to that conclusion? I see ECC corrections in the log, and I see the L3 cache notification, though cursory searches suggest those aren't immediately lethal. Just trying to learn how to dig into these logs. I'll shut it down and pull a stick now. Any recommendations on what to do on that single stick? Should I just try to attempt another data rebuild (again)?
July 30, 2025Jul 30 Community Expert 32 minutes ago, seredin said:When you say "Hardware issue" is there something specific you're seeing to lead you to that conclusion?The fact that there's nothing relevant logged before the crash and that a parity check/rebuild causes more stress on the system suggests hardware to me.
July 30, 2025Jul 30 Community Expert 33 minutes ago, seredin said:Should I just try to attempt another data rebuild (again)?You can.
July 30, 2025Jul 30 Author I removed a stick and, on a whim, removed the to-be-rebuilt drive from the array and started in maintenance mode, intending to Preclear the disk (again, and after a passed SMART test anyways) because I noticed it had a partition flagged on the drive when viewed in Unassigned Devices.Curiously, Preclear won't run on the disk, instead giving this error:#################################################################################################### # Unraid Server: verifying Preclear State of disk ZL2Q1LVA. # # Verifying disk 'ZL2Q1LVA' for Unraid's Preclear State. # # # # Step 1 of 2 - Verifying Unraid's signature: FAIL # # # # # # # # # # # # # # # #################################################################################################### # Cycle elapsed time: 0:00:01 | Total elapsed time: 0:00:01 # #################################################################################################### #################################################################################################### # S.M.A.R.T. Status (device type: default) # # # # ATTRIBUTE INITIAL STATUS # # Reallocated_Sector_Ct 0 - # # Power_On_Hours 20072 - # # Reported_Uncorrect 0 - # # Airflow_Temperature_Cel 31 - # # Current_Pending_Sector 0 - # # Offline_Uncorrectable 0 - # # UDMA_CRC_Error_Count 0 - # # # # # #################################################################################################### # Report genereated on: July 30, 2025 at 10:59:10 # #################################################################################################### --> RESULT: FAIL! /dev/sdc DOESN'T have a valid Unraid MBR signature!!!Very interesting, considering that this was my former Parity drive, it had been functioning happily for years, was happily Precleared between the "Mid" and "Now" steps above, and subsequently passed a SMART test.I don't think I understand what the MBR signature is or what Unraid was looking for there, but this at least is something to go on. I will start digging into it and hoping that someone responds here as to what this means.Sort of thankfully (but not for my wallet), I got spooked the moment I realized my system was unstable about 5 days ago, and ordered another 20TB drive which will be here as early as Thursday. So if it's a disk issue, I have one on the way, at least...edit: re-running Preclear on this drive. Suppose I'll report back tomorrow.edit2: Preclear is 8 hours in, and no issues. this is twice as long as any data rebuild attempt has survived. Edited July 31, 2025Jul 31 by seredin small update
August 2, 2025Aug 2 Author Not especially scientific of me, since I changed two things at once sort of. But I went in and pulled a stick of RAM and also went through the full Preclear process in the drive that needed rebuilding and it's presently 30% further along than it had gotten before.Not sure if it's the RAM or not, but I'm not seeing the warnings like before so I'll probably leave it out of the system if the rebuild completes.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.