Checked my server this morning (running 6.11.5 pro) when I saw this message:

 

kernel: critical medium error, dev sdm, sector 10920746360 op 0x0:(READ) flags 0x0 phys_seg 27 prio class 0

 

This is part of my parity, one of two 10 TB HGST drives that are mirrored.

SMART shows no errors.

Is this an indicator that the drive may be starting to fail? If so, do I have to replace the pair if I wish to increase the size of the parity drives or can I replace one at a time?

logterminal SDM.txt

Shut down dockers and vms and 'stop' the array. Click on the disk and Run Extended Smart.  Smart is a great tool but won't catch every possible drive failure scenario, even extended.

 

Since you have already stopped the array, might as well take her all the way down and reseat the cables. Shut down and remove power. Reseat:

 

  HDD if using a backplane

  Power cables

  Data cables

  Reseat the HBA in the PCI slot if you have one

 

 

And no, if you keep logging errors or having a problem, you only will need to replace the single drive that is failing.

Please post the diagnostics.
 

elmstorage-diagnostics-20240810-2115.zip

 

Did this get resolved?  If so, how?  While doing a mover action, I got 256 errors show up randomly on my 2nd parity drive of the array.

Quote

kernel: critical medium error, dev sdx, sector 15097745480 op 0x0:(READ) flags 0x0 phys_seg 256 prio class 2

This showed up in the disk log for that drive.   It happens to be a HGST 10TB SAS drive; same as the OP.  

 

Also getting this (or a similar version of this log) repeating a bunch of times throughout the disk log:

Quote

kernel: sd 1:2:15:0: [sdx] tag#2641 BRCM Debug mfi stat 0x2d, data len requested/completed 0x100000/0x0

This is a production server, so i want to make sure all is good.   I'm working remotely so i cant check the connections, backplane etc, till I'm onsite next week.  I will run a full smart test, but i was hoping for a bit more insight if anyone has any!  Thanks.

Edited August 11, 20241 yr by miicar
added diag

On 3/8/2024 at 4:14 AM, JorgeB said:

Please post the diagnostics.
 

 

It's logged as a disk problem, run a long SMART test.

13 hours ago, miicar said:

I will run a full smart test, but i was hoping for a bit more insight if anyone has any!  Thanks.

Just an update:  The server locked up, couldn't access the gui, and I was forced to reboot through the IPMI interface.  At that point it seemed to loose access to the USB boot key.  But after shutting down and restarting a couple times, the UBS drive appeared in bios again and it seems to have boot fine now!  This is a new (spring) Samsung Pro USB key; i did not expect any issues with it already!  I do have an older HBA controller, for the external drive racks, that likes to overheat and crash the system...so I am going to swap that out for something else the next chance I get, and hopefully that will fix the stability issues (not the drive that got the errors; that is managed from the motherboard onboard SAS controller).

 

I have a "BRCM Debug mfi stat 0x2d..." notice in the log of a second HGST drive, in the pool, as well...I hope I don't have 2 failing at once (and thank myself for having dual parity i guess).  

 

Since the USB/reboot mess, I am not getting any errors showing, on those drives, on the main page; and the server seems to be running properly (other then the persistent "mfi stat..." notices on the drives).  

I am running extended tests on the x4 HGST drives in the Array.  I will update this when it is finished.

On 8/11/2024 at 6:07 AM, JorgeB said:

It's logged as a disk problem, run a long SMART test.

So i ran the long test on all 4 of the (identical) array disks (2 parity, 2 data; there are an additional 2 data drives that are a different manufacture/size).  What exactly am I looking for, and where??  I am guessing "non-medium error count" is a tell-tale indication of a drive going bad?  Is there an acceptable number?  Or soon as that hits more then zero, its time to replace?  The 2 disks that i was having these "BRCM Debug mfi stat 0x2d, data len requested/completed 0x100000/0x0" messages in the logs, do have greater then zero (3 & 2) "non-medium error count"; and the other 2 SAS disks are still at zero.

(just a side note/question: I've had drives show "elements in grown defect list" greater then 0, and last years without growing that number or having issues, so I don't think that's a factor here.)

I am considering running a parity check (with corrections) next, but if these disks need to be replaced anyway, would it be better to replace the 2nd parity disk first, then the data disk, instead?  Then run a parity check again after replacing both disks?  Or will the replace/rebuild process be enough of a parity check (and save some wear on the disks)?  Also, I am assuming there is some sort of algorithm, with dual parity, that would ignore errors if one disk (either data or the other parity) returns corrupted data on a rebuild, and gets that data from a trusted source?

Thanks for the help ya'll!  

Edited August 12, 20241 yr by miicar

The main thing is to confirm if the test passed or not, other than that, these are never a good sign:

 

9 minutes ago, miicar said:

"elements in grown defect list" greater then 0

 

1 hour ago, JorgeB said:

The main thing is to confirm if the test passed or not, other than that, these are never a good sign:

 

 

the "growing defect list" hasn't grown at all.
So they all show "passed" in the identity tab; and none of the drives have "uncorrected errors" in the SMART log, so am i getting bent out of shape for nothing?  I am trying to understand the "Non-medium error count".
I have a sneaking suspicion that allot of this has been caused by that old (and horrible) HBA card that i have running the external drive cabinet.  I was doing a move of some local machine backups from a cache pool (that lives on the external cabinet), to the main array.  I think that caused the HBA card to overheat, and i am told by the manufacture, those cards did NOT throttle under heat, they just crashed!  I tried putting a fan blowing on it a while back and that seemed to make it happy, but maybe i pushed it too hard with a network-wide backup.  Time to look for a new HBA i, but all this has me worried about my main array as well.

Edited August 12, 20241 yr by miicar

13 hours ago, miicar said:

So they all show "passed" in the identity tab

Just to be sure, I mean the SMART test, not general SMART status, please post new diags.

15 hours ago, JorgeB said:

Just to be sure, I mean the SMART test, not general SMART status, please post new diags.

Yes.  I ran the long SMART test (took around 18 hours).  So the only place (i can find) that says passed, is the SMART status.  Would that not be updated there if it failed?  Otherwise, at the bottom of the "Self Test" page, it says completed...nothing there says passed or not.

New Diags posted as requested.

 

elmstorage-diagnostics-20240813-1821.zip

It did pass so the disk is OK for now, just wanted to confirm, since users sometimes refer to the SMART overhaul status as OK, and that doesn't means much.

On 8/14/2024 at 3:19 AM, JorgeB said:

It did pass so the disk is OK for now,...

That was my findings too.  I am going to conclude that the 256 "errors" (and the "kernel: critical medium error, dev sdx, sector 15097745480 op 0x0:(READ) flags 0x0 phys_seg 256 prio class 2" error), were due to that really old HBA overheating crashing itself and causing issues.  I think i should run a full (correcting?) parity check.

UPDATE:  as i wrote this, i realized my server was kind of locked up (SMB still worked, but no access to GUI and a ton of ngix errors in the log).  Did a reboot, and it failed to find the boot USB again.  there goes my night!

UPDATE 2: Array passed parity check with no errors.  Still concerned about why, after the "lots of tabs still open" crash and a reboot, my USB won't boot till i power down the server for a min.  But for now things are running fine.  When i get to site and swap out that old HBA, i will stress test the drive cabs again and see if i still get that other crash.   
Thanks for all the help community!

Edited August 18, 20241 yr by miicar
Final update