I finished converting all of my 14 data drives to XFS, which was a 2 week endeavor that tested my patience many times...thanks to all of you who helped me at various stages!

When all was done, I thought that it may be a good idea to re-check parity, which showed as valid but hadn't been checked in months. In the process one of the 3TB drives was off-lined with errors. I must have grabbed a replacement from my shelf that wasn't one of the "good" ones, installed it in exchange of the bad one but didn't check it or pre-clear it in my unRAID array before hitting the Start Array button. Yes, I know, not good practice.

The array almost immediately complained about the replacement drive and off-lined it but started to check parity which, to me at that moment, didn't seem like something that was a good idea with one drive being off-lined, so I immediately clicked the "Stop" (or maybe it is "Cancel") button on the parity check. Probably a bad idea, but that's what happened.

 

So I now have an array that says "Started" with disk 3 offline, "Parity is valid" but also: 

Last check incomplete on Sat 30 Sep 2017 12:40:58 AM CEST (today), finding 0 errors.  Error code: user abort Because I have a feeling that I did something wrong, I'd rather address the unRAID community before doing anything else, and ask you this: I don't have a fresh 3TB drive, and quite frankly don't have the money to buy another drive at this time. What I do have is an 8TB drive which I purchased a few weeks ago in anticipation of replacing my 6TB parity drive with this 8TB drive.    Is there a safe way for me to do this *without* first fixing the off-lined drive with a fresh drive? Maybe I can do a New Config after removing the bad 3TB drive and do a "Trust Parity", or can Parity not be trusted anymore after what I did? It seems obvious to me that I can't replace the parity drive before I haven't resolved the bad drive in the array, but what is the best way of doing so in light of my goal to end up with an 8TB parity drive and the former 6TB parity drive available to either replace the bad 3TB drive, or - if a New Config was done first - to *add* the former 6TB parity drive to the array. I hope this question wasn't as confused as I feel at the moment. What should be my next steps? I'm not touching anything until I get some guidance from members more knowledgable than I am.

 

I should add that I emptied the bad drive overnight, using the excellent unBALANCE plugin, and have over 18TB of free space on my 36TB array.

First you should always grab your diagnostics before rebooting when there's an issue and post them, the problem may not be the disk, more often than not it's a connection issue.

 

43 minutes ago, tillkrueger said:

I don't have a fresh 3TB drive, and quite frankly don't have the money to buy another drive at this time. What I do have is an 8TB drive which I purchased a few weeks ago in anticipation of replacing my 6TB parity drive with this 8TB drive. 

 

Is there a safe way for me to do this *without* first fixing the off-lined drive with a fresh drive?

 

Yes, thats what the parity swap procedure is for:

 

https://wiki.lime-technology.com/The_parity_swap_procedure

 

 

thanks jb, attached are my most recent diagnostics zipped up.

I posted them in a. different context a couple of weeks ago, and what immediately threw up some red flags was the fact that I have one of those Marvell-based drive controllers in my system...so you may be correct in alluding to the fact that it may not be a bad drive after all.

I'll try to wrap my head around this parity swap procedure now, butI'd feel more comfortable if that potentially bad drive was removed from the array before I do anything else. I won't be able to physically access this server until Monday, anyway, but would like to have a solid plan when I go there.

Is there any danger in stopping the array, un-assigning drive 3, doing a New Config and starting the array again (with "Trust Parity" selected, if that is when I'd be presented with that option)?

unraid-diagnostics-20170930-1104.zip

17 minutes ago, tillkrueger said:

Is there any danger in stopping the array, un-assigning drive 3, doing a New Config and starting the array again (with "Trust Parity" selected, if that is when I'd be presented with that option)?

 

Parity won't be valid anymore, you can do that but you'll lose all data on disk3 and need to re-sync parity.

Also, disk3 dropped offline so there's no SMART report, but it does look like the typical SASLP issue, you can reboot to see if it comes online and grab new diags so there's a SMART report.

ok, then I'll start with that: reboot, and if disk3 comes back online, post another diag report.

if it does not come back online, I guess it will be best if I wait until Monday and do the 3 drive swap procedure outlined in the wiki article you kindly linked to.

after a reboot the Fix Common Problems plugin gave me this alert, but disk3 still shows a red X (the array is now stopped)...new diag attached:
 

unRAID array errors: 30-09-2017 12:05

Notice [UNRAID] - array turned good
Array has 0 disks with read errors

unraid-diagnostics-20170930-1208.zip

29 minutes ago, tillkrueger said:

but disk3 still shows a red X

 

That's normal, you'll need to rebuild to re-enable the disk.

 

Disk is online and looks healthy, so most likely it was the SASLP, you may try rebuild using the same disk, but until you get rid of those controllers same thing will likely happen again in future to the same or a different disk.

 

https://wiki.lime-technology.com/Troubleshooting#Re-enable_the_drive

Ok, I followed the steps outlined in the wiki article you linked to and it is now rebuilding the formerly empty drive.

Would you be able to tell me, from the diagnostics file (or another file I could generate and post) whether the Marvel Controller in question is part of my motherboard or the 8-port expansion card I added to be able to use 15 drives? it's too long ago for me to remember exactly what I chose, when I built this server...I kinda hope that it's the controller card I added, as it would be a lot easier to swap out this controller for a good one (any pointers as to which one to replace it with?), compared to having to use a different motherboard, obviously.

Here's to hoping.

Looking at the System Profiler I can see that the mobo is a Supermicro - C2SEA, but I do not see any place where it tells me what drive controller cards are in the system.

thanks for all your help and patience with this, jb!

You have two Supermicro SASLP controllers, one with 7 disks connected, the other with 8, i.e., all your disks.

Haha, that’s exactly what I didn’t want to hear, but I guess that’s the situation i’ll now have to deal with: a complete system rebuild.

 

oh boy.

 

but thanks for breaking it to me straight, jb!

 

i guess the only good news in this is that many of the shelved bad disks of the past many years may not be bad at all...I was wondering why all those WD Green and Red drives were so unreliable, when they actually got pretty good reviews across the board...even replaced all of my drive cages 2 years ago bc uncanny failures...but I guess this all makes more sense now.

although...I got this error message while I went for a walk:

 

unRAID Disk 3 error: 30-09-2017 14:47

Alert [UNRAID] - Disk 3 in error state (disk dsbl)

WDC_WD30EFRX-68AX9N0_WD-WMC1T0098667 (sdf)

does that mean that disk3 may simply be bad, potential controller issues and all?
as I described earlier, I may have grabbed a replacement drive from my batch of shelved bad drives, not sure.

 

to me it looks like I might have to wait until Monday to start the "3 drive swap" method you linked to earlier, to replace my parity drive with my fresh 8TB drive and free the old 6TB parity up to replace this potentially bad 3TB drive.

50 minutes ago, johnnie.black said:

You have two Supermicro SASLP controllers, one with 7 disks connected, the other with 8, i.e., all your disks.

That's actually good news - those are add-on cards, not built in to the MB, so swapping them out should be easy. Price shouldn't be bad either - it seems suitable replacements go for about the price of a 4TB drive these days.

7 minutes ago, tillkrueger said:

does that mean that disk3 may simply be bad, potential controller issues and all?

 

It may be a bad disk despite the healthy SMART, you can replace/swap the cables connect it to the onboard controller and try to rebuild one more time, if it fails again replace it.

oh, I thought you meant that I had 7 on-board ports and 8 expansion ports, all connected and using the Supermicro SASLP controllers, but you're saying that *all* of my drives are connected to add-on controllers, both of those of the Supermicro SASLP variety, got it.

well, that is good news indeed...so I'll be searching for PCI (or some other variety PCI?) SATA cards with at least 8 ports, that are *not* Supermicro SASLP controllers? any controller that's recommended (or that you would recommend)? I know I could dive into the hardware forum, but since you know your thing, a pragmatic recommendation would save me some time, to be blunt.

LSI 9211 8i (IT MODE)

Edited September 30, 20178 yr by MowMdown

Interesting, MowDdown...so, doing a search for your recommendation I found this tutorial on converting your recommended LSI 9211-8i controller. I'm in Berlin, so I would have to get it from Amazon.de or eBay.de, if not a local supplier.

 

any particular reason why this solution is recommendable over an over the counter solution that doesn't require a modification such as this one?
I mean, I routinely flash my routers with the dd-wrt firmware, so I know a little bit about the potential benefits of modifying factory configurations/firmware, but I'd be interested in what makes the IT mode advantageous/necessary. (a link to read up on this particular topic would be fine)

 

IT mode provides direct access to disks, it's the preferred mode for unRAID, it's a very simple flash, you can also buy the 9201-8i or the 9207-8i, both are IT mode only, but they are usually more expensive, specially used on ebay.

you're right, jb...the used 9201 cards are considerably more expensive on eBay.de than the 9211 cards...the 9211's can be ordered from the US or China for about $65, the Dell PERC H200 8Port 6Gb/s Adapter SAS SATA Controller Card (LSI 9211-8I) for only $47. I may order a couple of the 9211's from the USA, which might take as much as 2-3 weeks, or pay a little more to order from within Germany or Europe and get them quicker.

I would I need *two* of these, right? or would one of them suffice with the right cabling?

You can use the 6 onboard ports, still going to need one more port besides the 8 ports on the LSI, but if you don't plan on going over 16 disks the 2nd controller can be a cheaper 2 port controller, e.g., Asmedia 1601.

I don't seem to find a source for the Asmedia 1601 within Europe, and what I find internationally appears to refer to some sort of USB3 card.

Be that as it may, if I decide *not* to replace that stubborn disk3, reducing my array from 38TB to 35TB with only 19.4TB used, I would only use 14 disks and could fly with that for a bit...and after swapping the 6TB parity for the 8TB fresh drive I have on-hand, I could use the 6TB to replace 3 of the old 2TB drives, allowing a number of scenarios, such as two parity drives, or two cache drives, or whatever else unRAID allows...when I do the hardware rebuild, which has been long overdue, it would be nice to be down to 8 or less drives already, allowing for a smaller case and the all those new-fangled niceties like VM's with direct hardware access and such, maybe?

3 minutes ago, tillkrueger said:

I don't seem to find a source for the Asmedia 1601 within Europe, and what I find internationally appears to refer to some sort of USB3 card.

 

Search fro PEX40039 on ebay or amazon.

 

4 minutes ago, tillkrueger said:

Be that as it may, if I decide *not* to replace that stubborn disk3, reducing my array from 38TB to 35TB with only 19.4TB used, I would only use 14 disks and could fly with that for a bit...and after swapping the 6TB parity for the 8TB fresh drive I have on-hand, I could use the 6TB to replace 3 of the old 2TB drives, allowing a number of scenarios, such as two parity drives, or two cache drives, or whatever else unRAID allows...when I do the hardware rebuild, which has been long overdue, it would be nice to be down to 8 or less drives already, allowing for a smaller case and the all those new-fangled niceties like VM's with direct hardware access and such, maybe?

 

Also a good option.

whoa, how would a $20 2-port controller get me to where I'll need to be after taking out my 8-port controller?