I just swapped out an 18TB for a new 28TB and data rebuild started. I guess I failed to notice the messages about unmountable drives. It seems as though the data rebuild is reading from these drives anyway. See image. 
 

can anyone offer suggestions as to what the heck is going on and what I should do?!?!

8 minutes ago, commander-flatus said:

swapped out an 18TB

Don't do anything with that original disk. It might be needed to recover files.

 

Attach Diagnostics to your NEXT post in this thread.

Sorry, I should've thought to attach to the first message. Thank you for your help. I've done a zillion drive swap. This system started out with a few 6TB disks in 2013, I believe. I've never had this happen. There's obviously been other hardware upgrades along the way.

tun-diagnostics-20250318-2030.zip

syslog has a lot of entries where different disks (even both parity) were missing and then later assigned. Can you explain what that was about?

1 minute ago, trurl said:

syslog has a lot of entries where different disks (even both parity) were missing and then later assigned. Can you explain what that was about?

Array was stopped. I pulled out my supermicro trays and put them back in to find the disk that I wanted to swap out.

See if scrub will fix disk2 then we can decide how to proceed with the other unmountable disks.

should i stop the rebuild?

You are currently rebuilding an unmountable filesystem, so might as well stop it.

well i can't scrub disk 2 b/c it's not mounted....

OK, lets try the 2 encrypted xfs disks.

Check filesystem on disk5 from the webUI. Capture the output and post it.

Ok, I stopped the rebuild, powered down the server, made sure that all the drive trays were firmly seated.

When I powered back up all disks mounted. I’m starting to wonder if my SAS backplane or my SAS PCI card need replacement - the card is going on 10 years old - it’s from an old Dell R710 and I have a spare on hand anyway.

And thank you for the offer to help and the prompt assistance.

1 hour ago, commander-flatus said:

Array was stopped. I pulled out my supermicro trays and put them back in to find the disk that I wanted to swap out.

I don't recommend hotswapping assigned disks.

12 minutes ago, commander-flatus said:

I stopped the rebuild, powered down the server, made sure that all the drive trays were firmly seated.

When I powered back up all disks mounted.

Is it rebuilding the new disk now?

Yes. Appears to be working fine. I’ll never hot swap again. 

I used to find disks that way, but stopped after a similar incident of disks dropping out unexpectedly just after mounting, just dirty connectors and a confused controller thankfully. I did pull the backplane out at one point and hit every connector with contact cleaner spray just in case, some of them were surprisingly dusty, obviously empty for years.

 

Now I have a map of my disks and which bays they are in, but also when the array is offline I spun all disks down, then spin up just the disk I want to pull and look at the activity LEDs on my enclosure to confirm before yanking the disk out.

 

I also never pull a disk with the array started, even if it's not part of the system at all, just in case the bending/flexing of the backplane causes an active disk to have issues.