My array consists of 8x 4T SSDs (all TS4TSSD230S made by Transcend): 1 parity drive, 7 data drives.

 

It started with one day I noticed that the parity drive (sdb, TS4TSSD230S_H589080035) and 1 data drive ("Disk 1", sdc, TS4TSSD230S_H491820003) are reporting errors. Per SMART:

• Both have reallocated sectors: data drive 288, parity 550
• Both have non zero "Uncorrectable sectors count when read/write" (per Transcend software, Unraid shows # 160 Unknown attribute): data drive 1, parity 958 (!)

Even though the parity drive seems worse off, it's still mounted, while the data drive is shown with a red cross icon in "Disabled. Contents emulated" state.

 

The first thing I tried was these steps, and I believe the net effect is to get the array to restore/recreate the content of this drive. After letting it run through its course overnight, I noticed that the drive was in "Disabled" state again.

 

At this point, I decided to check if these disks are okay, so I moved the 2 drives in question to a separate Windows machine, where I installed Transcend's SSD Scope to perform full diagnostic scans, both passed the scan (which seems very questionable). It's worth noting that the data drive took a couple of hours for a full scan while the parity drive took more than a day.

 

After the above, I plugged the drives back into my Unraid server and powered on. Ever since then, the server is always stuck at "Array Starting • Mounting disks...". The server always boots up in this state, and there's no option to "Stop", or start in maintenance mode. There's also no way to properly reboot or shutdown this server anymore: it seems that all attempts to do so would wind up stuck at something. And 1 thread of the CPU (5600X) is always stuck at 100% utilization

 

In this state, the parity disk is still shown as operational, but the data drive is stuck at "Mounting". So based on this, I unplugged the data drive and started the server again, and this caused "Disk 1" to show up as "Not installed", yet it's still stuck at "Mounting", see below:

Then regardless if I have this drive plugged in or not, the server seems to be stuck at this state.

 

And if that data drive (sdc) is plugged in, it actually shows up as an unassigned drive and it can be mounted and accessed even.

 

My questions are:

1. Is there a way to get the array out of this state?
2. How should I go about rescuing data off the array at this point?

 

Thanks!

 

zeegee-filer-diagnostics-20241012-1745.zip

Diags show errors with parity and disk1, looks more like a power/connection problem, check/replace cables for both disks and post new diags after array start.

@JorgeB thanks! I was suspecting that too and I already tried switching cables, and I think the earlier report was captured with the new ones, but here's another capture.

 

The current setup for those 2 disks is that they are on a separate (vs other drives in the hot swap bays) SATA power cable from the PSU, and each has a new SATA cable connecting them directly to the motherboard (vs other drives in the host swap bays via backplane and then slim SATA cables to the motherboard).

zeegee-filer-diagnostics-20241013-0730.zip

Still errors with parity, if the cables were replaced run an extended SMART test.

The SMART test seems a bit wonky, since for both drives, the test rans for relatively short period of time: first, it's "self-test in progress, xx% complete", then after a couple of minutes, it changes to "No self-tests logged on this disk". For the parity drive, it goes up to "10% complete", and for the data drive, it goes up to "20% complete". I also tried on a 3rd known good drive (all 3 drives are of the same model), and it can run extended SMART test normally.

 

Anyway, I noticed that in command line, if I run "smartctl -a [drive]", I can see some test results in "Selective self-test" section, so I've attached the full output of "smartctl -a" for both drives.

smartctl-all-sdb-parity.txt smartctl-all-sdc-disk1.txt

Some minor updates: I've been trying to get an old (~2009) RocketRAID 622 card to work with unraid, so that I can plug in more SATA drives to start exfiltrate files off the array. I can get the card to show up at boot and also detected in unraid, but I can't get the connected drive to show up.

 

And somehow during this trial and error process, my unraid server settings was changed but I was very sure that I didn't initiate any of those changes. Things I noticed so far: 1) the server name was "zeegee-filer" but now "Tower", 2) the cache drive setting was lost and the drive in question is now listed as unassigned, 3) the ssh access no longer works ("Connection refused") and I don't see sshd running (can still use web UI based command line console).

 

I'm not sure if this is indication of some larger problem (diagnostics attached), or perhaps just some sort of collateral damage as I was trying to get the the rocketraid card to work (but I was mostly fiddling with BIOS options and also updated BIOS).

tower-diagnostics-20241013-1931.zip

I think the immediate cause for SSH not working is because Settings > Management Access > Use SSH is now "No", however:

1. I never changed that setup myself (I guess this is along the same lines as losing the configured server name?)
2. I can't switch it back on: changing it to "Yes", then "Apply", gets a perpetual "Waiting for <server ip> ..." at the bottom of the Chrome browser
   • I've seen this similar behavior when trying to disable the auto start for the Array, so I guess this has something to do with the array stuck at "mounting disks" again.

8 hours ago, ZeeGee said:

The SMART test seems a bit wonky, since for both drives, the test rans for relatively short period of time: first, it's "self-test in progress, xx% complete", then after a couple of minutes, it changes to "No self-tests logged on this disk". For the parity drive, it goes up to "10% complete", and for the data drive, it goes up to "20% complete".

If you tried multiple times and it's the same the drive may be failing.

 

3 hours ago, ZeeGee said:

I think the immediate cause for SSH not working is because Settings > Management Access > Use SSH is now "No", however:

It's disabled by default, but it should enable, you may have a flash drive config issue.

10 hours ago, JorgeB said:

If you tried multiple times and it's the same the drive may be failing.

I'll contact the manufacturer for warranty. In the mean time, what's the right way to get the data off the array drives while I can? It seems I can just mount the block devices directly (e.g. /dev/sdc1), is that a safe thing to do? (I've mounted with read-only option via command line)

 

10 hours ago, JorgeB said:

It's disabled by default, but it should enable, you may have a flash drive config issue.

It's definitely been set to on, plus I've lost my server name for some reason: see earlier diagnostics in this thread have the file name prefix "zeegee-filer" while the more recent ones are "tower".

 

The thing is, when I try to set SSH to on, the setting never properly applies, and also the option to change server name is grayed out with a prompt that array must be stopped first. I also tried to disable the array from auto-starting, but again the setting change fails to properly apply. Should I just change /boot/configuration/disks.cfg manually?

 

Post new diags to see if there any any visible flash drive issues.

 

Regarding the disks, you can try to mount the actual disk1 with the UD plugin, if it's readable you can then copy the data, or do a new config to re-enable it in the array.

6 minutes ago, JorgeB said:

Post new diags to see if there any any visible flash drive issues.

Please find the latest diagnostics attached.

tower-diagnostics-20241014-1123.zip

Not seeing anything that suggests a flash drive problem, only the old parity errors, but you can try recreating the flash drive anyway.