I've never seen a problem like this one before. This morning I noticed some 'ticking' coming from my unraid system, when none of the disks should have been spinning. The main menu confirmed nothing spinning, But it really sounded like a disk. So I checked the log and found a lot of instances of this:

sd 11:0:79:0: Power-on or device reset occurred

[287068.089764] end_device-11:79: add: handle(0x0021), sas_addr(0x300062b2059b99d1)

[287068.093952] sd 11:0:79:0: [sdj] 42970644480 512-byte logical blocks: (22.0 TB/20.0 TiB)

[287068.093955] sd 11:0:79:0: [sdj] 4096-byte physical blocks

[287068.102641] sd 11:0:79:0: [sdj] Write Protect is off

[287068.102649] sd 11:0:79:0: [sdj] Mode Sense: 9b 00 10 08

[287068.112922] sd 11:0:79:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA

[287068.213670] sdj: sdj1

[287068.213807] sd 11:0:79:0: [sdj] Attached SCSI disk

[287076.845392] sd 11:0:79:0: device_block, handle(0x0021)

[287079.344353] sd 11:0:79:0: device_unblock and setting to running, handle(0x0021)

[287079.367803] sd 11:0:79:0: [sdj] Synchronizing SCSI cache

[287079.367852] sd 11:0:79:0: [sdj] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK [287079.368197] mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b2059b99d1) [287079.368201] mpt3sas_cm0: removing handle(0x0021), sas_addr(0x300062b2059b99d1)

[287079.368203] mpt3sas_cm0: enclosure logical id(0x500062b2059b99c0), slot(10)

[287079.368206] mpt3sas_cm0: enclosure level(0x0000), connector name( )

[302907.901584] scsi 11:0:80:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)

[302907.901586] scsi 11:0:80:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)

[302907.988422] sd 11:0:80:0: Attached scsi generic sg9 type 0

[302907.988464] end_device-11:80: add: handle(0x0021), sas_addr(0x300062b2059b99d1)

[302907.988468] sd 11:0:80:0: Power-on or device reset occurred

So I conclude what's happening is that an HBA (I have 2) is losing comms with a drive, the port and drive go away, the HBA finds it again and the drive reinitializes - rinse and repeat. Now, at this time unraid showed everything including sdj with a green ball. I ran a short SMART report on the drive which passed with flying colors. So I then started wondering about cabling. I didn't find anything. I stopped the array and got out my stethoscope thinking I would confirm that disk 9 - sdj - was the source of the clicking and try pulling and restoring power to it.

After I pressed the stop array button, the ticking rapidly increased. By the time the array stopped, Unraid had given disk 9/sdj the red X. I used the stethoscope and confirmed that that disk was indeed the source of the ticking.

So now I am getting concerned. I am far from convinced the drive has really failed; I decide to power-cycle the whole system and restart the array.

This is where it gets weird. When the system comes back up, of course the array isn't started. But disk 9 isn't sdj anymore! I happen to have another disk of the same make and model as sdj in the system, it's been precleared but has never added to the array. But now disk 9 is that empty disk - sdr - and Unraid has it red x-ed out. And on the array operatrions tab, Unraid does not have a message that starting the array will trigger a parity check or that data will be overriden - it seems willing to start the array with this completely wrong configuration.

Back on the devices tab, I change the configuration to the way it is supposed to be, with sdj as disk 9 instead of the empty space sdr. This triggers the message that starting the array like this will overwrite the disk. Which, if parity is good, would I hope overwrite the disk with the data that is already on it, but I don't want to take the risk yet. (Besides, it would take two days to overwrite the 22T drive).

What I really want to do is to but sdj back on the list, get Unraid to trust that array, and continue diagnostics. I know there used to be a "trust my array" procedure that I believe involved the "New Config" button, but for some reason I can't find it.

So this is where I appeal for help. What is my best way ahead here? Should I do a "trust my array"? And if so, can you point me at that procedure because I can't find it. Or is another way ahead better? I really don't want to overwrite the disk because I think the disk is likely good. And while I love having the protection of the parity disks, I hate using them because you can neveer 100% absolutely trust them. (That's why we do periodic parity checks, after all). If I end up having to do that, I'd rather have it write to my empty disk. As I said though, Unraid doesn't seem to think it needs to overwrite the empty disk if I start with that one.

Diagnostics are attached. Help!

tower-diagnostics-20260223-1315.zip

Edited February 23Feb 23 by jhyler

We don't usually talk about the sdX identifiers except in limited circumstances, because they have never been constant. Unraid identifies a disk by its serial number. Adding, removing, changing ports, disks disconnecting then reconnecting, even reboots, can cause the sdX identifiers to change.

The sdX identifiers often do not change because they are assigned as the hardware comes up during boot, and often that sequence is repeatable.

But Unraid doesn't use that to assign disks, and we only talk about those when we need to do something at the command line, and only after confirming that we are discussing the correct disk.

Examining diagnostics now...

FYI - my 22 TB drive which was ticking but which has a lot of data on it has the serial number ending in GPE. The Empty disk's serial ends in MXN.

Can't tell anything about your filesystems until you start the array in normal (not maintenance) mode. Since you already assigned a new disk, unassign it and leave disk9 with nothing assigned to it. Then start the array and post new diagnostics.

New diagnostics after restarting with no disk9 are attached. Thanks for helping!

new-tower-diagnostics-20260223-1337.zip

SMART for both disks looks OK. Neither have had extended self-test.

Since you rebooted before getting those earlier diagnostics can't see why the disk was disabled. Probably a bad data connection or power to the disk.

Emulated disk9 is 97% full so should be fine for rebuilding on top. Or even safer to rebuild to a new disk and keep the original as it is in case of problems.

Clicking can also be a sign of a power problem not a disk problem. How are your disks connected to power? Do you have a backplane? Power splitters?

Ideally, no more than 4 disks per PSU cable, MOLEX-SATA power splitters are better than SATA-SATA power splitters.

I have to leave for an hour or so. I will check back when I return.

No backplane - the rackmount I used to have made too much noise. I now have a Fractal Design 7 XL case which some have gotten 18 disks into. I've only managed 16 plus an SSD. I think it's a 850W power supply, way overkill for normal Unraid use, but useful sometimes. The SATA cables allow for 5 connections per cable but distance from the power supply doesn't let me usually use more than 4 of them for disks. I do attach an extention to that fifth sata connection sometimes, but only for fans. A couple of the drives are powered off of a Molex cable using Sata converters.

You didn't mention if "trust my array" is still a thing, and if so is it advisable. Also, I'm not sure how to get Unraid to restore the data from my GPE disk to the empty MXN disk. Usually you replace a disk with another and it just happens. In this case Unraid seems to think MXN has been there all along. Edit: I've checked again, now Unraid promises (threatens?) to rebuild the disk if I assign either GPE or MXN to slot 9. So I plan to pull GPE and put it away for safekeeping for the next couple of days while (I hope) Unraid recreates its contents on MXN.

I appreciate you looking at this for me.

Edited February 23Feb 23 by jhyler

You can New Config and rebuild parity instead of rebuilding the data disk, but technically, it is the data disk that is out-of-sync. When a write to the disk failed because it had disconnected, that failed write and any subsequent writes to that disk were emulated by updating parity but were not written to the disk since it was disabled. So the data disk is missing some writes that can be recovered by rebuilding it.

1 hour ago, jhyler said:

So I plan to pull GPE and put it away for safekeeping for the next couple of days while (I hope) Unraid recreates its contents on MXN.

Good plan.

And rebuilding to a new disk will allow you to see how well your hardware is working.