I was planning on upgrading a HDD this weekend and started a parity check yesterday. When I came home, one of the drives (Drive 10) had errors and had been disabled and had the message "Device is Disabled, Contents Emulated". While doing some troubleshooting, Another Drive (Drive 3) threw some errors as well. I shutdown for the night and started again this morning.

 

First thing I did was check cables. I don't have any spares to replace at the moment, but I unplugged and reseated each cable. I was attempting to get Drive 10 up and running again, so I unassigned the disk, started in Maintenance Mode, stopped, and reassigned the disk. It began the process of rebuilding data when Drive 3 started throwing errors again, but this time the parity drive also had errors. All three disks pass the SMART short self test. Just need to figure out next steps to get back up and running. Diagnostics attached from last night and this morning.

 

SOLUTION: Reseated SATA cables, Controller card, and power cables and everything was up and running.

 

tower-diagnostics-20250410-2048.zip tower-diagnostics-20250411-0927.zip

Edited April 14, 20251 yr by johngalt
.

6 minutes ago, johngalt said:

unplugged and reseated each cable

Both ends? Including power? Any splitters?

Any tension in cables? Don't bundle data cables to make things "neat".

Nothing's bundled, everything is loose. I learned that lesson about a decade ago when I first built the server. 

 

I unplugged both ends. The one from the motherboard/controller and the one from the drive bay. No splitters; straight from motherboard/controller to the drive bay. I didn't do anything with power. 

1 minute ago, johngalt said:

No splitters; straight from motherboard/controller to the drive bay

Power splitters I'm asking about.

Not sure the proper terminology, but I'm using three Norco SS-500 bays. One goes direct to the PSU with nothing else. The other two are daisy chained... one bay to the other bay to the PSU with nothing else.

 

I'm about to check to see if all three of these drives are in the same bay. These are over 12 years old at this point.

3 minutes ago, johngalt said:

three Norco SS-500 bays

Are all of these on this controller?

01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
	Subsystem: Broadcom / LSI SAS9300-8i [1000:30e0]
	Kernel driver in use: mpt3sas
	Kernel modules: mpt3sas

 

Yup. All three drives are in the same Drive Module and all are going to that controller.

 

Replace the controller?

Could also be bad power, don't see anything logged that suggests a controller problem, also not logged as a disk issue in both cases, do you have a spare PSU you could use?

Not without taking apart another computer. Is there any way to test other than trial and error?

 

So everything in diagnostics looks good? Is there a risk of data loss/corruption from the multiple disk errors? Fortunately I wasn't writing anything at the time. I'm going to attempt to unplug and reseat all the SATA and power cables again; do I just need to try to continue with the data rebuild as normal or anything special to do at this point?

Edited April 11, 20251 yr by johngalt

51 minutes ago, johngalt said:

So everything in diagnostics looks good?

No, there are what look like power/connection issues with the disks.

I redid all the SATA cables and did the power cables this time. I'm at a million writes on the disc rebuild without any errors. Assuming this completes without issues, is there anything else I need to do due to the disk errors?

Post new diagnostics

tower-diagnostics-20250411-1324.zip

Looks good so far.

 

You really need to upgrade, and increase capacity.

Once I add the new parity and move the current parity over, it will give me an additional 10tb. I normally upgrade my drives before this point, but it kind of creeped up on me.

 

Will upgrade software once I get the drives settled. 

 

Just noticed my dockers are empty. Is that normal from rebuilding a drive, or did something get deleted in the process?

Edit: Was able to add Plex back using "Previous Apps" but my database was gone. Have to rebuild the media library from scratch.

Edited April 12, 20251 yr by johngalt