Slightly starting to panic, disc errors

[Jeff in Indy]

After a small problem with one of my disks, I followed the suggestions listed there.  I then started a Parity Check.  It is taking much longer than usual (usually takes anywhere from 1-11 days, it's been 4 days and it's at 5.7%, and has an estimated 55 days remaining.  It's reporting 853,000+ Sync errors detected.

 

Today I started Plex to watch a show, and it couldn't find any of the shows I chose.  Looking at the share in Windows, it only showed 2 TV show folders in that share.  Perusing the drives using Krusader, I can see the TV folder in several drives, and see many series in them.

 

However, when I look at the DISKS folder, disk1 is not showing as a folder, it is showing as an unknown file type with 0 bytes.

 

How should I proceed?

[Jeff in Indy]

kaokulk-diagnostics-20220503-1829.zip

[JorgeB]

Looks like a connection problem with disk1, check/replace cables and post new diags.

[Jeff in Indy]

OK, I'll try that later today.  One more thing I noticed, in case it makes a difference..

 

It is showing as normal on the Main tab.  When I click on the folder icon on the right to see the files, it says:

 

No listing: Too many files

 

I'm worried that means a corrupted file system.

[Jeff in Indy]

So I swapped the data cables for Disc 1 and 2 (sde and sdg, respectively) and all seems to be working well.  The files are appearing as normal, and I started a new parity check.  Here is the new log.

 

Let me know if you think something is amiss.

kaokulk-diagnostics-20220504-1949.zip

Edited May 4, 2022 by Jeff in Indy

[JorgeB]

Still issues with disk1, also swap/replace power cable.

[Jeff in Indy]

OK, the same two drives have had their power cables swapped.  

 

Diag attached.

 

Interesting thing: during the parity check, 3 drives are doing an awful lot of writing:  the 2 parity drives, and for some reason, disc 4 (now sdj).

kaokulk-diagnostics-20220505-0722.zip

[JorgeB]

11 minutes ago, Jeff in Indy said:

OK, the same two drives have had their power cables swapped.  

Looks clean so far.

 

11 minutes ago, Jeff in Indy said:

Interesting thing: during the parity check, 3 drives are doing an awful lot of writing:  the 2 parity drives, and for some reason, disc 4 (now sdj).

Something is writing to disk4

[Jeff in Indy]

OK, it's been almost 10 hours.  The Parity was zooming along, now it has slowed down quite a bit.

 

kaokulk-diagnostics-20220505-1712.zip

[Jeff in Indy]

9 hours ago, JorgeB said:

Something is writing to disk4

Well, that I figured.  I find it interesting that the byte count is almost identical to the Parity drives, as if it's completely re-writing the disc.

 

[itimpi]

53 minutes ago, Jeff in Indy said:

I find it interesting that the byte count is almost identical to the Parity drives, as if it's completely re-writing the disc.

That would be expected as every sector written to an array data can be expected to generate corresponding sector writes to each parity drive.

 

[Jeff in Indy]

18 minutes ago, itimpi said:

That would be expected as every sector written to an array data can be expected to generate corresponding sector writes to each parity drive.

 

Duh.  Of course.  My brain is thinking there has been something wrong with the parity, and that it's creating a new parity.  Thanks for pointing out the obvious that I didn't see...

[JorgeB]

Issues with disk2 now, so possibly the problem are the cables that were swapped there from disk1, you should replace them.

[Jeff in Indy]

Replaced the data cable, need to figure out how to replace the other one.

kaokulk-diagnostics-20220506-1055.zip

[Jeff in Indy]

I swapped out the data cables a couple of days ago.  Can you give it one more lookover?

 

And yes, I realize that Disc 8 now has a problem.  I returned from a trip to find that it has died.  I'll be replacing it soon.  Hopefully that's the only one.

kaokulk-diagnostics-20220508-1657.zip

[JorgeB]

Check filesystem on disk1, ATA errors on disks 2 and 8, disk8 issue doesn't look like a disk problem, could be power/connection related or the controller since it's using a SATA port multiplier and those are a known issue.

 

[Jeff in Indy]

I hadn't had time to work on this for a while, but today Disc 2 became unmountable.

 

I shut down the server, and tried replacing the SATA data cables on both drives.

 

I started it up, went to Maintenance mode, and ran the Check Filesystem Status.

 

Disc 8's results are unremarkable, nothing to report.  Disc 2's results are nearly 6,600 lines long, errors everywhere.

 

How should I proceed?  I have 2 parity drives, so I could rebuild if I need to.  Drive 8 (my smallest drive, 4TB) is currently empty, but I haven't zeroed it out.  

kaokulk-diagnostics-20220519-1955.zip

[JonathanM]

12 minutes ago, Jeff in Indy said:

I have 2 parity drives, so I could rebuild if I need to. 

Parity is kept realtime, so what is showing on the disk is in parity, if parity is valid. File system errors are faithfully followed in parity, just like any other write.

 

You need to follow through with the filesystem check on disk2.

[Jeff in Indy]

1 minute ago, JonathanM said:

You need to follow through with the filesystem check on disk2.

To be clear, "follow through" is to run it WITHOUT "-n"?

[JonathanM]

Just now, Jeff in Indy said:

To be clear, "follow through" is to run it WITHOUT "-n"?

Yep. If it asks for the command line switch to clear the logs, that too. It should tell you what it wants.

[Jeff in Indy]

My lost+found folder is 2.2TB... 

 

OK, so now what do I do?  Both drives are still listed as disabled.  I'm not sure where to go.

kaokulk-diagnostics-20220519-1955.zip

[JorgeB]

You can check if the data in the actual disks is in better shape, unassign both disabled disks, start array, stop array, now you can use UD to mount the old disks and check contents, note that if you want to mount them with the array started you'll need to change both XFS UUIDs in UD settings first.

[Jeff in Indy]

I'll try that.

 

How can I tell why the disks are unmountable?  What should I try?  I don't particularly care about Disc 8, it's only 4TB and I've got the space.  But I don't want to keep using Disc2 (10TB) if there's a physical problem with the drive...

[JorgeB]

3 minutes ago, Jeff in Indy said:

What should I try?

What I wrote above...

 

4 minutes ago, Jeff in Indy said:

if there's a physical problem with the drive...

Both disks look healthy.

[Jeff in Indy]

It took a while to get it right, but I got Disc 2 in UD and it LOOKS like the data is OK, but I can't figure out how to check the disc or copy the files.

 

I can live with the 2.2TB in lost+found.  I just want to be able to get the discs to not be disabled.  Do I need to wipe the drives and rebuild?