clowncracker

It ended up being a power problem, I replaced the sata power cables and the problem is now resolved.

There has been no unclean shutdown since the last check. Both of those drives are on power splitters though, not sure if that makes a huge difference. https://www.amazon.com/gp/product/B012BPLW08/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1 Attached are my PSU specs. Just for my own knowledge, how did you identify the issues were disk9 and parity?

Any way to check which device it is referring to? That can be any drive or any cable, it's near impossible to troubleshoot.

Parity check started last night and I'm already at 873 errors. Parity check completed last month with no errors, so I was hoping someone could look at the diagnostics and troubleshoot the issue.

So I should leave that blank if I only want to keep 4 backups?

I want to only keep 4 backups, but the older versions are not getting deleted (5 copies are here). I want to make sure I have my settings set up correctly: Delete backups if older than x days:28 Keep at least this many backups: 4

That ended up being the issue, parity check and mover were running at the same time.

I think I found the issue, it looks like mover started running during the parity check. I thought that couldn't normally happen, but that explains the writes to drive 3 and the parity drives. I noticed the amount of free storage on my cache was increasing, so I paused the parity check and sure enough mover was running. Once mover is done I'll resume the parity check and see if that solves the problem.

I'd like to note that writes to parity and disk 3 are continually increasing every second:

I've started running my monthly non-correcting parity check and it was going smoothly until about an hour ago. Now the estimated finish is in ~30 days. It looks like a lot of writing is happening to disk 3, but I don't know what is happening:

@itimpi I opted to go for the correcting parity check since it was functionally the same. Did a second parity check afterwards and there were 0 issues. @trurl thank you for your help with the original issue, I wish I could give you both the solution.

So I shouldn't be deselecting any drives then? What is the functional difference between doing a sync by creating a new config and and just doing a correcting parity check?

Then won't @JorgeB's solution lead to permanent data loss? Aren't I just better off just doing a correcting parity check?

Just wanted to follow up.

Based on my comments what do you think about checking the parity drives? Do you think it's worth the risk, especially since they are essentially brand new? Something else I'm curious about. If I start the array with a new config and a disk missing, won't I just lose the data currently on the disk? Since it isn't on the array and it won't be emulated since the disk doesn't exist in the new array.

Do you think it's worth testing the parity drives even if they are brand new? I wasn't having any issues until a drive failed mid-parity check (non-correcting), so it makes me think the parity drives cannot be the problem. If I did want to test them, I should remove both parity disks and put in a new drive and rebuild parity? Part of me is concerned about rebuilding the parity disks if one of the data disks might be an issue. If another drive fails during a parity rebuild, then at that point won't I have corrupted data permanently? If not testing parity, which disk do you think is most likely the culprit? Disk 3 that was just recently replaced that started the whole problem? Disk 4 that has some relocated sector counts? Or Disk 10, which is the newest disk in the array?

Even if it detects issues? I should be running a non-correcting parity check right?

I'm going to assume the parity disks aren't the issue, since they've both been replaced in past 4 months with brand new drives. I'm going to test disk 3 that was just replaced (which caused all of these issues to begin with), disk 4 that has sector count issues and disk 10 (which is the newest drive in the array). So to confirm: 1) Save a backup of super.dat (I'll just use the file from the diagnostics). 2) Stop the array. 3) Tools > New config, selecting all in the Preserve current assignments section. 4) DESELECT DISK 3 - making sure Parity is Valid is UNCHECKED. 5) Start the array. 6a) Run a parity check, if there are issues I should restart the process with another drive. 6b) If there are no issues, run another parity check to make sure there are no issues. 7) Once I've identified the drive with problems, stop the array and rebuild it with a new disk. Run two parity checks to make sure there are no issues.

I honestly don't think it's a hardware issue (memory, controller, cables, etc). It might be disk related, but how would I go about testing that? Stopping the array, disabling a disk, starting the server and just running a parity check? If that's the case how would I actually go about fixing the issue?

Memtest came back with no issues.

I've attached another diagnostic file after restarting the parity check after a fresh reboot. I'm at around 390 errors at the moment. clowncracker-diagnostics-20231107-1201.zip

Sorry for the delay, but I've been out of town for a few days. I just started another parity check. I've already found 7685 errors in the past 10 minutes, so I'm concerned there is still a problem. I decided to cancel the parity check and restart my server. The config still looks correct, I've attached is another diagnostics log. clowncracker-diagnostics-20231107-1044.zip

Rebuild just finished. Attached i s the new diagnostics file without rebooting (pulled it immediately after the rebuild finished). clowncracker-diagnostics-20231102-1555.zip

Just to confirm: clowncracker-diagnostics-20231101-1125.zip

Flash drive looks fine. Just plugged it in and it looks like there are no issues. I guess I could rebuild disk 3 again, but I am concerned with the number of parity errors it was throwing out yesterday when I ran it after the rebuild. It also might have been throwing the parity errors because for some reason disk 3 wasn't recognized correctly after I did the Check Filesystem. Maybe I should have done the Check Filesystem before I rebuilt disk 3? It did show disk 3 only having 5TB after the rebuild, which is weird. The screenshot for that is in the original post.