vmax5000: Help with extremely slow parity check

[vmax5000]

I'm also having this trouble, it started after a power outage, I do have battery backup but it's not recognized by the system so no auto shutdown occured. After it came back up I started getting massive sync errors on one of my parity drives, but that turned out to be the sata cable (I have ordered a full replacement for all drives). After I put in the new cable I started getting around 50mbs per second which was great, but after about 10 min's it's barely 1 mb per second. Not sure what's going on now.  Any ideas would be great, see snapshot and diags..

Thanks in advance..

ultimate-diagnostics-20200316-1215.zip

[trurl]

I have split your post into its own thread.

[trurl]

55 minutes ago, vmax5000 said:

After I put in the new cable

Probably disturbed other connections.

 

You are having problems communicating with multiple disks. Check all connections, power and SATA, both ends, including power splitters.

[itimpi]

You are currently getting continual resets on your parity drive which is why it is running slowly.   That is typically a cabling problem, but it could also be power related, or even a problem with the drive itself.

 

It is probably worth running an extended SMART test on that drive to check its health.

[vmax5000]

Thanks for the support, I will give those suggestions a try and see how it goes.

 

[vmax5000]

I thought I would try a more radical approach, so I removed the parity drive that I was having issues with and since I had 2 parity drives, I thought I would try running on just the one. As it turns out without that drive I am getting much more normal speeds, I need to let the parity check finish first and then I will try and figure out if it's the drive, the cable or the motherboard port (I had one motherboard sata port die already), I have ordered a pci express sata card and new cables. I will let you know the final result.

Thanks again for the help...

V

[trurl]

2 minutes ago, vmax5000 said:

I removed the parity drive that I was having issues with

Which parity was it? They are not interchangeable.

[vmax5000]

I know they are not interchangeable, I just took the one offline and left the other one where it was, I didn't switch them or anything.

[JonathanM]

8 hours ago, vmax5000 said:

I know they are not interchangeable, I just took the one offline and left the other one where it was, I didn't switch them or anything.

That's not the point.

trurl asked a specific question, "Which parity was it?"

Did you remove parity1 or parity2?

[vmax5000]

Sorry, I removed parity one

[vmax5000]

Parity one was the drive that had all of the errors, I was under the impression that because I have 2 parity drives, if one parity drive failed then I would have the second parity drive as a backup. I assumed that both parity drives had the exact same data, is that wrong?  Everything seems to be working ok and the parity check is running at normal speed right now. Should I be worried?

Thanks...

V

[itimpi]

You are correct in that as long as you have a single parity drive then you are protected against any single array drive failing.    You are incorrect about the two parity drives having the same data - the calculations for parity2 are different to those for parity1.

[trurl]

Parity2 wouldn't really provide any extra redundancy if it were the same. If parity and parity2 were the same, then if one parity failed you would still have the other and could recover the data for a single data disk. But if 2 data disks failed at the same time, then you wouldn't have enough parity information to recover both data disks. If both parity were the same they wouldn't have any more information than a single parity disk and could only recover a single data disk.

 

 

[trurl]

The reason I said they weren't interchangeable, which you said you knew even though you thought they were the same, is because the parity on parity1 is not valid for the parity2 slot, and the parity on parity2 is not valid for the parity1 slot.

[vmax5000]

Thanks for the explanation!  I think I should be ok, I did have one drive 1 in the array have issues as well, all of the other drives seem ok. I have ordered a replacement drive for the first parity drive, but I won't install it until the current parity check is complete and my array has no more errors.  I really appreciate your information.  I have another question, are the motherboard sata ports locked to the drives in the array?  I have ordered a pci express card with 8 ports just in case the motherboard ports are failing.  I think I may have to build a new system soon, that old motherboard may not be in great shape.

Thanks again,

V

[trurl]

4 minutes ago, vmax5000 said:

I have another question, are the motherboard sata ports locked to the drives in the array?

Unraid keeps track of disk assignments using the serial number of the disks, so it doesn't matter where they are plugged in. The only thing that sometimes gets people in trouble is if they use a controller that doesn't report the drive serial numbers correctly. As long as you don't use a RAID card you should be OK.

[vmax5000]

Perfect, I saw the card was NOT a raid card.

Thanks again

[vmax5000]

Well here is an update. I let the parity check complete, and the drive I lost data on did not get recovered, I had about 7 tbs on it before and it only recovered 580 gbs.  I also put in the new sata card and re-installed the original parity 1 drive and the parity 2 drive to the new card. When I booted up the system the old parity drive had to be overwritten when doing the parity check. It's not too much of a problem as I have hard backups, but I don't know why parity 2 could not rebuild my drive 1 in the array.  Anyway the parity check is running at normal speed with the original parity 1 drive back in place.  Lesson learned: Buy a smart ups!

V

[trurl]

13 minutes ago, vmax5000 said:

I let the parity check complete, and the drive I lost data on did not get recovered, I had about 7 tbs on it before and it only recovered 580 gbs. 

Maybe I missed it somewhere, but this is the first mention I've seen about trying to recover any data. Parity check will not recover any data, all it will do is check (and optionally correct) parity.

 

15 minutes ago, vmax5000 said:

I don't know why parity 2 could not rebuild my drive 1 in the array. 

As far as I know you weren't trying to rebuild drive 1.

 

17 minutes ago, vmax5000 said:

I had about 7 tbs on it before and it only recovered 580 gbs

If I had to guess, and I do since you haven't given any information, I would say you probably formatted that disk.

[vmax5000]

Yup, I had to format because the files system was not recognized on the parity 1 drive, but I had that drive disconnected when I ran the parity check with the parity 2 drive which I did not format.  I don't know how far back the logs go, but here is a copy that I just created.

 

V

ultimate-diagnostics-20200318-1308.zip

[trurl]

Haven't looked at those because syslog is in RAM and only goes back as far as the last reboot. You have to setup syslog server to get it saved somewhere:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=781601

 

9 minutes ago, vmax5000 said:

I had to format because the files system was not recognized on the parity 1 drive

I assume you mean filesystem wasn't recognized on disk1. Parity has no filesystem.

 

You didn't have to format. In fact, that was exactly the wrong thing to do. Even if the filesystem was fine, rebuilding a formatted disk from parity will result in a formatted disk.

 

Format is a write operation. It writes an empty filesystem to the disk. All write operations update parity. So, after formatting a disk in the array, parity agrees the disk has an empty filesystem.

 

The correct thing to do would have been to repair the filesystem. Too late now, but if you had told us what your problem was before you did anything we could have told you exactly what to do.

 

Please ask for advice in the future.

 

[trurl]

26 minutes ago, vmax5000 said:

Yup, I had to format because the files system was not recognized on the parity 1 drive, but I had that drive disconnected when I ran the parity check with the parity 2 drive which I did not format.

And you can't format any parity drive, because parity has no filesystem. As mentioned, format means "write an empty filesystem to this disk". That is what it has always meant in every operating system you have ever used. Many people seem to have a very vague and incorrect idea about what format does.

 

Did you format disk1 after you removed parity1? If so it might be possible to rebuild disk1, filesystem problems and all, from that parity1 with a little trickery. Then we could try to repair it. If parity1 was still installed when you formatted disk1 then nothing to do but go to your backups.

[vmax5000]

I honestly don't remember the steps in order anymore. Point taken on asking for help before moving on, next time I will try and be a bit more patient. I'm currently running the parity check with parity 1 running again, so I doubt that I would be able to try any trickery.  But as I was lucky enough to have hard backups, it's not too much of a concern, it will be a bit time consuming but that's about it.

I really do want to thank you for your help though, I have been on other forums with other nas devices and it sometimes takes days or weeks to get any answers, I wasn't expecting such a quick response from this forum! So going forward I will be more willing to wait before moving on.

 

V

[vmax5000]

Oh and yes it was disk 1 that I formatted in the array, that's the one that lost the data.