Help Needed - Terribly slow parity-checks after ~30% completed

[DLCIncluded]

Hi, so I have spent the past few weeks googling, and attempting to properly troubleshoot why whenever I try to parity check I can get to about 30% or so, at about 100-120 MB/s, but past that it drops to about 800 KB/s. I had tried most fixes I saw on the forums, I can't remember all I have done, but then I figured I would try backing up all my data, load up a fresh install of unraid activate and transfer my data back on. But alas, it was no good. It came back. 

I am not sure of hardware failures, as no errors point to it, but there is always the possibility. 

Hardware:
Gigabyte X370 K5
Ryzen 7 2700x
48GB Ram (cannot remember model off top of my head)

boot USB is Kingston Datatraveler 16gb

Cache drive is a 1tb NVME M.2 ssd

Parity drive is a 12 TB WD drive,

storage array: 
1 x 12tb WD drive

3 x 4tb Seagate drives

 

some of my share settings such as use cache drive are a bit messy right now, as I was testing things seeing if it mattered having anything on the cache or not. Never moved everything back. 

The only real issue/warning I get in "Fix Common Problems" is the app data is not cache only which is explained above. 

I have attached my diagnostics zip, to hopefully be more helpful than this. If anything else is necessary please let me know! Thank you in advance for any help/suggestions 

loki-diagnostics-20201005-1716.zip

[trurl]

4 minutes ago, DLCIncluded said:

storage array: 
1 x 12tb WD drive

3 x 4tb Seagate drives

This makes me wonder if you know that parity only needs to be as large as the largest single data disk. Parity is not a backup and contains no data, but it looks like you may have sized parity so it is as large as the total data storage. 8TB of that parity is currently wasted, but it does mean you can use 12TB data disks in the future.

 

Once parity check hits 4TB it isn't doing anything at all except verifying the rest of parity disk is zeros. In fact other disks can spin down at that point unless something else is using them.

 

Doesn't explain the slower progress at that point though.

 

Syslog seems to indicate parity is being corrected, probably beyond the point where no other disks are involved and parity should be all zeros already.

 

Are you sure you did the initial parity build when you added that parity disk? What else can you tell us about how you got to this point?

 

[trurl]

Also Call Trace at the end of syslog. Since you are running Ryzen maybe something here is relevant to that:

 

[trurl]

18 minutes ago, DLCIncluded said:

some of my share settings such as use cache drive are a bit messy right now, as I was testing things seeing if it mattered having anything on the cache or not. Never moved everything back. 

The only real issue/warning I get in "Fix Common Problems" is the app data is not cache only which is explained above. 

Not only appdata, but domains and system shares should also be on cache and set to stay on cache. You have these shares set to be moved from cache to the array.

 

Not only is there a performance impact for dockers / VMs due to parity, but having these on the array will also keep disks spunup since these will always have open files.

[DLCIncluded]

Thank you for your quick replies.

I do know that the parity needs to be as big as the largest drive in the system. I believe the way I worded the system hdds is a bit misleading / incorrect. Total there are two 12 tb drives, and 3 4tb drives. one of the two 12tb drives is parity.

I had another 12 tb to put in, but it was DOA unfortunately. That is why the 12tb drive was set as a parity disk.  
This is the current drives ^ 

I did allow it to completely build the parity when I setup the server after reinstalling Unraid. I started with all data backed up onto other systems in my network, and just launched this server with the drives, formatted and started a parity build. it finished in a little over 25 hours or something, I cannot remember, it was probably two or three months ago. 

I added dockers, and VMS back put my files back and did a parity check with out much issue. Now a few weeks went by and it started again. 

Mind I ran the server for almost a year prior with the same hardware with out issues. 

The cache settings were to test a theory that I found in another forum thread, that the cache drive file system was corrupted. So I had moved everything off it and formatted, and set it back up with the cache drive, and the issue still happened. I never got around to fixing the settings because I haven't had the chance. 

I will look into the Unraid v5 on Ryzen thread you linked to and see if I can work anything out of that. 

Thanks!



 

[trurl]

How many sync errors is it reporting now? When you added the 12TB data disk did you let Unraid clear it, or did you use one of the preclear utilities on it? I am still wondering if your parity was valid since it seemed to be correcting a lot of parity errors later in the parity check.

[DLCIncluded]

All drives were added at the same time in this current system. I let unraid do it's magic on the drives I did not do a preclear with any other app or utility. I had reinstalled Unraid fresh on this USB applied my key, and then setup the array with the reformatted drives from the previous setup. 

 

I found the BIOS setting in that thread and have disabled it as recommended. I am not sure the parity is valid either at this point. Would it be best to rebuild it? If so what would the best way to do that, as it's something I have never done before. 

[DLCIncluded]

Also as to the number of errors, I am not sure, I did not check before powering down. I cancelled the parity, and shut down, went into BIOS and disabled the c-state settings from the Ryzen thread above. But I believe it was rising still. 

[trurl]

Just do a correcting parity check and let it complete, then do a non-correcting parity check to verify there are no parity errors.

[JorgeB]

8 hours ago, DLCIncluded said:

went into BIOS and disabled the c-state settings from the Ryzen thread above.

You also have the RAM overclocked, max speed for your config is 1866Mhz, it's at 3000Mhz.

[DLCIncluded]

After disabling the C-State, I have made it to 71.4%, and am at 54.0 MB/sec with only 1  sync error. As long as this continues the C-State appears to be the main issue. Appreciate the help

[trurl]

4 hours ago, DLCIncluded said:

only 1  sync error

1 too many. You must correct

[DLCIncluded]

Hey I'll take 1 over the like 2000+ it was saying previously lol Appreciate the help, we are at 91.7% still at a good speed and still only that one error. Once done, I'm gonna run another just to be sure but this may have sorted it out

[JorgeB]

Very unlikely that the C-state setting has any impact on sync errors, though it can help if the server crashed when idle.

[DLCIncluded]

I believe the C-State was the cause of the original issue, as I noticed yesterday when it dropped to the like 800kb/s that I couldn't get any setting changes to stick in the web ui. I didn't really put it together until I thought about it today, but that makes sense if the system was hanging, it'd not want to continue. 

As for the sync errors, you're right, but if those 2000+ were fixed with yesterdays parity check, before the system hung, and there was only 1 more after that, the parity check after changing the C-state caught that one. Hopefully this corrects the issue I had, so I can go back to actually using the server with out worry.

 

[itimpi]

If you still have the RAM over clocked then be aware this can cause both stability issues and (potentially) data corruption.