Has anything changed in the way unRAID does parity checks since 6.2? or prior? My monster server is doing its monthly parity checks which have always taken about 30hrs (two 8TB Seagate SMR drives for parity) but its been running for almost 10 hrs now and crawling at 4.0MB/s or less and its only completed 1.8% so far. This doesn't seem right.

Dual parity calculation is known to be an issue for some lower end CPUs but I suspect you have another issue. Post diagnostics.

Could be, nothing is apparent though.

 

Diags too large to upload here, so uploaded to dropbox, here is the link

 

https://www.dropbox.com/s/kywmvrvl09cx4o2/tower-diagnostics-20161121-1002.zip?dl=0

Clearly that is NOT running as it should.

 

A couple of questions ...

 

(a)  Have there been any configuration changes since the last parity check?

 

(b)  Is the array being actively used while the parity check is running?  THIS can cause a VERY significant slowdown ... especially if there's more than one client using it.    Remember that a "client" could be a Docker or VM that's actively using the array -- doesn't have to be some other system accessing it via the network.

 

©  Does the Web GUI show any errors on any of the drives ?

 

Your most recent syslog entries:

Nov 21 09:34:12 Tower vsftpd[6268]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:34:12 Tower vsftpd[6268]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:34:14 Tower vsftpd[6270]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..g", 25 bytes, 0.02Kbyte/sec
Nov 21 09:34:19 Tower vsftpd[6248]: [ashman] OK UPLOAD: Client "192.168.111.120", "/mnt/user/C..4", 5136814 bytes, 550.49Kbyte/sec
Nov 21 09:34:21 Tower vsftpd[6260]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..4", 5197821 bytes, 571.99Kbyte/sec
Nov 21 09:34:24 Tower vsftpd[6474]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:34:24 Tower vsftpd[6474]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:34:33 Tower vsftpd[6530]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:34:34 Tower vsftpd[6530]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:34:40 Tower vsftpd[6594]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:34:40 Tower vsftpd[6594]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:34:41 Tower vsftpd[6608]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:34:41 Tower vsftpd[6608]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:34:44 Tower vsftpd[6610]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..g", 25 bytes, 0.02Kbyte/sec
Nov 21 09:34:44 Tower vsftpd[6537]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..4", 5232171 bytes, 542.02Kbyte/sec
Nov 21 09:36:01 Tower vsftpd[7491]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:36:01 Tower vsftpd[7494]: connect from 192.168.111.120 (192.168.111.120)
Nov 21 09:36:01 Tower vsftpd[7491]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:36:01 Tower vsftpd[7494]: [ashman] OK LOGIN: Client "192.168.111.120"
Nov 21 09:36:10 Tower vsftpd[7565]: connect from 192.168.111.120 (192.168.111.120)
Nov 21 09:36:10 Tower vsftpd[7565]: [ashman] OK LOGIN: Client "192.168.111.120"
Nov 21 09:36:11 Tower vsftpd[7573]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:36:11 Tower vsftpd[7573]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:36:15 Tower vsftpd[7604]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:36:15 Tower vsftpd[7604]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:36:17 Tower vsftpd[7620]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:36:17 Tower vsftpd[7620]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:36:19 Tower vsftpd[7628]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..g", 25 bytes, 0.02Kbyte/sec
Nov 21 09:36:20 Tower vsftpd[7568]: [ashman] OK UPLOAD: Client "192.168.111.120", "/mnt/user/C..4", 5143677 bytes, 599.99Kbyte/sec
Nov 21 09:36:23 Tower vsftpd[7578]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..4", 5224649 bytes, 481.20Kbyte/sec
Nov 21 09:39:33 Tower vsftpd[9931]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:39:33 Tower vsftpd[9931]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:39:41 Tower vsftpd[9991]: connect from 192.168.111.121 (192.168.111.121)
Nov 21 09:39:41 Tower vsftpd[9991]: [ashman] OK LOGIN: Client "192.168.111.121"
Nov 21 09:39:50 Tower vsftpd[9994]: [ashman] OK UPLOAD: Client "192.168.111.121", "/mnt/user/C..4", 5148005 bytes, 610.54Kbyte/sec
Nov 21 09:54:00 Tower vsftpd[20371]: connect from 192.168.111.120 (192.168.111.120)
Nov 21 09:54:00 Tower vsftpd[20371]: [ashman] OK LOGIN: Client "192.168.111.120"
Nov 21 09:54:07 Tower vsftpd[20536]: connect from 192.168.111.120 (192.168.111.120)
Nov 21 09:54:07 Tower vsftpd[20536]: [ashman] OK LOGIN: Client "192.168.111.120"
Nov 21 09:54:07 Tower vsftpd[20544]: connect from 192.168.111.120 (192.168.111.120)
Nov 21 09:54:07 Tower vsftpd[20544]: [ashman] OK LOGIN: Client "192.168.111.120"
Nov 21 09:54:10 Tower vsftpd[20546]: [ashman] OK UPLOAD: Client "192.168.111.120", "/mnt/user/C..g", 25 bytes, 0.02Kbyte/sec
Nov 21 09:54:10 Tower vsftpd[20560]: connect from 192.168.111.120 (192.168.111.120)
Nov 21 09:54:10 Tower vsftpd[20560]: [ashman] OK LOGIN: Client "192.168.111.120"
Nov 21 09:54:19 Tower vsftpd[20562]: [ashman] OK UPLOAD: Client "192.168.111.120", "/mnt/user/C..4", 5156314 bytes, 606.71Kbyte/sec

Are you transferring data to your server at the same time it is running the parity check?

I replaced a failed drive the other day, when I replaced it, I put in a dive I thought was good but was actually bad and it started displaying errors right away. At this point I stopped the array, went out, got a replacement drive and replaced the 'good' drive with the new one. I must of screwed something up because this sequence of events didn't go the way it should have and I had to end up removing the drive from the config, stopping the array, starting the array, stopping the array again, adding the drive and starting the array once again, at this point unRAID recognized the drive and started a parity check/rebuild, however when completed the drive had only a 30 some odd GB of data when it should of had over 2TB. As this is my backup server, I simply did a compare/sync with my primary data sources to copy over the missing data from this failed drive which has completed. This may explain the slow parity check?

 

The array is not being used by anyone, as for dockers, there is nothing active with them (nzbget and sonarr are not actively downloading anything and plex isn't doing anything)

 

We gui does not show any errors with the drives.

No that is an ftp server I run that transfers images from security cameras to a share on the array, I can turn it off.

Should I maybe do a new config? I have all my drive locations documented.

Should I maybe do a new config? I have all my drive locations documented.

Not entirely clear how you got to this point. Is your parity check actually correcting a lot of parity errors? If so then that would indeed slow it down. If you somehow reconfigured your array without rebuilding parity, then correcting a lot of parity errors and transferring security camera images at the same time would definitely have an impact on parity check speed.

I've turned off the FTP server, it hasn't made a difference. Certainly it could be correcting a lot of parity errors I guess. At this point its saying its going to take 23 days, I can't let it run for that long, what are my options?

No its running at 58MB/s and says its going to take 1day 13 hrs which is about normal from when it starts. At this point I'll just let it run to completion as long as it doesn't slow to a crawl again.

Interesting that it's sped back up.

 

I know you turned off your cameras, but was the "... compare/sync with my primary data sources ..." still in progress when the parity check started?  You said it's now completed -- just wondering if that was true BEFORE the parity check started.

 

Another possibility is that one (or more) of your archive drives were in a "clear out the persistent cache" mode => this could have been a result of all the writes you did to resync the data.    When they get in this state, they can be VERY slow.

 

... and when they finish, performance will return to normal.    So if the copy was not in progress, I suspect this is likely what happened.

 

No the compare/sync finished on Friday, the drive failed Thursday morning and I spent Friday and Saturday copying over 2TB of data back to the array, data that was not replaced by the parity compare/sync. Oddly enough, it didn't put all the data back on the drive I replaced either. So originally, drive 7 was 2.12 TB full out of 3TB, after I copied the 2.12 TB back to the array, drive 7 is only 1TB full. I guess it doesn't matter, unRAID puts the data wherever.

Yes, UnRAID will put the data wherever it wants in accordance with your allocation method and split levels.

 

I suspect that your archive drives were doing a bit of "persistent cache maintenance" when you started your parity check; and that once that finished the speeds jumped back up to normal.  No way of knowing that for certain; but given what you observed; and the fact that you didn't have anything else going on with the array; I'd say there's a pretty good chance that is what was causing the excessively slow speeds.

 

Mind you I actually recall they they usually start off pretty slow when doing a parity check, it just doesn't usually last 12 hrs.

Did you by any chance notice the position in the check where the speed picked back up?  From what you described, I don't think it's likely it was at any disk size boundary (e.g. 3TB), but if it was, that would point to one of those smaller disks having an issue.  I think it's almost certainly related to the archive disks and their persistent cache; but the fact it ALWAYS starts off like this tends to suggest it might be something else.

 

You're SURE nothing else was going on -- right?  [No Plex streams; no active downloads; etc.]

 

 

No I didn't notice but I'll try to next time it runs, obviously I'm not always in front of my system when the parity check kicks off since I have set to run once a month.

... Certainly it could be correcting a lot of parity errors I guess...

No need to guess. This is displayed in the webUI.

No the compare/sync finished on Friday, the drive failed Thursday morning and I spent Friday and Saturday copying over 2TB of data back to the array, data that was not replaced by the parity compare/sync...

Not sure what you meant by this but it suggests you don't really understand parity. A lot of what unRAID does makes a lot more sense if you understand parity.

I understand how parity works, and so the missing data should of been rebuilt during the parity compare/sync, but was not for some reason.

 

When a drive fails in unRAID and is replaced, the parity sync/compare is supposed to rebuild missing data from parity data, correct?

 

This didn't happen for some reason in my case and it's the second time it's happened to me, I don't know why, if I am doing something wrong or what. Fortunately this is a backup server, so replacing the missing data is not an issue.

Formatting is never part of a rebuild.

 

This should be simple I don't know why I am having problems. A drive indicated it was having read errors, I stopped the array, removed the drive  (not physically) so the slot was empty. I powered down, put in a new drive, powered up, the array had already started, unRAID indicated the drive was unmountable, but offered to format it, I formatted it, now the drive has a red X next to it indicated the contents are emulated and my only option is to do a parity check, not a rebuild?

I understand how parity works, and so the missing data should of been rebuilt during the parity compare/sync, but was not for some reason.

 

When a drive fails in unRAID and is replaced, the parity sync/compare is supposed to rebuild missing data from parity data, correct?

 

This didn't happen for some reason in my case and it's the second time it's happened to me, I don't know why, if I am doing something wrong or what. Fortunately this is a backup server, so replacing the missing data is not an issue.

Not sure what you mean by parity sync/compare, but that is not the terminology I would use when what I really meant was a data disk rebuild. Parity sync implies building parity, compare implies a parity check. Rebuilding a data disk is not either of these, and though the calculation is similar, these don't appear at all the same in the webUI. Maybe you should ask for more help next time.

Thanks Johnnie, I understand formatting is never part of a rebuild, but after putting in what I thought was a good disk and it giving a ton of errors, when I put in the new disk, unRAID didn't recognize it, perhaps at this point I should of done the following:

 

Stopped the array, emptied the slot for drive 7 so there was no disk present. Started the array. Stopped the array, added the new disk, and started the array again, perhaps at this point unRAID would of realized drive 7 had failed and a new drive was in its place and started rebuilding the drive from parity data.

 

Because I didn't follow these steps and unRAID didn't recognize the new disk, it gave me the option to format it, which I did, then it gave me the option to rebuild from parity data, but it didn't do that part properly, likely due to my missteps in handling the situation.

Because I didn't follow these steps and unRAID didn't recognize the new disk, it gave me the option to format it, which I did, then it gave me the option to rebuild from parity data, but it didn't do that part properly, likely due to my missteps in handling the situation.

Your statement implies that you think parity has some concept of data which is not true.  It only has the concept of sectors and what bit pattern they should contain with no idea of what that bit pattern represents.

 

The moment you told unRAID to format a disk you were telling it to create an empty file system (I.e. One with no files) and parity is updated to reflect this.  If you now decided to rebuild that disk all you are doing is rebuilding the disk back to the empty file system created by the earlier format.

Guys, I understand that parity has no concept of data, (I'm just clearly not good at expressing it)  I realize that formatting the drive was my mistake and I think it's the second time I've done it, which I can't believe but I think next time I will just walk away to gather my thoughts rather than charging on and making these mistakes.

When you formatted the drive, that would cause parity to be updated to reflect the formatted state of the drive.

 

So even if you do a rebuild after that, the rebuilt drive will be formatted ==> i.e. once you format the drive, you have lost all previous data from that drive.

 

It's not exactly clear just how you ended up in that state, but in any event it happened.  Next time you have a failure, step back a bit, and perhaps ask for a bit of help from the forum, before you do the rebuild.