I am running beta10a (non-Xen) and recently swapped out my 4TB WD Red parity drive with a new 6TB WD Red parity drive. I did 3-pass pre-clear to ensure the disk was good, and then did the swap and built parity, and then immediately did a parity check.

 

Parity check corrected 128 errors, which I thought may have been due to changes made during the parity build - so I ran a parity check again. The second pass came up with 128 parity corrections as well, and I also noticed on the main GUI page that the parity disk reported 512 errors. I checked all disks in SMART reports and there were no issues reported.

 

Someone recommended I try the parity check under a Xen config as others had reported issues, and sure enough, the parity check completed with no parity corrections and no disks errors.

 

I've included the syslog from the nonxen boot and the xen boot so you can review the differences (both are in the same zip file). The non-xen boot showed a number of IO errors with a couple of disks, but I think the xen boot was clean - which is very odd since there are no changes other than updating the syslinux.cfg file and rebooting.

syslog.zip

First, thanks for the pair of zipped syslogs, very helpful.  I looked through them when you first posted, but syslog analysis is a LOT of hard work, and there was a lot of info there, but no good easy conclusions initially.  I was tired and went to bed.  I have an awful time picking projects back up, so didn't, until another idea hit me today.  But now I'm out of time, have to work all weekend, try to get back to it as I find time late at night, or Monday.

 

I'm easily sidetracked, and did lose time with a CacheDirs issue.  This is offtrack, but I believe CacheDirs is partially broken if you don't specify the desired folders to be cached with the -i option.  I'll deal with that separately, in the right thread.

 

The 2 syslogs are similar, but not completely comparable, in that the Xen version of Oct 7 is only a boot plus immediate parity check, while the non-Xen of Sep 30 has quite a bit of other activity first, which may change the environment before the parity check is finally run.  Someone elsewhere has already suggested that.  The way they repeated the problem was after another heavy process before the parity check was run.

 

I have to run now, but what I recall of your syslogs is that you have the SAS card partially crashing like others, with drive trouble resulting, but then your SAS card recovers and no drives are lost.  That is different from the others.  More analysis is needed, when I have time.  Sorry...

First, thanks for the pair of zipped syslogs, very helpful.  I looked through them when you first posted, but syslog analysis is a LOT of hard work, and there was a lot of info there, but no good easy conclusions initially.  I was tired and went to bed.  I have an awful time picking projects back up, so didn't, until another idea hit me today.  But now I'm out of time, have to work all weekend, try to get back to it as I find time late at night, or Monday.

 

I'm easily sidetracked, and did lose time with a CacheDirs issue.  This is offtrack, but I believe CacheDirs is partially broken if you don't specify the desired folders to be cached with the -i option.  I'll deal with that separately, in the right thread.

 

The 2 syslogs are similar, but not completely comparable, in that the Xen version of Oct 7 is only a boot plus immediate parity check, while the non-Xen of Sep 30 has quite a bit of other activity first, which may change the environment before the parity check is finally run.  Someone elsewhere has already suggested that.  The way they repeated the problem was after another heavy process before the parity check was run.

 

I have to run now, but what I recall of your syslogs is that you have the SAS card partially crashing like others, with drive trouble resulting, but then your SAS card recovers and no drives are lost.  That is different from the others.  More analysis is needed, when I have time.  Sorry...

 

Thanks for providing at least preliminary thoughts.

 

Would it be helpful for me to reboot into non-Xen, do a parity check and then post a cleaner syslog? Or just leave it alone for now until someone comes up with a better testing method.

 

Also, should I disable cache_dirs for the moment while testing? I installed out of novelty - not real need. So I have no problem removing it to avoid false trails.

Whenever reporting issues here, it is best to first try disabling all plugins (such as cache dirs) to help us narrow down the root cause.  This can be done simply by booting into "safe mode" and seeing if your problem persists.  If not, we know at least that the issues sits outside the core of unRAID itself.

Fair enough.

 

I didn't really think about cache_dirs until it was brought up by RobJ. I installed it and forgot about it as I haven't really seen it do anything for me.

 

I will reboot into safe mode and kick off a parity check.

I didn't really think about cache_dirs until it was brought up by RobJ. I installed it and forgot about it as I haven't really seen it do anything for me.

In normal use cache_dirs should be largely invisible.  One tends to notice the absence of cache_dirs when browsing shares suddenly starts taking much longer 

Fair enough.

 

I didn't really think about cache_dirs until it was brought up by RobJ. I installed it and forgot about it as I haven't really seen it do anything for me.

 

I will reboot into safe mode and kick off a parity check.

I had tested with and without it, and it had the same issues.  I have been running more than a week under Xen with cache_dirs running/migrating data and normal use, and not seen a single issue.  Under non-Xen, even with no plugins, I could never get more than 2 parity checks to run before having an issue.

We are trying to replicate this issue today.  Stay tuned.

I had gotten 87% of the way through a parity check with no parity disk issues and no parity corrections, however we blew a breaker roasting turkeys (Canadian Thanksgiving weekend), and it was unfortunately on the same line as my servers. Since I am running without plugins I am guessing that means no APCUPSD either? I just checked my server and it's at 1% on a parity check which I am guessing was due to a power blip.

 

Regardless... the disk errors and corrections were usually near the beginning of the parity check, and since I have a 6TB parity disk and my largest data disk is 4TB I would assume I was in whitespace territory at 87%, so it looks like it was a clean check.

 

Once this check is done, I am going to reboot into normal mode, and try a check again and see what happens with apcupsd and cache_dirs installed.

Just because this experience has brought this to front of mind... this is another (great) example of why things like apcupsd should be part of the core product and not a plugin. I have no issue testing without 3rd party plugins, but am sort of annoyed that my system was left exposed by this process and wasn't able to complete a clean shutdown.

 

Just saying...

Working to resolve that concern.

 

On a side note, tom and I both tested this today and were not able to replicate on our systems.  Need more time on this one...

I am 46.4% through the parity check on the safe mode boot and no issues. It should finish at some point over night. I will boot up normally tomorrow, but will keep cache_dirs disabled, and will try a parity check again and see what happens.

Please do. We cannot replicate this issue on our test systems. We do not have cache dirs enabled on any test systems.  Love cache dirs, but not ready to add that into the mix for testing yet.

Just an FYI, it appears I have only found three people that have had this issue.  I also cannot replicate this on any of my test systems nor can Tom.

 

We need a path to repeat this consistently without the use of plugins to narrow down the root cause.  If we cannot, then this could be a driver / firmware issue that isn't working right with the newer Linux kernels for a small portion of users.

 

The testing I performed yesterday was by writing a block directly to the parity disk using "dd" and then doing a sync.  After the sync, I ran a parity check with "write corrections to disk" checked and all errors found were resolved.  Tested this multiple times without failure in non-Xen mode.

 

If anyone here can provide a 1, 2, 3 guide to recreating this issue consistently, it would be greatly appreciated.

I am not sure if it is relevant but my brother and I seem to have this issue on our systems.  One thing that might be relevant is that we are using AMD processors rather than Intel ones?

 

I seem to be able to reproduce it reliably on my system when running in the default KVM mode - the question is what (if any) diagnostics we can capture that might be of use.

I am not sure if it is relevant but my brother and I seem to have this issue on our systems.  One thing that might be relevant is that we are using AMD processors rather than Intel ones?

 

I seem to be able to reproduce it reliably on my system when running in the default KVM mode - the question is what (if any) diagnostics we can capture that might be of use.

 

Ok, that MAY be related, but I don't want to assume that just yet.  Right now, we just need to find a way to reproduce it on our end.

 

So is every time your array is running with beta 10a you are seeing parity errors and when you run a check with the "write corrections to disk" checkbox clicked, the array reports to correct the errors, but then it still reports them on the webGui after the check is completed?

 

Maybe post a video of the webGui or screenshots of the different steps?  We are missing some vital details here...

I had a clean parity check with no plugins and have started the new parity check on a standard boot (without cache_dirs). So far, so good and I will post final results.

 

I am running an Intel processor, so don't think it's necessarily a purely AMD issue - however if my problem has gone away, then I am at a loss as to the exact cause and may have just been an anomaly.

 

Ok, I'm going to leave this issue open for now.  As of now, there are 4 people who have reported having this:

 

Yourself, itimpi, his brother, and jphipps.  If anyone is still having this issue, please post back with more insight so we can try to help diagnose.  We're pretty close to ruling this one as an anomaly that is hardware or configuration-specific, as we haven't been able to reproduce.  I myself have tested this with 3 different systems so far and I cannot get the issue to happen...

I also find it strange that I only seem to get in KVM mode - and not Xen mode!  Even then I have only managed to get in the parity check - not the initial parity build.  However I am not sure if that is really the case or simply that I have not tried long enough?  I agree it is extremely hard to debug this sort f thing if it cannot reliably be reproduced.

 

I am hoping that one of the beta's (the next one?) will have the option to boot without KVM or Xen support in the kernel to see if that changes things.  It could be something obscure like a kernel setting that is activated by what virtualisation support is enabled and needs a particular hardware environment to trigger it?

Highly possible this is a driver/timing issue. I would suggest posting hardware configs to help reveal patterns.

Highly possible this is a driver/timing issue. I would suggest posting hardware configs to help reveal patterns.

It could well be.  I have only recently introduced a number of WD 6TB Red drives into my system.  I also have a mix of disk sizes. Current hardware config is in my signature.

 

My brother has also noted that he has a WD 6TB red that seems to drop offline unexpectedly, and does not think it is a cable/power issue.  I wonder if it is due to the fact that these drives take a considerable time to spin up which is one reason for their exceptionally low power consumption?

Is everyone experiencing this problem using 6TB HDDs?  I am doubtful that's the cause, but trying to identify similar scenarios. 

Is everyone experiencing this problem using 6TB HDDs?  I am doubtful that's the cause, but trying to identify similar scenarios.

 

My issue arose when when I added a 6TB WD Red as parity. The build was fine, but the check had 128 corrections, as did a second check (plus the drive errors).

 

 

OK, so there are 3 people out of the 4 that have reported this issue specifically when adding a 6tb drive to replace the original.

 

That's highly suspicious.  Could just be coincidence, but when you're debugging, you can't believe in coincidences.

 

Unfortunately we don't have any 6TB drives to test with.  Might have to buy a few.

I don't have any 6TB drives, all 4TB and 2TB drives.

 

Strange thing, I have 2 servers, an only one of them has an issue:

 

Server #1 with Issue:

CPU: AMD

Memory: 4GB

Motherboard: EVGA

Controller:  SUPERMICRO AOC-SAS2LP-MV8 ( Had another Marvel based card with same issue swapped this one in since I had no issues with this same card in my other server.  Even swapped cards )

Drives: 3x4TB 4x2TB

 

Server #2 with no Issues:

CPU: I3

Memory: 8GB

Motherboard: Supermicro

Controller: 2 x SUPERMICRO AOC-SAS2LP-MV8 4 onbaord Sata

Drives: 11x4TB  8x3TB

 

Server #2 has never had an issue.  Server #1 has the issue.  Almost seems load related or memory related.  If I shutdown NFS and Samba and do  parity check under non-Xen I can get a parity check to complete.  If I have them started and have clients writing data to the array or doing rsync's I get a crash during the parity check, or within 24 hours.  I have tried with and without cache_dirs started and it didn't seem to make any difference.  One thing I did notice, but maybe coincidence, it seem to happen more frequently with XFS volumes.  I had tried converting all volumes to BTRFS, and it would take longer before an issue occurred.  Actually I first noticed this issue when I start converting my drives form reiserfs to other filesystems, but it could have just started with that beta since I converted within a few days.