Hi All,

 

Been running on Unraid for a long time now (10+ years), and more recently on 6.9.2 on an asus z370/i7-8700k build (14 drives of various sizes, few dockers, couple of VMs).

 

I've been thinking about converting my REISERFS drives (which is all of them, haha) to xfs for quite some time.  So, when the urge hit me to start working on that a couple days ago, I decided to do a non-correcting parity check first .  I don't have a scheduled one and hadn't done one in almost a year, so it was definite long overdue!

 

Parity check came back with what it called errors (324, but showing up in the log as green?), but also, strangely still says 'Parity is valid' on the Dashboard...

 

I skimmed through the syslog and that's when I noticed a few REISERFS errors that appear to be happening only on the parity drive.  Not sure if these are related.

 

The parity errors all look like the below, with different sector #s referenced:

Feb 10 06:23:05 nas1 kernel: md: recovery thread: P incorrect, sector=128

 

Probably unrelated: There were also some BTRFS errors on my cache drive, which I think I have addressed by following some advice in another post.  I found the file from the inode referenced, removed it, and ran a scrub (non-correcting) that came back clean after the file removal.  Just thought I'd include this bit, as it's responsible for most of the log errors.

 

On the REISERFS/parity check issue, I'm looking for recommended next steps.  Diagnostic attached.

 

Thanks!

nas1-diagnostics-20230211-0859.zip

1 hour ago, garys said:

I've been thinking about converting my REISERFS drives

 

1 hour ago, garys said:

Parity check came back with what it called errors (324, but showing up in the log as green?), but also, strangely still says 'Parity is valid' on the Dashboard...

Parity Valid and green just means the last parity operation completed successfully. That small number of sync errors is common for unclean shutdowns.

1 hour ago, garys said:

I skimmed through the syslog and that's when I noticed a few REISERFS errors that appear to be happening only on the parity drive.  Not sure if these are related.

You must be misinterpreting something. Parity has no filesystem so it can't possibly have reiserfs errors.

Nothing wrong with this but just to make sure I'm not overlooking something.

 

Why don't you have a disk12?

Dec  2 03:49:13 nas1 kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 7532155 off 1222004736 csum 0x434c92e3 expected csum 0x469e61b4 mirror 1

btrfs csum errors often indicate bad RAM.

 

You should run memtest immediately before doing anything else.

1 minute ago, trurl said:

You should run memtest immediately before doing anything else.

Will wait on the results before continuing

Hey - thanks for the replies!

 

2 hours ago, trurl said:

Parity Valid and green just means the last parity operation completed successfully. That small number of sync errors is common for unclean shutdowns.

So, just to be clear, even though it I'm getting a notification that says Parity check finished (324 errors), there's nothing that needs to be done about that?

 

2 hours ago, trurl said:

You must be misinterpreting something. Parity has no filesystem so it can't possibly have reiserfs errors.

Here are the log entries I was referring to:

May  6 16:08:17 nas1 kernel: REISERFS error (device sde1): vs-5150 search_by_key: invalid format found in block 661592588. Fsck?
May  6 16:08:17 nas1 kernel: REISERFS error (device sde1): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [1 2 0x0 SD]

 

I thought it was odd too, that it would be referring to parity, but the device mentioned (sde) is my parity disk.  What could that mean?

 

2 hours ago, trurl said:

Nothing wrong with this but just to make sure I'm not overlooking something.

 

Why don't you have a disk12?

Haha, yeah, for some reason I decided to skip that slot because of how the drives were positioned in my 5in3 bays when i did a re-org years ago.  No idea why I did that now, but it hasn't seemed to cause any issues, so I've left it.  I have 13 data disks + parity.

 

2 hours ago, trurl said:

Dec  2 03:49:13 nas1 kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 7532155 off 1222004736 csum 0x434c92e3 expected csum 0x469e61b4 mirror 1

btrfs csum errors often indicate bad RAM.

 

You should run memtest immediately before doing anything else.

Interesting.  Ok - I will run the memtest now and post back the results.

 

Thanks!

6 hours ago, trurl said:

Will wait on the results before continuing

I didn't finish all I had to say because you don't want to do anything else at all unless and until RAM is working perfectly.

4 hours ago, garys said:

even though it I'm getting a notification that says Parity check finished (324 errors), there's nothing that needs to be done about that?

You need to run a correcting parity check, but not until RAM is working perfectly.

 

4 hours ago, garys said:

Here are the log entries I was referring to:

May  6 16:08:17 nas1 kernel: REISERFS error (device sde1): vs-5150 search_by_key: invalid format found in block 661592588. Fsck?
May  6 16:08:17 nas1 kernel: REISERFS error (device sde1): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [1 2 0x0 SD]

 

I thought it was odd too, that it would be referring to parity, but the device mentioned (sde) is my parity disk.  What could that mean?

I don't know what that was about either. We don't see reiser that much in syslogs anymore, not since I have been studying diagnostics. It seemed to be doing some sort of reiser check against all your devices regardless of how they were assigned, and it didn't think sde looked like a good reiser disk, which it isn't.

 

Usually when working with an array disk, it would be referred to as md# instead of sdX, because operations on the md devices include parity updates when writing or all the array to calculate the data when emulating a missing disk or whatever needs to be done with a disk assigned to the parity array. If you were to write the sdX device in the array, for example, it would invalidate parity since parity is bypassed unless you use the md# device.

Hey - just an update...

 

You were dead-on with the suggestion run a memtest... it's only been running a few hours and it's already reported 2 errors.

Specifically:

Tst:9, Pass:0, Failing Address:0042afffb58 - 17071.9MB, Good: c90606c3, Bad: c90606c1, Err-Bits: 00000002, Count: 1, CPU: 0

 

I don't know much(anything) about memory or memory tests.  Have run a handful of tests over the years and have always come out clean in the past.  This has me pretty concerned, actually.  Knowing that a memory issue is actually causing data issues that are probably not recoverable.  Obviously it's why we have backups, but still a bit unnerving.

 

So, the obvious next question is, where to go from here.  I'm not 100% sure if it's 2 sticks or 4 (I'm thinking it's 2), so wondering if I should be able to tell which stick has the problem from the error info.  If not, I could run the test with only 1 stick in, then switch to the other.  I don't think I'm using even close to half of available memory, at least not currently, so i might be able to get by on one stick for awhile.

 

Other than doing the above, I'd be open to other suggestions.

 

Thanks for all the help so far!

2 minutes ago, garys said:

I don't know much(anything) about memory

You don't even want to attempt to run any computer unless memory is working perfectly. Everything goes through RAM, the OS and other executable code, your data, everything. The CPU can't do anything with anything until it is loaded into RAM.

 

That's why I said

7 hours ago, trurl said:

You should run memtest immediately before doing anything else.

 

7 minutes ago, garys said:

run the test with only 1 stick in, then switch to the other

That is what I have always done.

 

Also when testing RAM, some possibility of bad memory slot on the mobo, but I think that would show up more drastically with lots of errors.

Thanks - I will try testing with 1 stick at a time and see how that goes.  I'll post back the results when finished either this evening or in the morning.

 

Here's hoping it's 2 sticks (and not 1 or 4!)

Finally done with the memory tests - got the bad stick isolated (turned out to be four 8GB sticks).  Also ran several rounds of testing with the two sticks I plan to use for now, and they've come out unscathed.

 

So, for the next step, would you say another non-correcting parity check, or just go straight to the correcting one?

 

Thanks!

 

If you run a non-correcting parity check and you still have sync errors, you will have to do a correcting parity check to get those corrected because there is no way to know which disk might be out-of-sync. And a few sync errors are not uncommon if you have had unclean shutdown.

 

If you run a correcting parity check and there are no sync errors to correct because those found previously were due to bad RAM, then it won't correct anything.

 

So, I don't see any reason to not do a correcting parity check.

Ok - I've started the parity check (correcting).  I'll post back with how it went.

 

Thanks!

Alright - ran a correcting and fixed 400+ errors, then a non-correcting and came back with 0.

 

I'm considering this issue resolved.

 

Thanks again for all the help - appreciate it greatly!