Odd disk activity.

[bubbaQ]

Version 6.8.1 with dual parity.  Freshly booted.

 

I'm moving several TB of large (over 500 GB) image files from an old 6TB drive to a new 12TB using DISK shares only.  I enabled turbo write (reconstructive write) expecting things to go faster.  All data drives are WD Reds with xfs.

 

I expected to see writes but no (or few) reads for parity 1 and parity 2 and the destination drive, and reads but no (or few) writes to the other data disks.  I expect the read rate of the other data disks to be about the same as the write rate to parity and the destination drive.

 

But I am seeing two odd things:

 

1) the read rate from the OTHER data disks (about 50MB/s) is only about half the write rate to parity and the destination (about 100 MB/s); and

 

2) is a significant rate of READS from parity and the destination drive (about half the rate of WRITES to those drives)

 

Yesterday I wiped an old data drive with 0 so it could be removed from the array (dd to /dev/mdX) with reconstructive write and I got the expected 170 MB/sec to the data drive and parity with no reads, and pure reads at 170MB/sec from the other data drives.  Removed it and did a parity check and all was fine.

 

As a test, I tried turning off reconstructive write, and got  90MB/sec reads and 90MB/sec writes for parity and destination drives.... just as expected.

 

Any explanation?  Is xfs spreading these large files around on the disk rather than doing linear sequential writes?  If so, why is it doing so much READING from the destination drive and from parity?

 

See attached image.  Source is disk4 and destination is disk5.

[JorgeB]

16 minutes ago, bubbaQ said:

Any explanation?

I don't have an explanation, but I did observe similar behavior, but only sometimes, usually for a short while, and not in a reproducible way, so I never tried to investigate further, but is likely related to btrfs (or COW in general) since all my servers are also btrfs.

[bubbaQ]

39 minutes ago, johnnie.black said:

I don't have an explanation, but I did observe similar behavior, but only sometimes, usually for a short while, and not in a reproducible way, so I never tried to investigate further, but is likely related to btrfs (or COW in general) since all my servers are also btrfs.

I first stopped and restarted the copy and got the same results.  Then I restarted the server, and got the same results.

 

It's been running for several hours and the results have been the same throughout the copy.

[JorgeB]

Yeah, when I got that was for brief periods and just ignored, next transfer was usually OK, I would guess it happened 1 or 2 in 100, and it's been some time.

 

1 hour ago, bubbaQ said:

All data drives are WD Reds with btrfs.

Just noticed screenshot shows XFS for all disks, so not a btrfs related...

[bubbaQ]

You are correct... I corrected the OP to reflect xfs.

 

My bad.... I was working on some btrfs issues on another server the last few days.... and had googled btrfs issues a mission times. 

 

[JorgeB]

20 minutes ago, bubbaQ said:

My bad...

No worries, my bad also because I didn't read the entire post carefully, I missed that this was an array to array copy, so not related to what I was talking about.

 

The issue here is that since v6.8.x Unraid reverts to read/modify/write mode when activity is detected in multiple array disks, and unfortunatly this can't be turned off, I did ask Tom to only use this if write mode was set to auto, but no dice, it can't be disabled for now, so your writes are constantly going from reconstruct write to r/m/w, hence why parity and the destination disk also have reads, for the r/m/w part, if you downgrade to v6.7.x. you'll see the expected behavior, but speed won't likely be much better since disk4 will be constantly seeking for the data and parity calculation reads.

[bubbaQ]

21 minutes ago, johnnie.black said:

No worries, my bad also because I didn't read the entire post carefully, I missed that this was an array to array copy, so not related to what I was talking about.

 

The issue here is that since v6.8.x Unraid reverts to read/modify/write mode when activity is detected in multiple array disks, and unfortunatly this can't be turned off, I did ask Tom to only use this if write mode was set to auto, but no dice, it can't be disabled for now, so your writes are constantly going from reconstruct write to r/m/w, hence why parity and the destination disk also have reads, for the r/m/w part, if you downgrade to v6.7.x. you'll see the expected behavior, but speed won't likely be much better since disk4 will be constantly seeking for the data and parity calculation reads.

 

I don't think so.  I tried setting write to r/m/w and got very DIFFERENT disk activity... the same that I normally see with r/m/w.  Same amount of reads and writes from target and parity, and reads with no writes from source disk... all other disks no activity.

 

[JorgeB]

8 minutes ago, bubbaQ said:

 

I don't think so.  I tried setting write to r/m/w and got very DIFFERENT disk activity... the same that I normally see with r/m/w.  Same amount of reads and writes from target and parity, and reads with no writes from source disk... all other disks no activity.

 

Yes, rmw still works the same, with reconstruct mode (and because you're doing an array to array transfer) you're seeing a combination of both write modes, like mentioned you can confirm it's that by downgrading to v.6.7.2 and repeat that transfer.

[bubbaQ]

2 hours ago, johnnie.black said:

The issue here is that since v6.8.x Unraid reverts to read/modify/write mode when activity is detected in multiple array disks, and unfortunatly this can't be turned off, I did ask Tom to only use this if write mode was set to auto, but no dice, it can't be disabled for now, so your writes are constantly going from reconstruct write to r/m/w, hence why parity and the destination disk also have reads, for the r/m/w part, if you downgrade to v6.7.x. you'll see the expected behavior, but speed won't likely be much better since disk4 will be constantly seeking for the data and parity calculation reads.

Is there a thread somewhere that explains this bug?

[bubbaQ]

Reverted to Version 6.7.2 and turbo-write seems to be working better.  Still some oddness though -- it seems to read for several seconds from the source with no writing to the destination, then stop reading from the source while writing to the destination and reading from the other data disks to reconstruct parity.
 

 

[JorgeB]

7 hours ago, bubbaQ said:

Is there a thread somewhere that explains this bug?

It's not really a bug, it's a feature, though I don't like it either you can turn it off.

 

5 hours ago, bubbaQ said:

Reverted to Version 6.7.2 and turbo-write seems to be working better.  Still some oddness though -- it seems to read for several seconds from the source with no writing to the destination, then stop reading from the source while writing to the destination and reading from the other data disks to reconstruct parity.

That is a bug with v6.7.x, where any array writes starve reads on other array devices.

[bubbaQ]

8 hours ago, johnnie.black said:

It's not really a bug, it's a feature, though I don't like it either you can turn it off.

How can this NOT be a bug?  Both the setting in the GUI and the command line option are broken of you can't set it to always use turbo write.

[JorgeB]

3 minutes ago, bubbaQ said:

How can this NOT be a bug?

Because Tom did on purpose, I also don't agree it's the best option, and I complained about it.

[Squid]

16 hours ago, bubbaQ said:

Is there a thread somewhere that explains this bug?

https://forums.unraid.net/topic/86028-unraid-os-version-68-available/

Quote

Introduced "multi-stream" support:

Reads on devices which are not being written should run at full speed.  In addition, if you have set the md_write_method tunable to "reconstruct write", then while writing, if any read streams are detected, the write method is switched to "read/modifywrite".