I upgraded yesterday from 7.2.x (want to say 7.2.4 but not sure).

Prior to doing the upgrade I ran a parity-check (it had been a couple of months), right from the start it told me it would be just over a day to complete, and it was accurate within a couple of hours.

Post upgrade, I decided to kick off another parity check to generate some heat (I am working on tweaking my fan curves on the host system, I run Unraid in a VM with pass through disk controller). This post upgrade parity check is estimating over 3 days.

It shows it reading my disks at 44MB/s, compared to my typical ~135MB/s. Anyone seeing similar behaviour?

Here are the 7 drives that make up my array, they are all plenty fast enough.

/dev/sdf:

Timing O_DIRECT disk reads: 622 MB in 3.00 seconds = 207.00 MB/sec

/dev/sdg:

Timing O_DIRECT disk reads: 534 MB in 3.02 seconds = 177.02 MB/sec

/dev/sdh:

Timing O_DIRECT disk reads: 648 MB in 3.00 seconds = 215.65 MB/sec

/dev/sdi:

Timing O_DIRECT disk reads: 566 MB in 3.01 seconds = 188.15 MB/sec
/dev/sdj:

Timing O_DIRECT disk reads: 474 MB in 3.01 seconds = 157.45 MB/sec

/dev/sdk:

Timing O_DIRECT disk reads: 438 MB in 3.00 seconds = 145.98 MB/sec

/dev/sdl:

Timing O_DIRECT disk reads: 450 MB in 3.01 seconds = 149.71 MB/sec

Yet:

Two thoughts came to mind.

-- Check that there is no other drive activity taking place on the array, except for the parity check. It doesn't even need to be something the server originates, another system using the array for a network drive could be reading/writing.

-- I see you have a mix of 14TB and 3TB drives. How far into the parity check was it? if it is near the end of checking 3TB, this is likely the slowest speed the parity check sees. 2 things to remember - Drives are their fastest at the first sector, slowest at their last; and (as long as the drives are syncing up nice) the parity check will only run as fast as the slowest drive in the array. Once the 3TB are done being read, I suspect speeds will jump back up significantly.

Also, I would get rid of the 3 3TB drives in the array, and instead of dual parity just use one of the 14TB drives and change the second to be a data drive.

Dual Parity is a bit overkill for 7 total drives (Parity is not a backup). So one parity drive for 4 drives is good, you will have an additional 5TB of storage, and will reduce power consumption, heat and points of possible failure.

best way to see what disk(s) are causing the slowness is to use the disk speed docker container to benchmark them.

Your 3TB disks are likely the reason why, they’re probably older tech and just generally slow overall. Your parity check will likely speed up after 3TB.

4 hours ago, ConnerVT said:

Two thoughts came to mind.

-- Check that there is no other drive activity taking place on the array, except for the parity check. It doesn't even need to be something the server originates, another system using the array for a network drive could be reading/writing.

None, the drives were spun down when I started the parity scan.

4 hours ago, ConnerVT said:

-- I see you have a mix of 14TB and 3TB drives. How far into the parity check was it? if it is near the end of checking 3TB, this is likely the slowest speed the parity check sees. 2 things to remember - Drives are their fastest at the first sector, slowest at their last; and (as long as the drives are syncing up nice) the parity check will only run as fast as the slowest drive in the array. Once the 3TB are done being read, I suspect speeds will jump back up significantly.

Right at the beginning. As you can see in my first reply, the drives test at ~150MB/s I also verified this by copying files between drives (though I didn't share that). Its only the sync that seems to be slow.

4 hours ago, ConnerVT said:

Also, I would get rid of the 3 3TB drives in the array, and instead of dual parity just use one of the 14TB drives and change the second to be a data drive.

The 3TB drives are from my previous array... there are 5 more 3TB drives in the system that are unassigned. They have never been an issue until this upgrade, and I'd prefer to keep them.

4 hours ago, ConnerVT said:

Dual Parity is a bit overkill for 7 total drives (Parity is not a backup). So one parity drive for 4 drives is good, you will have an additional 5TB of storage, and will reduce power consumption, heat and points of possible failure.

I agree that its a bit overkill, but I have been bit before in a Raid 5 array and chose Unraid over a raid 6 array because I could mix drive sizes, allow drives to spin down, and still have comparable redundancy. Drives are cheap

4 hours ago, MowMdown said:

best way to see what disk(s) are causing the slowness is to use the disk speed docker container to benchmark them.

Your 3TB disks are likely the reason why, they’re probably older tech and just generally slow overall. Your parity check will likely speed up after 3TB.

The 3TB drives are slower, I show that in my speed test results... but not 44 Mbps slow.
As I said, I just did a parity scan 2 days ago, and it was running at about 135 Mbps from the start (probably sped up after 3TB, but not by much because it finished within an hour of its initial estimate).
As far as I can tell, the only thing that changed was the upgrade of Unraid. I did upgrade the Proxmox host as well, but I am passing the drive controller through to Proxmox, so the host OS shouldn't have much impact.
Copying files between drives, and performing "hdparm -t --direct /dev/sdX" shows all of the drives with the expected read performance.

Do you see the same speed with the array started in maiantnce mode?

7 hours ago, JorgeB said:

Do you see the same speed with the array started in maiantnce mode?

Slightly faster in maintenance mode, 50MB/s vs 45MB/s, still no where near where it should be.

Looks like I am somehow being CPU limited. When running a sync, the mdunraidd0 process is consuming 100% of one core. This wasn't a problem before the unraid upgrade.

I am using an AMD EPYC 7402P 24-Core, the individual cores aren't super fast, but they don't need to be for this. Should certainly not be pegged by a simple parity calculation. This tells me that something is not right.

It could be something on the Proxmox side, since I did apply updates there at the same time. Though nothing that should have made this large of a difference. I'll keep digging, but would appreciate any input if anyone has seen this, particularly on AMD CPUS.

SOLVED - though I cannot explain how.

After seeing that it was related to a pegged CPU, I shut down my Unraid VM and tweaked it's settings:

Reduced Cores from 12 to 6 (12 was always just a flex anyway but with 48 threads available, why not)
Changed the vIOMMU setting to VirtIO from "Intel (AMD compatible)". I am not running any VMs in Unraid, so I don't think this would even apply anyway.
Increased RAM from 64000 to 65536 (another flex, I should probably reduce it).
Added a TPM State devices - just in case I can use it to get off the thumbdrive.

None of these changes should have had any impact, but clearly they did. Or it was just the reboot?


And with a sync running its only using 30% of one core:

15 hours ago, 2Confused said:

Or it was just the reboot?

Possibly this