For the last 6 weeks or so I've been working through a series of server upgrades (including some intentional downgrades too) and in doing so made the ever present error of changing more than one thing at once, chasing efficiency and time management at the expense of knowing exactly what change has had what impact.

Overall the project has been successful, and isn't that exciting so I won't bore you with irrelevance. What I have learned though might be useful for others in that I can't really seem to find much documentation on this anywhere else.

I should also stress that my conclusions are based on my best efforts to identify what's going on here. I am not really an expert so maybe they're wrong, and if so I've no doubt you wonderful people will correct me ;). In fact I'd love for this to be the case as it would suggest there's a way out of the hole I've dug for myself!

I now run a 30 drive Unraid Array utilising 2 Parity drives that I've just upgraded to be Seagate Exos 28TB Factory Recertified CMR HAMR drives (ST28000NM000C), available fairly readily at the moment for a good price/TB especially considering the current market. The remainder of the array is comprised of random 10TB drives and a number of 16TB Exos, with 1x 28TB HAMR drive waiting in the wings to replaced the oldest 10TB imminently...

I failed to do much research on this in advance, and honestly even if I have I'm not sure I'd have found anything to put me off. These are after all CMR drives, not the dreaded SMR so I thought that was box ticked. I needed to increase capacity, saw a great deal on these 28TB options so went for it. My experience now says otherwise, and (if I'm concluding correctly), I'd warn the community about considering current generation HAMR drives for Unraid arrays, and ESPECIALLY not for Parity drives.

My understanding is that the key property of HAMR, the use of lasers to heat the platter before writing data to it enabling higher density, comes at a significant latency cost. Latency that, for most sequential write operations such as of large files, has no material impact. It does however appear to be a serious problem when it comes to something like writing parity data for Unraid Arrays, and I would assume any other type of parity file system e.g. ZFS RaidZ1/Z2/Z3.

Historically I could rely on any Parity rebuild operation, or writing data to the array generally (of course also therefore writing parity) to run at around 150-200MB/s no problem... but since "upgrading" to these lovely large drives, my Parity rebuild is now running at more like 25MB/S, meaning estimates for completion for 28TB drives is looking like around 12 days to complete.

As far as I can tell this is nothing to do with the rest of the chain... I have 9305-16e HBA's running to NetApp DS4246 Disk Shelves where bandwidth should be no problem - and certainly not this slow even if it was a bottleneck. In fact I have several ZFS pools on the same hardware that's having no problem with write speeds at all.. All firmware is updated, testing the individual speed of any given drive using something like IOSTAT shows high 200/250MB/s write speeds as you expect... It's just that the very specific nature of writing Parity data seems to just be killed by these things, meaning all writes to the array suffer the same fate.

I've investigated everything I can, including posing the problem to Claude and going over every suggested root cause and test it could think of, only for the same conclusion to be reached.

These HAMR drives, perhaps all HAMR drives, are not suitable for parity operations if you're ever at all concerned about write performance, which of course becomes more of a problem the larger the drives become.

My conclusion and advice is therefore to avoid until something changes, or unless one of you fine people somehow tell me the cause is something else!

For now I'm stuck with it until I can find non-HAMR drives of a similar size at an affordable and justifiable price to swap them out. They'll be kinda ok as normal Data drives which are overall rarely written to in my use case, and read performance is no problem at all.

Good luck out there!

Edited May 19May 19 by Ph1lo

Thanks for the detailed write-up.

However, I think the broad conclusion that HAMR drives are unsuitable for Unraid parity is premature.

Many users are running the newer Seagate Exos HAMR drives (22–28TB) with rebuild speeds in the 120–180+ MB/s range.

Since you mentioned that individual drive tests show normal speeds (~200-250 MB/s), the bottleneck is almost certainly not the HAMR drives themselves. Something in your specific setup is limiting performance during parity operations.

A few things worth investigating:

1. What HBAs/controllers are the parity drives connected to?

2. Are the drives and HBA getting significantly hotter during the rebuild compared to normal operation?

3. Any difference if you temporarily move one HAMR parity drive to a different HBA/slot?

4. Have you tested sustained sequential writes directly on those drives while the array is active?

Changing multiple things at once (as you noted) makes it hard to pinpoint the real cause.

It’s very possible the slowdown is related to one of the other changes in your setup rather than the HAMR technology.

25MB/s sustained on drives capable of 250MB/s individual throughput is a configuration or infrastructure problem.

HAMR physics cannot explain a 10x throughput reduction that only manifests during parity operations on a specific platform while ZFS on the same hardware performs normally.

The conclusion is almost certainly wrong.

Hi.. Thanks for replying. I'd love it if you're right!

Relevant detailed setup is as follows:

LSI 9305-16e HBA. 4 external ports, each plugged into individual Netapp DS4246 Disk Shelves (single cable into single IOM6), providing total drive capacity of 96 Drives (not all populated).
*I have also just installed a second identical HBA and divided the disk shelves evenly between them, 2 on each HBA, but that hasn't made any difference - I didn't expect it to but I had the card so thought I'd give it a go.

The 30x Unraid Array drives are installed in just 2 of the disk shelves - 24 in one, 6 in the other. Other disks are then used for ZFS pools, most of which are spun down and idle 90% of the time.

The whole system is rack mounted in a small room which is sealed and air-conditioned. Drive temps range from 26 to 34 as I look at it now and it's been running for 2 days so far. The 2x Parity HAMR drives are currently at 33.

I have not tried moving any of the drives to different slots. I have this morning just realised that for some reason a couple of the drives have negotiated only at 3GB/s instead of 6GB/s, including one of the Parity drives. This absolutely explains why the parity process has been even slower than it was the last time I did it. I have now resolved that problem but it still only gets me to Parity write speeds of around 40MB/s during Parity check/rebuild instead of the 150-200 it used to be.

If you have other suggestions as to the problem I'd love to hear it!

Not suggesting this should be considered 100% fool proof, but as it's beyond my capability to do manually I also asked Claude to analyse the issue and look at IOSTAT data whilst the correcting Parity Check is running. It's analysis is as follows (it has always suggested HAMR is the issue here):

This data is very revealing. Let me focus on the key findings.

sdn (parity 2) is your bottleneck — confirmed:

• %util: 99.80-99.85% consistently across all samples — completely saturated

• It's doing both reads AND writes simultaneously: ~35 MB/s read + ~35-37 MB/s write

• w_await: 33-49ms — elevated, consistent with HAMR write latency

• aqu-sz: 7-8.7 — large queue depth, requests backing up

sdm (parity 1) is fine:

• Read-only during this phase, 71-78% utilization

• r_await: 44-58ms — reads are also a bit slow but not the bottleneck

The data drives:

• Read-only, low utilization, r_await varies 12-56ms across drives

• Not the bottleneck

What this tells us:

The HAMR conclusion is confirmed, but more precisely than before. It's not just write latency in isolation — it's that sdn is being asked to both read (for parity comparison) and write (correcting errors) simultaneously, and with HAMR's elevated w_await of 33-49ms it simply can't keep up, hitting 100% utilization and becoming the hard ceiling for the entire operation.

A CMR drive in the same position would have w_await of perhaps 8-15ms, meaning it wouldn't saturate at 100% and the overall throughput would be significantly higher.

So HAMR is genuinely the root cause — the data confirms it directly. The 38-40 MB/s ceiling you're seeing is essentially sdn's maximum sustainable write throughput under load.

57 minutes ago, Ph1lo said:

It's doing both reads AND writes simultaneously: ~35 MB/s read + ~35-37 MB/s write

There's your explanation, this is actually not significantly different with "normal" drives. Anytime there's a read and a write at the same time 30-60ish is normal.

It's avoiding that that's the key and has always been on unraid.

Thanks. Whilst I would agree with you, this is a little misleading and I can't prove it until this Parity process runs it's course. Whilst this might explain what's happening right now, as I'm doing a corrective parity check rather than a full parity disk rebuild, I'm certain I used to see writing data to the array at high speeds, but since installing these drives as my Parity drives that is now severely impacted.

I'll have to come back to this in 10 days or so with some test results - I don't want to interrupt the parity process right now as it's finding plenty of corrections to make for some reason and I want the confidence that the data is protected as best it can be.

Before letting this run for another 10 days, I’d strongly suggest checking the syslog (/var/log/syslog) for hardware errors -- especially SAS/ATA resets, CRC errors, or link speed drops.

If you see those, stopping the check and fixing the underlying hardware issue first would be safer.

Edited May 19May 19 by Lolight

Correcting check typically would only have at most a few blocks to change, if you know parity is invalid and there will be lots to correct it makes no sense to run a correcting check instead of a full sync as like you noticed it'll be a hell of a lot slower with no benefit.

8 minutes ago, Lolight said:

Before letting this run for another 10 days, I’d strongly suggest checking the syslog (/var/log/syslog) for hardware errors -- especially SAS/ATA resets, CRC errors, or link speed drops.

If you see those, stopping the check and fixing the underlying hardware issue first would be safer.

Good advice thank you. I've done so and can confirm that there's nothing of the sort in the logs for the last 3 hours (since I restarted the array after resolving the 3GB->6GB Link negotiation errors on a couple of drives).

3 minutes ago, Kilrah said:

Correcting check typically would only have at most a few blocks to change, if you know parity is invalid and there will be lots to correct it makes no sense to run a correcting check instead of a full sync as like you noticed it'll be a hell of a lot slower with no benefit.

Yes, 16% through and 43Million errors, it's clear that somehow that second parity drive is significantly wrong.

I've therefore started a clean parity build after all, and just hope everything survives the process!

Write speeds are now slightly better, but still slow at only 45-48MB/s against each of the two Parity drives with all other drive activity reading of course. Significantly slower than the last time I rebuilt parity which was around 150MB/s for the duration.

As the Parity rebuild has made progress and got past the 10TB point, lots of my drives are now off. So I'm down to 7 data drives and the 2x28TB HAMR drives still going, yet the overall speed hasn't changed - still going at about 45MB/s per drive. It's therefore conclusively not anything to do with any kind of read bandwidth issue.

Again tried seeing if Claude could come up with any explanations - it's still pretty certain the HAMR drives are the issue, made worse as I have 2 of them for parity it seems. Without me swapping out the HAMR drives for others I can't prove this either way, and that's not something I'm able to do now or likely in the particularly near future!

Claude:

With only 7 drives actively reading and the speed unchanged, the bandwidth saturation theory is definitively ruled out. The aggregate read traffic from 7 drives at ~35 MB/s each is only ~245 MB/s — a tiny fraction of what a single SAS cable can handle. Yet parity write speed is still the same.

So we're back to the HAMR write latency being the sole bottleneck, but let's think about this more carefully than before.

The parity engine has to:

1. Read a stripe from all active data drives

2. XOR compute parity

3. Write to both parity drives simultaneously

4. Wait for both writes to complete before advancing

That last point is key — both parity drives must confirm their write before the next stripe begins. You have two HAMR parity drives, and the engine can't advance until the slower of the two completes each write. So you're not just bottlenecked by one HAMR drive's write latency — you're bottlenecked by whichever of the two is slower on each stripe, every stripe, across the entire 28TB.

This is actually worse than a single parity setup would be. With dual parity the write latency of the two drives effectively compounds because you're always waiting for the laggard on each stripe.

The iostat data confirmed this earlier — sdn was at 100% utilisation and w_await of 33-49ms. That's your ceiling and no amount of reducing read traffic will change it.

Short of replacing the parity drives there's genuinely nothing left to tune here. It will take as long as it takes.

Having completed the dual parity drive rebuild, I'm now upgrading a data disk from 10TB to my 3rd 28TB HAMR drive. The data drive rebuild is performing exactly the same - writing at around 45MB/s.

Once this completes in a week or so, I then have 2 x 16TB CMR disks to use to upgrade 2X10b data drives. I expect they'll run much faster, and if so that will somewhat definitively prove the HAMR drive write performance issue.

Unfortunately, there is something with your system that isn't right. Maybe you got a really bad drive or something else.

For comparison point, here is my experience with Seagate 28TB drives of type says they are perfectly fine and speedy for Parity and Data. Using 2 28TB drives, 1 parity, and 1 data. I recently went through a full parity build and have had several months of normal use. Both drives models "ST28000NT000" which are identified Seagate IronWolf Pro 28TB (model ST28000NT000) built using HAMR (Heat-Assisted Magnetic Recording) technology. The initial parity build was very fast. The monthly parity checks are fast too.

Here's the summary view from the Parity Operations window:

Scheduled Non-Correcting Parity-Check

2026-05-04, 05:26:05 (Monday)

28 TB

1 day, 10 hr, 55 min, 49 sec

222.7 MB/s

I