Ph1lo

Having completed the dual parity drive rebuild, I'm now upgrading a data disk from 10TB to my 3rd 28TB HAMR drive. The data drive rebuild is performing exactly the same - writing at around 45MB/s. Once this completes in a week or so, I then have 2 x 16TB CMR disks to use to upgrade 2X10b data drives. I expect they'll run much faster, and if so that will somewhat definitively prove the HAMR drive write performance issue.

As the Parity rebuild has made progress and got past the 10TB point, lots of my drives are now off. So I'm down to 7 data drives and the 2x28TB HAMR drives still going, yet the overall speed hasn't changed - still going at about 45MB/s per drive. It's therefore conclusively not anything to do with any kind of read bandwidth issue. Again tried seeing if Claude could come up with any explanations - it's still pretty certain the HAMR drives are the issue, made worse as I have 2 of them for parity it seems. Without me swapping out the HAMR drives for others I can't prove this either way, and that's not something I'm able to do now or likely in the particularly near future! Claude: With only 7 drives actively reading and the speed unchanged, the bandwidth saturation theory is definitively ruled out. The aggregate read traffic from 7 drives at ~35 MB/s each is only ~245 MB/s — a tiny fraction of what a single SAS cable can handle. Yet parity write speed is still the same. So we're back to the HAMR write latency being the sole bottleneck, but let's think about this more carefully than before. The parity engine has to: Read a stripe from all active data drives XOR compute parity Write to both parity drives simultaneously Wait for both writes to complete before advancing That last point is key — both parity drives must confirm their write before the next stripe begins. You have two HAMR parity drives, and the engine can't advance until the slower of the two completes each write. So you're not just bottlenecked by one HAMR drive's write latency — you're bottlenecked by whichever of the two is slower on each stripe, every stripe, across the entire 28TB. This is actually worse than a single parity setup would be. With dual parity the write latency of the two drives effectively compounds because you're always waiting for the laggard on each stripe. The iostat data confirmed this earlier — sdn was at 100% utilisation and w_await of 33-49ms. That's your ceiling and no amount of reducing read traffic will change it. Short of replacing the parity drives there's genuinely nothing left to tune here. It will take as long as it takes.

Yes, 16% through and 43Million errors, it's clear that somehow that second parity drive is significantly wrong. I've therefore started a clean parity build after all, and just hope everything survives the process! Write speeds are now slightly better, but still slow at only 45-48MB/s against each of the two Parity drives with all other drive activity reading of course. Significantly slower than the last time I rebuilt parity which was around 150MB/s for the duration.

Good advice thank you. I've done so and can confirm that there's nothing of the sort in the logs for the last 3 hours (since I restarted the array after resolving the 3GB->6GB Link negotiation errors on a couple of drives).

Thanks. Whilst I would agree with you, this is a little misleading and I can't prove it until this Parity process runs it's course. Whilst this might explain what's happening right now, as I'm doing a corrective parity check rather than a full parity disk rebuild, I'm certain I used to see writing data to the array at high speeds, but since installing these drives as my Parity drives that is now severely impacted. I'll have to come back to this in 10 days or so with some test results - I don't want to interrupt the parity process right now as it's finding plenty of corrections to make for some reason and I want the confidence that the data is protected as best it can be.

Not suggesting this should be considered 100% fool proof, but as it's beyond my capability to do manually I also asked Claude to analyse the issue and look at IOSTAT data whilst the correcting Parity Check is running. It's analysis is as follows (it has always suggested HAMR is the issue here): This data is very revealing. Let me focus on the key findings. sdn (parity 2) is your bottleneck — confirmed: %util: 99.80-99.85% consistently across all samples — completely saturated It's doing both reads AND writes simultaneously: ~35 MB/s read + ~35-37 MB/s write w_await: 33-49ms — elevated, consistent with HAMR write latency aqu-sz: 7-8.7 — large queue depth, requests backing up sdm (parity 1) is fine: Read-only during this phase, 71-78% utilization r_await: 44-58ms — reads are also a bit slow but not the bottleneck The data drives: Read-only, low utilization, r_await varies 12-56ms across drives Not the bottleneck What this tells us: The HAMR conclusion is confirmed, but more precisely than before. It's not just write latency in isolation — it's that sdn is being asked to both read (for parity comparison) and write (correcting errors) simultaneously, and with HAMR's elevated w_await of 33-49ms it simply can't keep up, hitting 100% utilization and becoming the hard ceiling for the entire operation. A CMR drive in the same position would have w_await of perhaps 8-15ms, meaning it wouldn't saturate at 100% and the overall throughput would be significantly higher. So HAMR is genuinely the root cause — the data confirms it directly. The 38-40 MB/s ceiling you're seeing is essentially sdn's maximum sustainable write throughput under load.

Hi.. Thanks for replying. I'd love it if you're right! Relevant detailed setup is as follows: LSI 9305-16e HBA. 4 external ports, each plugged into individual Netapp DS4246 Disk Shelves (single cable into single IOM6), providing total drive capacity of 96 Drives (not all populated). *I have also just installed a second identical HBA and divided the disk shelves evenly between them, 2 on each HBA, but that hasn't made any difference - I didn't expect it to but I had the card so thought I'd give it a go. The 30x Unraid Array drives are installed in just 2 of the disk shelves - 24 in one, 6 in the other. Other disks are then used for ZFS pools, most of which are spun down and idle 90% of the time. The whole system is rack mounted in a small room which is sealed and air-conditioned. Drive temps range from 26 to 34 as I look at it now and it's been running for 2 days so far. The 2x Parity HAMR drives are currently at 33. I have not tried moving any of the drives to different slots. I have this morning just realised that for some reason a couple of the drives have negotiated only at 3GB/s instead of 6GB/s, including one of the Parity drives. This absolutely explains why the parity process has been even slower than it was the last time I did it. I have now resolved that problem but it still only gets me to Parity write speeds of around 40MB/s during Parity check/rebuild instead of the 150-200 it used to be. If you have other suggestions as to the problem I'd love to hear it!

For the last 6 weeks or so I've been working through a series of server upgrades (including some intentional downgrades too) and in doing so made the ever present error of changing more than one thing at once, chasing efficiency and time management at the expense of knowing exactly what change has had what impact. Overall the project has been successful, and isn't that exciting so I won't bore you with irrelevance. What I have learned though might be useful for others in that I can't really seem to find much documentation on this anywhere else. I should also stress that my conclusions are based on my best efforts to identify what's going on here. I am not really an expert so maybe they're wrong, and if so I've no doubt you wonderful people will correct me ;). In fact I'd love for this to be the case as it would suggest there's a way out of the hole I've dug for myself! I now run a 30 drive Unraid Array utilising 2 Parity drives that I've just upgraded to be Seagate Exos 28TB Factory Recertified CMR HAMR drives (ST28000NM000C), available fairly readily at the moment for a good price/TB especially considering the current market. The remainder of the array is comprised of random 10TB drives and a number of 16TB Exos, with 1x 28TB HAMR drive waiting in the wings to replaced the oldest 10TB imminently... I failed to do much research on this in advance, and honestly even if I have I'm not sure I'd have found anything to put me off. These are after all CMR drives, not the dreaded SMR so I thought that was box ticked. I needed to increase capacity, saw a great deal on these 28TB options so went for it. My experience now says otherwise, and (if I'm concluding correctly), I'd warn the community about considering current generation HAMR drives for Unraid arrays, and ESPECIALLY not for Parity drives. My understanding is that the key property of HAMR, the use of lasers to heat the platter before writing data to it enabling higher density, comes at a significant latency cost. Latency that, for most sequential write operations such as of large files, has no material impact. It does however appear to be a serious problem when it comes to something like writing parity data for Unraid Arrays, and I would assume any other type of parity file system e.g. ZFS RaidZ1/Z2/Z3. Historically I could rely on any Parity rebuild operation, or writing data to the array generally (of course also therefore writing parity) to run at around 150-200MB/s no problem... but since "upgrading" to these lovely large drives, my Parity rebuild is now running at more like 25MB/S, meaning estimates for completion for 28TB drives is looking like around 12 days to complete. As far as I can tell this is nothing to do with the rest of the chain... I have 9305-16e HBA's running to NetApp DS4246 Disk Shelves where bandwidth should be no problem - and certainly not this slow even if it was a bottleneck. In fact I have several ZFS pools on the same hardware that's having no problem with write speeds at all.. All firmware is updated, testing the individual speed of any given drive using something like IOSTAT shows high 200/250MB/s write speeds as you expect... It's just that the very specific nature of writing Parity data seems to just be killed by these things, meaning all writes to the array suffer the same fate. I've investigated everything I can, including posing the problem to Claude and going over every suggested root cause and test it could think of, only for the same conclusion to be reached. These HAMR drives, perhaps all HAMR drives, are not suitable for parity operations if you're ever at all concerned about write performance, which of course becomes more of a problem the larger the drives become. My conclusion and advice is therefore to avoid until something changes, or unless one of you fine people somehow tell me the cause is something else! For now I'm stuck with it until I can find non-HAMR drives of a similar size at an affordable and justifiable price to swap them out. They'll be kinda ok as normal Data drives which are overall rarely written to in my use case, and read performance is no problem at all. Good luck out there!

Parts all finally arrived. Connecting shelves 2 & 3 directly to my SAS expander worked first time, so whatever problem I was having, it was specific to the daisy-chain setup of the 3 disk-shelves.

Thanks... The expander is necessary as I use all the ports on my HBA internally already. It's frustating as everything I read says that this should work just fine - even other people that appear to have done it almost exactly the same way. However it is true to say that I can do it differently by taking internal SAS connections that are availabel on the expander, using an adapter much like you have suggested, and effectively connecting each of the 3 DS4246 directly to the Server rather than chained together. It's pretty cost neautral as I can return all of the external cabling - seems highly likely that it'll work seeing as I can get 1 of them to work this way... It's just really annoying as it shouldn't be necessary

Thanks! This much I'm confident on, I'm sure they are wired correctly.

Thanks - I'll take a look.

Hi. Thanks for trying to help. I've not done this now, honestly I wouldn't know how. When the PC boots there's no obvious BIOS being shown for the HBA - though I've seen online screenshots to suggest that I should see something. Perhaps that means it's already disabled?

Hi, I have an AMD 9950X based "server" in a 4U Chassis, running the LSI 9305-24i HBA which, until earlier this week, has been running 24 HDD's without issue for a long time. I also have 2 other machines simialr to this, but the processing power I was using them for is no longer required. In an effort to slightly reduce power consumption and sell of still valuable hardware, I wanted to go down to 1 server but retain all my storage - and allow room for futher expansion. So I bought 3 x Netapp DS4246 Disk Shelves and associated cables and an Adaptec AEC-82885T 2283400-R SAS Expander, having never used such things before. I've connected the 9305-24i to the AEC-82885T with a single SAS cable, which then in turn runs back to the internal backplane of the main chassis keeping all those drives connected, and then externally to the DS4246's. My problem is daisy-chaining. Everything is working fine, as long as I only connect 1 of the D4246's. Each of them work fine independantly, all 48 disks are found (24 internal, 24 external), but as soon as I power on any of the other shelves the PC just hangs pre-boot. If I boot with just one of them on and then turn on the others, no problem as such, but they are not detected. Is the AEC-82885T not suitable for chaining 3x DS4246's behind one of it's single external ports (single path config - keeping things nice and simple)? If so is there another similar card that you'd reccomend? I have no spare PCIe ports on my MoBo and my HBA has no external ports so whatever it is needs to allow for molex or sata power like this Adaptec does... Thanks!

It has now been over a month since this last happened. I changed nothing, have installed no relevant updates, it's just magically stopped doing it in spite of having done it consistently for several weeks prior to that. Weird, but great news..

Hi, I have 3 Unraid servers, all fully up to date with Stable release channel and all assocaited dockers and plugins also upt to date. I have an unassigned device (a single SSD) on Server 1, which is shared and then connected to via an auto-mount SMB Mount on both of the other 2 Unraid servers. Everything has worked fine for the last year or more, until the last maybe 6 or 8 weeks. Now, consistently after 7 days (it happens somewhere on the 8th day, not precisely as the 7th day ticks over), the two "slave" Unraid Servers can no longer access the SMB share on the 1st - however it's weirder than simply not working. The link still works in the Dashboard of the 2 servers, but it doesn't take me to the correct location. Clicking on the link in dashboard says it is taking me to "Index of /mnt/remotes/SHARENAME" but that is not at all the folders that it is then displaying. It is instead showing me the local filesystem for the server in question (2 or 3) i.e. i can access, and click through, bin, dev, etc, mnt and so on, all of which shows the correct content for the local device. The share itself on server 1 still works, whatever the problem is occurs on the other 2 simultaneously. I know this as I can access the share itself just fine from any other machine on the network (windows and linux), and my resolution involves forcibly rebooting both slave Unraid servers - no need to touch the master if that makes sense). And I say "forcably" rebooting, as it is more often than not, not a simple task. The shutdown and/or reboot icons normally used from the dashboard almost always do nothing in this scenario. I occaisionally have success by using the "powerdown -r" command from terminal, but that is also unreliable. Sometimes I'm forced to actually cut the power to the server in order to get it to restart and then resolve the issue for another 7 days. It's driving me mad, and no doubt not doing my array much good having to keep being shutdown like this. Any advice would be most appreciated! Thanks...