I've got 2 parity disks (16TB Seagate ST16000NM001G's) connected to the main board (supermicro X10SLM+-F SATA 6G ports), and various array disks (shucked WDs and a few Seagates) connected to Norco 4224 backplane via flashed m1015 and RES2SV240 expander.  A while back I upgraded the parity disks to those 16TBs. Adding first went really slow for rebuild (26 MB/s), but the second parity went around 220 MB/s (as it had been doing regularly before the upgrade).  Since added a couple of 16TB seagates and seeing the 26MB/s again during parity checks and now a rebuild.

 

All disks are CMR, verified write cache on for everything with sdparm, m1015 has no error LEDs or anything in the log i can see and is in the 8x slot.  Turbo write doesn't seem to do anything but i tried toggling it off auto anyways. Nada. Overall reads during the parity check/rebuild show at 500 MB/s and 26MB/s write to the disk. When writing to/between unassigned disks (attached via m1015/RES/backplane, i get 250 MB/s writes. Is 500MB/s some cap on my HBA?

 

I thought maybe a breakout cable is having issues but i don't see anything obvious, or possibly the PCI slot is at a slower speed (nothing in BIOS to manually set 8x that was obvious to me).  Everything is snug in its slot or connector. All the above is what i've been able to figure out to check from the forums here, however now i'm stumped.  Anything else i should be checking?

wolfwood-diagnostics-20230325-1650.zip

Edited March 25, 20233 yr by Glycerine

So not an expert at this so don't take my word as doctrine, but if I'm not mistaken I believe you have SATA drives not SAS drives if I'm correct. If so SAS drives can both read and write at the same time, while as SATA can not. So if your updating your parity drive, all the drives in your array only have to read, and then the parity drive has to write. But if your trying to update a drive in your array then it has to both read and write, which can cause it to slow down as the one needle has to go back and forth between reading and writing. I'm the same when moving between drives I can get 250mb/s but when parity check is running it will be 32mb/s. So I think your normal. It just has to do with the fact all your drives are trying to read at the same time, combined with the fact they can't read and write at the same time. Second paritys don't take as long as it's basically duplicating the first parity disk, so it's like your transferring between drives. Again not an expert at this, but I have the same speeds as you and my system is a lot bigger, so I believe it's normal. 

There appears to be something else reading from disk13 at least, stop docker/VM service and make sure nothing else is accessing the array.

16 hours ago, Tindog said:

I'm the same when moving between drives I can get 250mb/s but when parity check is running it will be 32mb/s. So I think your normal. It just has to do with the fact all your drives are trying to read at the same time, combined with the fact they can't read and write at the same time.

That's what I was kinda wondering. If i hit some sort of limit on the number of array drives.  It's just seems like an odd coincidence that 500MB/s is the top, since that is the top speed of PCI 1x.  If it was at 8x, it should be giving me the numbers i got before.  I think my m1015 is silently failing.  Got 850W PSU so it should have enough juice for everything and then some. Got another m1015 on the way to test and will update afterwards, but i think you and I might be in the same boat.  Hopefully this fixes it and i won't have week long parity checks any more

On 3/26/2023 at 7:50 AM, JorgeB said:

There appears to be something else reading from disk13 at least, stop docker/VM service and make sure nothing else is accessing the array.

Hmm nothing should be writing to the array. Did you mean disk12? That's the data drive that's being rebuilt.  I've shutdown the VM service. Docker shouldn't be writing to the array but it does have a process going with an unassigned drive at the moment. Will shut that down too when complete to see if there's any effect. Thanks!

 

EDIT: Disabling Docker/VM had no effect. I really think it's something to do with the HBA.  Waiting on a replacement to confirm...

Edited March 27, 20233 yr by Glycerine

Well it's finally completed its rebuild and i was able to poke around once more.  It's definitely something in the card.  BIOS reports the M1015 as 1X PCIe.  That jives with what i saw at the end of the rebuild, where it was reading from one device and write speeds went up dramatically, but total read/write stayed within that 500MB/s total.

 

At this point, i think the card is roasted unless there's other ideas?  I have a new one coming this weekend and should be able to test then.

You can try a different PCIe slot if available, but yeah, an x1 link will cripple the bandwidth.

Moved the card over a slot and got 8x in BIOS, but speeds were still wonky.  Placed new HBA card into that same slot, still at 8x, and getting a solid 150MB/s on parity with a total of 2.7GB/s r/w.  I would think this means it's running at 4x since 8x ought to be around 4.5GB/s total unless UNRAID itself is slowing down the reads? Maybe because two parity? 150MB/s seems to be the mark on a majority of parity speeds i've seen here so I'm assuming it's running full speed.  Just over a day estimated to do 16TB dual parity check

 

In either case, i'm chunking this up to HBA HW failure and marking as resolved.