My array currently consists of 2 8TB HHDs for parity and 12 4.1 TB SSDs.

 

Yes, I know support for SSDs is only considered "experimental". But before anyone harasses me about that, I would dearly hope that they would consider there's nothing about using SSDs alone that should explain performance THIS BAD.

 

I've definitely seen these drives show much faster read and write speeds. Functioning in the array the read speeds are always high, as high my 2.5 GB wired network allows at around 200-250 MB/s. Write speeds are slower in-array, but still typically in the 50-100 MB/s range. One of the same SSDs used as an "unassigned disk" allows writing at up to 150-250 MB/s.

I recently took all of the disks out of the array configuration, gave them a good TRIM which they can't get while in-array, then reconstructed the array and rebuilt parity. That was a bit on the slow side (it was my first experiment with HDDs for parity along with SSDs for data), but still took less than a day.

 

I've tried some recommended disk settings like this:

 

 

...but to no avail.

It seems to be that there's got to be some crazy kind of bottleneck here, but what that might be I don't know. Blaming individual drive performance doesn't seem to cut it as an explanation for what's happening.

terminus-diagnostics-20250302-1523.zip

No errors logged, suggesting it's just a slow SSD, they likely can write faster for a few minutes, but not sustain that, though 4MB/s is extremely slow, so it could be just be a bad drive. Also, are those some kind of white lable drives or some Chinese brand?

12 hours ago, JorgeB said:

No errors logged, suggesting it's just a slow SSD, they likely can write faster for a few minutes, but not sustain that, though 4MB/s is extremely slow, so it could be just be a bad drive. Also, are those some kind of white lable drives or some Chinese brand?

I think the worse thing about these drives is not the actual performance, but that when something doesn't work well that no one will consider anything else other than it's a cheap drive.

I have a new Crucial brand drive I be willing to give a try, but I'm guessing that if it's just a little bit smaller than the drive it would be replacing (4,000,787,030,016 bytes vs 4,096,805,658,624) that Unraid won't consider using it for rebuilding a drive that had been a even a little bigger. I'd probably have to copy files from an emulated drive 5 to the slightly-smaller drive as an unassigned drive, then swap that drive into position in the array using a new config that keeps all drives as-is, and then rebuild my parity drives instead.

Edited March 3, 20251 yr by RasterEyes

I paused the rebuild, but from the read/write speeds I see, it's still actually going on even though it's supposed to be paused.

 

I have the Crucial drive plugged into an external USB enclosure, and I'm trying to fill it from the emulated content of drive 5 like this:

cp -R /mnt/disk5/* /mnt/disks/TerminusBU2

 

But the same slow read speeds are still happening, the slow write speed to the drive that was being rebuilt continues, and content is being copied to the external drive just as slowly.
 

New diags attached just in case that matters.

 

terminus-diagnostics-20250303-1541.zip

Edited March 3, 20251 yr by RasterEyes

I got tired of waiting so I stopped my array and took the drive I was trying to rebuild to out of the array. I figured I'd just restart with drive missing and copy from the emulated drive to the external drive.

 

Well, a few minutes later and I'm still at "Starting...". Under /mnt none of the array drives have appeared yet. One CPU of four keeps frequently pegging at 100% although I can't figure out what the array is up to with all of that CPU effort.

 

Some more diags for ya.

 

 

terminus-diagnostics-20250303-1603.zip

I shut down through the web interface, but that wasn't enough. I had to force power-down my array.

 

When I started back up I took the drive I was trying to rebuild to out of the array first, figuring I'd start up without it, the former contents emulated.

 

The array started up very fast this time. I think we have to consider that the real reason my rebuild was going so slowly before was some rogue process eating up a lot of CPU time and locking up resources, not the oh-so-easy-to-blame drive with a funny-sounding brand name.

 

Sadly I didn't realize that the emulated contents of drive 5 would only be available via a share, and not /mnt/disk5. That makes copying what would have been on drive 5 to the new Crucial drive quite a challenge unless there's some shortcut I don't know about.

Edited March 3, 20251 yr by RasterEyes

47 minutes ago, RasterEyes said:

I paused the rebuild, but from the read/write speeds I see, it's still actually going on even though it's supposed to be paused.

Linux caches things, you might have paused but there are still probably GBs in cache to flush and that'll take a while if the drive is that slow, need to wait for that to be finished.

 

6 minutes ago, RasterEyes said:

Sadly I didn't not realize that the emulated contents of drive 5 would only be available via a share, and not /mnt/disk5.

It should be available as /mnt/disk5.

 

But yeah it just seems like the typical low end SSD, when the SLC cache is full they can slow down to single digit MB/s.

 

Edited March 3, 20251 yr by Kilrah

8 minutes ago, Kilrah said:

It should be available as /mnt/disk5.

 

Nope. Not there.

 

Quote

But yeah it just seems like the typical low end SSD, when the SLC cache is full they can slow down to single digit MB/s.

 

That's the favorite go-to answer at least, no matter how weird and inconsistent that is with how the drive in question functions when not used in the array.

And that's not normal. The last diags were taken with the array stopped so can't see what could have caused it.

As you know there is no TRIM in the array, that can severely impact performance. You're also not typically doing full drive writes in one go outside of the array.

 

The Chinese no-name isn't the focus, terrible performance on long sequential writes happen on name brands too, some Crucial SSDs are known to be absolutely awful for that.

Edited March 3, 20251 yr by Kilrah

2 minutes ago, Kilrah said:

And that's not normal. The last diags were taken with the array stopped so can't see what could have caused it.

As you know there is no TRIM in the array, that can severely impact performance. You're also not typically doing full drive writes in one go outside of the array.

 

The Chinese no-name isn't the focus, terrible performance on long sequential writes happen on name brands too, some Crucial SSDs are known to be absolutely awful for that.

I've posted diags at a number of points, including with the array started. Here are some more, with the array started, shares and files visible from other computers, but still no /mnt/disk5.

What's more disturbing is this: Although the tooltip for disk 5 says "Device is missing (disabled). Contents emulated"... the contents aren't emulated. I seem to be missing whatever was on that drive, as a comparison to my backup array shows.

terminus-diagnostics-20250303-1651.zip

disk5 doesn't mount because its emulated filesystem is now corrupted, probably from the impatience and forced shutdown. It did mount properly in the previous diags between the start and stop. 

 

https://docs.unraid.net/unraid-os/manual/storage-management/#checking-a-file-system

Even weirder still... I put the very same drive back in place to be rebuilt, and now the rebuild is going MUCH faster (no problem writing quickly to the same drive now!), but /mnt/disk5 still doesn't exist, and I'm pretty sure I'll have to replace a bunch of missing stuff from a back up.

5 hours ago, Kilrah said:

The Chinese no-name isn't the focus, terrible performance on long sequential writes happen on name brands too, some Crucial SSDs are known to be absolutely awful for that.

For the past 25 minutes the very same drive (not just same size and brand, the same drive, now pulled out of my Unraid array) has been happily slurping up data at a steady speed of over 300 MB/s, plugged into my Windows systems via an external USB enclosure.

Are you going to say "Just you wait! It'll drop to a crawl soon!"? Will you surmise that reformatting the drive to NTFS somehow sped up the performance enormously, despite the fact that the drive had been very recently TRIMmed while connected as an unassigned device to my Unraid array, after which it was filled with data just ONCE with little or no re-writing afterward?

The drive never has had to do much more than hold a collection of videos which hardly ever change. With the exception of a little metadata occasionally being updated, and the occasional 2K movie updated to 4K, content mainly just sits on the drives in my array waiting to be viewed. The drives also occasionally receive new movies which are written into bountifully available free space. Further, it's only been maybe two weeks I think since the drive had been TRIMmed, put back into the array, and parity (on HDDs, not SSDs) was rebuilt.

Little or nothing that tends to degrade SSD performance is going on.

Is someone willing to admit that maybe, just maybe, something else is going wrong here that can't be blamed on drive performance?

Windows is currently estimating that the copy of approximately 3.4 TB of data that I'm using for a test will be finished in another 2 hours, 35 minutes. Perfectly inline with a steady 300+ MB/s write speed. Want to take any bets that the estimate doesn't more or less hold right to the bitter end?

Update: The super-fast 300+ MB/s speed lasted for about 50 minutes, dropped for a while, getting as low as a mere 14 MB/s, but has climbed back up to anywhere between 35-200 MB/s. After one hour 1.5 TB (1.38 tebibytes) of data had been copied, so still well over 300 MB/s average for that first hour. Worst case is still looking easily less than a day, nowhere near the crawl I was experiencing trying to rebuild this drive.

If I get surprised by a sudden sustained run of "single digit MB/s", I'll let you know.

Edited March 4, 20251 yr by RasterEyes
Progress update

It took 10 hours, 20 minutes to copy 3.76 TB onto this drive which is being blamed for my terrible rebuild performance, a drive which had an average write speed of 101 MB/s in a test that filled all but about 0.3 TB of the drive's total capacity.

 

Not 1.01 MB/s.

Not 10.1 MB/s.

101 MB/s.

 

Still not worth considering there's something else going on here? An undiagnosed hardware issue apart from drive performance? Bad performance of the checksum procedure, perhaps trying to time data writing of data for synchronizing rotational speeds which don't exist for the SSDs?

 

 

Edited March 4, 20251 yr by RasterEyes
Fix typo

1 hour ago, RasterEyes said:

Still not worth considering there's something else going on here?

Yes, the problem could also be, for example with the reads, or a controller bottleneck, run both the disk and controller tests with diskspeed docker container and post the results.

9 hours ago, JorgeB said:

Yes, the problem could also be, for example with the reads, or a controller bottleneck, run both the disk and controller tests with diskspeed docker container and post the results.

I'll wait until parity is rebuilt before trying the diskspeed thing... although if I'd known about diskspeed earlier I'd have tried it much sooner. Oddly, I never saw that come up in any of the searches I made looking for answers to my slow drive reconstruction problem.

I've been concentrating on getting data back onto the drive that was at first being emulated, and then ceased to even be emulated -- because I guess something went wrong when I tried to pause reconstruction and it wouldn't actually pause. There was nothing on screen to warn me that the UI would falsely tell me that reconstruction had paused while drive activity made it look like reconstruction was still going, and that I should possibly wait... several minutes (which I did do) or possibly hours (which I didn't do) for the task that was supposedly paused to actually come to a halt.

It still seems odd to me, at any rate, even if my forced shutdown turned out to be a bad idea, that the data on all array drives but the emulated drive, still unmodified, wouldn't continue function to recreate the missing drive. Ah, well. Live and learn.

I eventually started my array up with both parity drives taken out of the array and a freshly formatted drive to replace the failed drive. (Reconstruction was never going to work on this particular new drive because it was ever so slightly smaller than the one it was replacing.) I then rsync-restored every file no longer found in this parity-less array from back up, files which mostly went onto the new blank Crucial drive.

Mostly.

And this is where I think a controller issue might be indicated.

Even with no parity operations possible, writing to the new drive was still sluggish, in the 25 MB/s to 50 MB/s range. Not anywhere as bad as the slow reconstruction that had been going on, but still far from ideal. I couldn't blame parity generation for this slowness.

At a certain point, however, new data stopped being directed to the new Crucial drive, but rather onto one of the old Fikwot drives that had been such easy scapegoats. Writing speed picked up greatly. It's hard to know how fast it averaged because I didn't time it, and the displayed speed wildly bobbed around from as low as 25 to as high as 400. Suffice to say my file restoration wrapped up much faster than it had started out.

Once all files were back in place I stopped the array, added back the parity drives, started up again... and that's where I'm going to be for a while yet, with parity being reconstructed, at a too-slow but not-at-as-hideously-slow speed as drive reconstruction had been going.

I'll be very curious to find out if one or more drives are simply running on slow controllers. Some drives are plugged directly into the motherboard, while others are plugged into two different added-on SATA cards. Maybe some of those SATA connections suck.

Edited March 4, 20251 yr by RasterEyes
A little rewording for clarity

I'm trying to benchmark my array with diskspeed, but for at the least a half an hour, as I fruitlessly searched for a power adapter for an old Raspberry Pi 3B+, diskspeed has been stuck in this "waiting" state:

 

 

diskspeed works fine, by the way, on my all-HDD backup array.

Edited March 5, 20251 yr by RasterEyes
By the way...

Is there still some i/o going in the devices?