SSD v HDD - reliabity and speed

SSDs and newer NVMEs are seriously cool and FAST compared with older spinning HDDs.

I am looking at options, and running tests as I plan to upgrade my oldest Unraid Server and bring it out of the dark ages (v4.5.3).  It has been a good reliable server, and still is.

for a quick reference point, here are various system speeds:

SATA - 
SATA 1 - 1.5 Gb/sec = 150 MB/sec
SATA 2 - 3.0 Gb/sec = 300 MB/sec
SATA 3 - 6.0 Gb/sec = 600 MB/sec

Hard Drives (typical maximum speeds as of Jan 2024)
5400 RPM up to ? - up to 180-210 MB/sec - 
7200 RPM up to 1030 Mb/sec disc to buffer - up to 255 MB/sec - 204 MB/sec sustained

NETWORK limits 94% efficiency- - 
10Mb = 1.18 MB/sec
100Mb = 11.8 MB/sec
1Gb = 118 MB/sec
2.5Gb = 295 MB/sec
10Gb = 1180 MB/sec

As can be seen quickly looking at the above numbers, HDD can still be used with great throughput results with 1Gb and under networks, and even be fully acceptable on 2.5Gb networks.  They still cost less to purchase, but do consume more energy when running than SSDs/NVMEs.

We will assume a case with decent airflow should be used for any drive array to increase the usable life of the data and reduce the likelihood of data corruption and loss.

For HDDs CMR (Conventional side by side recording) is preferable to SMR (Shingled overlayed recording) storage technologies.

Following are based on my experiences to date:

HDDs typically give advance warning of failure, and from what I have seen, usually can have all or most of the data recovered from them if needed when they go bad.

SSDs seem to usually fail abruptly, with no option to read data from, them once they fail.  I expect the same from NVMEs.

HDDs have a higher likelihood of soft read errors (that can be recovered) than SSDs.  Read errors are not common on SSDs.

HDDs typically are more consistent in transfer speeds than SSDs. 

So my original thoughts were that HDDs may still be the best overall for Unraid use, where the main purpose is to store then stream movies/TV shows, and Music.  SSDs would be faster for Parity Builds, Checks and Drive Rebuilds, but would see little in performance increase for network usage.

While testing different devices and array configurations, I was surprised when I cam upon two 2.5" SSDs from different vendors, that when hot with a large queue of read requests during an array parity build/parity check that the read speed on each device plummeted to 100MB/s and less!  This was in ALL SSD/NVME drive pool tests that started out in the 500MB/s range for a little while.  Removing BOTH of these drives allowed for full array operation at 450MB+ for the full duration of Parity Build/Verification.  Introducing either of the drives into the array dropped sustained speeds back down to the 100MB/s and under speeds.

Now, placing two 4TB WD Black spinning drives in as Parity drives of course limits the array speed for Parity reads and writes to about 250MB/s.  Interestingly this added delay on the array, prevents the two slow SSDs in the previous tests from becoming saturated in read queues, allowing them to complete the full array Parity Build/Verification cycles at 250MB/s!  No array speed change when adding another 4TB WD Black spinner to the data array

Adding another 4TB WD Red Plus, to the data array slows the array to about 200MB/s for parity operations.

These test speeds are consistent when the SATA SSDs and HDDS used with the motherboard or expansion card SATA III ports when not limited by bandwidth such as too many SATA ports going though a PCIe lane.

More tests, data, and numbers to come.