decommission current HDs and move to SSDs

[Laup]

Hi,

I currently have 3 HDs with one being the parity drive. I would now like to replace the HDs with SSDs and move the motherboard etc to a new PC case. Everything is backed-up so can I just unmount all HDs and start again with SSDs? Will unRAID complain that something is missing...? Any particular steps that I should take. Time in not important so I'm happy to start fresh and copy all data from the backup to the new system. Any help appreciated, thanks.

Paul

 

[JorgeB]

3 minutes ago, Laup said:

Will unRAID complain that something is missing...?

Yes, you'll need to do a new config: Tools -> New Config

[SSD]

@Laup -

 

I would be cautious using SSDs for array drives. Here is some background.

 

The basis of unRAID protection is parity. Parity creates redundancy at a very low level - down to the disk sector. And reconstruction is performed at that level as well. Whether a disk sector has useful data or not is irrelevant to unRAID. Every sector contains 512 bytes. Could be all zeros. Could be remnants of a deleted file. Could be part of a real file, or part of the disk's housekeeping. But every sector has 512 bytes. And those 512 bytes don't change unless and until that sector is updated. When a disk is rebuilt, each sector on the disk is restored by looking at the corresponding sector or every other disk. So if the disk is heavily fragmented, when it is rebuilt it is fragmented in the exact same way. It's not like restoring from a backup, where each file is individually written fresh to a new disk. Instead it is a mirror image.

 

SSDs are not the same as hard disks. There are no physical sectors. Sectors are simulated by SSD cells. But each cell on an SSD has a limited number of writes. If one spot is written too many times, that cell will fail. To protect against having individual cells get worn out while the vast majority of the SSD is fine, the SSD has the ability to remap the cells. So while sector 1234 might have been simulated by cell 1000, it is possible that sector 1234 could be changed to be simulated by cell 5555. So long as the SSD preserves the data, the disk will still behave perfectly fine. So in essence the SSD has the smarts to avoid cells failing due to a lot of writes, by invisibly shuffling the cells so that the SSD will wear in a consistent way.

 

The process of shuffling the cells CAN result in unused sectors on the SSD ("unused" meaning they contain no data of value) having their values change. Since the data is unused, who cares that they are not maintained. So unlike a hard disk where the only way a sector can change is by the disk writing to that sector, the SSD sector values can change invisibly if they are unused. And if this happens, parity will be broken and the ability to rebuild any failed disk reliably will be impacted. Use of the TRIM command may be the highest risk, but the TRIM command is important to maintaining the SSD and helping it last as long as possible.

 

Its important to understand this risk. And if you are using SSDs in the array, do some research on the particular model to confirm that even unused sector values will be preserved, making them suitable for use in a RAID or unRAID array.

[JorgeB]

All indications are that SSDs are safe to use as array devices, they can't be trimmed so there might be some write performance loss, but there should be no sync errors, though I recommend doing daily parity checks at least in the beginning, they shouldn't take long anyway, to confirm there are no issues, as long as there are no unexpected sync errors you'll be fine.

[pwm]

7 hours ago, SSD said:

So unlike a hard disk where the only way a sector can change is by the disk writing to that sector, the SSD sector values can change invisibly if they are unused.

 

As long as there is no TRIM, then the drive will perfectly maintain all data visible to the OS.

 

The SSD doesn't understand the file system that the OS operates on. So the SSD have zero knowledge of which bytes are in use and which bytes aren't. So it will assume every byte of every LBA contains important information that needs to be retained however much files are added, modified or removed.

 

Interesting things happens first when you use TRIM. The TRIM command tells the SSD "this address range is not actually in use - feel free to do whatever you want with it".

 

An area tagged by a TRIM need not be maintained when the SSD shuffles data around.

 

This breaks parity in RAID-5, RAID-6. But also breaks RAID-1 mirroring. And it breaks any scheme where a program might compute a hash over parts of the raw device image.

Because of these issues, the SATA standard has been extended so that the SSD can report if it supports reporting consistent data from TRIM:ed regions.

 

hdparm -I from one of my drives:

ATA device, with non-removable media
        Model Number:       INTEL SSDSC2KW240H6
...
           *    Data Set Management TRIM supported (limit 8 blocks)
           *    Deterministic read data after TRIM

And from a different drive:

ATA device, with non-removable media
        Model Number:       Samsung SSD 840 PRO Series
...
           *    Data Set Management TRIM supported (limit 8 blocks)
           *    Deterministic read ZEROs after TRIM

An important issue here - if the drive always reports ZERO for regions that have been TRIM:ed, then an unRAID system would have to read out the old content before issuing the TRIM command. And then update the parity as if a program had explicitly written ZERO to the disk blocks. Because of this, TRIM isn't really a good option together with parity even when the drive supports deterministic results. It's way better to have overprovisioning so the drive can give good performance receiving any TRIM commands. With overprovisioning, the drive can always keep a pool of already erased flash blocks that are ready to be written to, so they don't suffer the huge slowdown some SSD can show when all flash media has been dirtied by writes.

[Laup]

Thanks for the info...

 

Now I'm not sure if I should move to SSD or stay with HDs. I'm planning to use 3 Samsung 512G SATA SSD 860 Pro. On the reseller website it says that it's suitable for NAS but can't find any statement about that on Samsung's website. "PWM" says as long as there is no TRIM, I assume unRAID wouldn't have that option anyway whereas Windows would.

 

Any more thoughts before I take the plunge?

 

Thanks,

Paul

 

[SSD]

@Laup -

 

Not running trim is going to seriously affect performance over time. I had an SSD in my windows box holding VMs and never knew to run trim.  It seemed to be getting slow so i ran some performance benchmarks and sure enough it was very slow - slower than a spinner. Running trim had a dramatic impact on its performance. 

 

I'm not a fan of SSD array.  Why not use an ssd as a UD, And back it up to the array.  Could even set up several SSDs as a cache pool that provides redundancy. Use it instead of the unraid array.

[JorgeB]

6 hours ago, Laup said:

Any more thoughts before I take the plunge?

Like mentioned TRIM won't work so there might be some write performances issues, you can overprovision the drives to try and minimize that, other than that there should be no issues, but run frequent parity checks in the beginning to make sure there are no sync errors, I tested many different SSDs as array devices and almost all worked without issues, but there is at least one model, the Kinsgton SV300, that would cause 2 or 3 sync errors after most power cycles, these errors were always in the very first or last sectors, so unlike to cause any data issues, but still not recommended for use in the array if you don't always get 0 sync errors.

[pwm]

3 hours ago, SSD said:

 

Not running trim is going to seriously affect performance over time.

 

Overprovisioned drives do not slow down over time since they don't have to switch to erase flash blocks at write time - with or without trim, they will always have flash blocks pre-erased and ready to use.

 

The only difference is that without TRIM, there will be a more wear, since the drive doesn't know which LBA that are unused and will have to move a number of LBA that doesn't contain any file data when it rotates data to new flash blocks. But that doesn't really affect the performance, since the main cost for flash devices is the erase cycle.

[SSD]

On 6/20/2018 at 2:33 AM, pwm said:

 

Overprovisioned drives do not slow down over time since they don't have to switch to erase flash blocks at write time - with or without trim, they will always have flash blocks pre-erased and ready to use.

 

The only difference is that without TRIM, there will be a more wear, since the drive doesn't know which LBA that are unused and will have to move a number of LBA that doesn't contain any file data when it rotates data to new flash blocks. But that doesn't really affect the performance, since the main cost for flash devices is the erase cycle.

 

Does unRAID give the option to underprovision array drives?

[JonathanM]

21 minutes ago, SSD said:

 

Does unRAID give the option to underprovision array drives?

No, any overprovisioning would need to be done manually with HPA. Currently array drives are automatically partitioned with a single partition expanded to fill all logical space at each mount. It would be a fairly major rewrite of code logic that has been around since almost the beginning of unraid.

 

I would love to see it added, but let's be honest, most of the unraid additions have been focused on tuning and cleaning up the relatively new additions of docker, vm, and encryption. Base level array drive logic hasn't been touched in a while, the last major thing I remember being done was addition of logic to handle sdXX instead of sdX.

 

(Yes, I know you @SSD are aware of everything I said here, I'm just summarizing and putting some things in context)

[SSD]

24 minutes ago, jonathanm said:

(Yes, I know you @SSD are aware of everything I said here, I'm just summarizing and putting some things in context)

 

Yes and No. I do realize that unRAID would want to use the entire disk as the array disk, and you'd not be able to have it take up a subset without something like an HPA. And I know HPAs very well - I once purposefully created HPAs to reduce several 3T disks to under 2.1T so I could use it with unRAID until support for larger drives was added to unRAID (5.0 beta 8 I think. It was a slight PITA, but allowed me to buy the cheaper/TB 3T disks and use them at reduced capacity initially, without continued investment in the 2T drives. And once support for larger drives was added to unRAID, I removed the HPAs and was able to use the full capacity. Of course had to copy the data off and then copy the data back on, a PITA but not so bad.

 

But what I was not aware of, is if SSDs have some sort of internal "underprovisioning" configuration that would make the SSD appear smaller than it truly is to the OS. I remember when setting up SSDs under Windows being asked about underprovisioning, but never saw anything like that in unRAID. Again, if the underprovisioning is an SSD configuration, its possible unRAID would see the size as smaller, and happily add it to unRAID at that size. This was my question. Are you certain that SSDs don't have this feature?

[JonathanM]

6 minutes ago, SSD said:

But what I was not aware of, is if SSDs have some sort of internal "underprovisioning" configuration that would make the SSD appear smaller than it truly is to the OS. I remember when setting up SSDs under Windows being asked about underprovisioning, but never saw anything like that in unRAID. Again, if the underprovisioning is an SSD configuration, its possible unRAID would see the size as smaller, and happily add it to unRAID at that size. This was my question. Are you certain that SSDs don't have this feature?

This article covers it pretty well.

https://www.seagate.com/tech-insights/ssd-over-provisioning-benefits-master-ti/

In summary, the factory overprovisioning is fixed, and part of the hardware design. Anything done after that, either manually with partitions or with the manufacturers tools, is purely managed in software, and the drive controller only acts on it when told to by the OS. What is NOT clear, and is the sticking point for unraid, is how exactly the drive controller deems an address unused.

 

This quote from the article implies the controller software reads partition tables.

"Actually, even without special software, any user can set aside a portion of the SSD when first setting it up in the system by creating a partition that does not use the drive’s full capacity. This unclaimed space will automatically be used by the controller as dynamic over-provisioning."

 

So, the way I read it, is that if @limetech would give us the ability to keep a specified % of a drive unpartitioned, the whole issue would be solved.

[JorgeB]

For overprovisioning to work you just need to secure erase the SSD first, e.g, with bkldiscard, then create an HPA.

[SSD]

2 minutes ago, johnnie.black said:

For overprovisioning to work you just need to secure erase the SSD first, e.g, with bkldiscard, then create an HPA.

 

I see.

 

Based on my experience with creating HPAs, I found that some controllers allowed the HPAs to be created and others did not. Motherboard ports were the best. Once created, the disk could be moved to any controller and it would recognize/respect the HPA. And once an HPA operation is performed, you had to power down the drive/server in order to do another HPA operation on that disk. below is a thread from 2011 that documents my experiences with the 3T to 2.2T HPA. It gives a lot of the commands and techniques for creating and removing HPAs.

 

 

[pwm]

1 hour ago, jonathanm said:

This quote from the article implies the controller software reads partition tables.

 

It doesn't need to understand partition tables.

The drive can keep track of state for all LBA.

And an LBA that has never received any write is quite similar to an LBA that has been trimmed - the drive knows it has no important information that needs to be maintained.

 

So if only 80% of the LBA are used, then the disk has a lost of 20% LBA that are unused and are backed by flash blocks that have no information that needs to be retained. So these flash blocks can be kept as pre-erased.

 

This means the drive can pick up one of these pre-erased flash blocks and instantly write data to. At the same time, it can issue an erase of the flash block that contained LBA from the used 80% of the drive and rewrite the mapping so this flash block is pooled into the 20% of LBA that are flagged as unused.

 

This also means that an SSD can manage a pool of pre-erased flash blocks even if the paritition is 100%  - but under the condition that the file system itself never uses some of the LBA so that they will remain flagged as empty by the controller.

 

Several file systems have rules that they shouldn't continue to use new LBA with high addresses as long as there are ample with suitable LBA with lower addresses. This was good for HDD since it optimized storage to use fast tracks and reduced the stroke length for the heads. But also means that a drive that is never filled more than 50% can continue to have a rather large percent of LBA that is never written to. So you end up with automatic overprovisioning. But the same moment the drive gets filled enough that the allocation algorithms have written at least once to all LBA then the drive slows down because it must constantly erase flash blocks (unless the drive have a small overprovisioning pool hard-coded from factory) 

[JonathanM]

15 minutes ago, pwm said:

It doesn't need to understand partition tables.

The drive can keep track of state for all LBA.

And an LBA that has never received any write is quite similar to an LBA that has been trimmed - the drive knows it has no important information that needs to be maintained.

 

So, manually repartitioning after the drive has been written to doesn't work all by itself, you have to either blkdiscard first, or issue a trim command after the repartitioning.

 

(I'm speaking to the unraid usage specifically, where the entire partition is guaranteed to have been written once if the drive was added to a parity protected array)

 

Now that I think this through in the context of unraid, I can see where it would be possible to get into big trouble.

 

Blkdiscard all SSD's, and set up a new config. Unraid would partition and format the drives, but the only drive to get fully written would be the parity drive. All the data SSD's would still be relatively untouched, able to be used for block management by their controllers, unbeknownst to unraid.

 

@johnnie.black, did you ever recreate this with your SSD testing?

[JorgeB]

2 minutes ago, jonathanm said:

Blkdiscard all SSD's, and set up a new config. Unraid would partition and format the drives, but the only drive to get fully written would be the parity drive. All the data SSD's would still be relatively untouched, able to be used for block management by their controllers, unbeknownst to unraid.

 

This wouldn't be a problem, parity for the unused sectors would still be 0, when those sectors are changed on the data SSDs, by normal writes, parity is updated accordingly

[JonathanM]

Just now, johnnie.black said:

 

This wouldn't be a problem, parity for the unused sectors would still be 0, when those sectors are changed on the data SSDs, by normal writes, parity is updated accordingly

Yes, but what about the unused sectors that are swapped out for block erasure by the ssd controller? They have never been written to by the OS, only read on the initial parity build.

[JorgeB]

3 minutes ago, jonathanm said:

Yes, but what about the unused sectors that are swapped out for block erasure by the ssd controller?

Those won't be read, any sectors on the HPA won't be read, or am I misunderstanding your question?

[JonathanM]

Just now, johnnie.black said:

Those won't be read, any sectors on the HPA won't be read, or am I misunderstanding your question?

No HPA, just plain empty blkdiscarded SSD added to a new config.

 

43 minutes ago, pwm said:

an LBA that has never received any write is quite similar to an LBA that has been trimmed - the drive knows it has no important information that needs to be maintained.

 

 

[JorgeB]

Just now, jonathanm said:

No HPA, just plain empty blkdiscarded SSD added to a new config.

Aah, I get it now, on the SSDs I tested after running blkdiscard all sectors are 0, e.g., when I need to a new array to test something, I blkdiscard all SSDs, do a new config, check parity is valid and I'm ready, since all SSDs are all zeros parity is in sync.

 

Now, I did read sometime ago, that this might not be true for all SSDs, some may not clear all sectors with blkdiscard, IIRC the Intel 520 was one of them, in those cases you'd need to run a real secure erase, so that the overprovisioning woks correctly, but during a parity sync parity it should still be work correctly, as parity should be built accordingly.

[pwm]

1 hour ago, jonathanm said:

So, manually repartitioning after the drive has been written to doesn't work all by itself, you have to either blkdiscard first, or issue a trim command after the repartitioning.

Exactly. And worse than that - it also takes a quite smart controller to use this trick. So it isn't all SSD that can gain this overprovisioning boost by having a subset of LBA that are never written to. I'm not sure if there is any specific datasheet term or SATA compatibility flag that will describe if the drive can "auto-create" overprovisioning by having some LBA never written.

 

44 minutes ago, johnnie.black said:

Now, I did read sometime ago, that this might not be true for all SSDs, some may not clear all sectors with blkdiscard

 

Notice the two alternative descriptions of result after TRIM from my previous post with hdparam output for two SSD:

On 6/20/2018 at 12:16 AM, pwm said:

Deterministic read data after TRIM

And from other drive:

On 6/20/2018 at 12:16 AM, pwm said:

Deterministic read ZEROs after TRIM

 

So based on hdparm output, there are at least four possible options regarding TRIM support:

1. Not supported.
2. Supported but without deterministic read.
3. Supported with deterministic read.
4. Supported with deterministic read of zero.

My guess is that 3 is similar to 4 but that the drive might instead decide to return 0xff. A constant value so the original content doesn't need to be retained, but the controller has to teach itself what that constant value is while option 4 has strengthen the promise to a pre-defined value.

 

Alternative 2 can't be used with any form of parity or mirroring.

Alternative 3 can probably be used with at least mirroring if identical devices are used so the returned pattern is the same.

Alternative 4 could be also used for parity - but parity must be recomputed as if the trimmed LBA has been zeroed.

But I don't know if any of the above alternatives can be used as indicator for support of "auto-generated" overprovisioning for disks that leaves some LBA never written.

[JorgeB]

10 hours ago, pwm said:

Notice the two alternative descriptions of result after TRIM from my previous post with hdparam output for two SSD:

I was referring that for some SSDs blkdiscard won't trim the entire SSD, not related to the type of trim used, I found where I've read that before:

 

Quote

 

Let's put it this way:

The truth is, in most, if not all, SSDs, ATA Secure Erase is an equivalence to a full device TRIM. Except those with "hardware encryption", where ATA Enhanced Secure Erase is basically a regeneration of the encryption key. So after all ATA Secure Erase in SSDs is not really THAT secure, unless yours support "hardware encryption".

On the other hand, fstrim is not the only way to TRIM a device. You can use blkdiscard to wipe a whole block device (disk/partition) including the GPT and superblock and whatsoever.

However, be aware that TRIM implementation in some of the disks has a "requirement" in the TRIM commands issued, so only when it is fulfilled, all the blocks on the drive will then "read zero" after TRIM.

For example, the Intel 530 SSD requires the TRIM block ranges to be "aligned" to 8 blocks. So when I want to wipe it clean with blkdiscard or hdparm, I will need to make sure that no "minimum unit" is "covered" by two TRIM ranges.

With blkdiscard, I'll need to specify -p 33550336 (65528 blocks * 512 bytes, where 65528 = 65535 (max no. of block in a single range) - 65535 % 8), assuming the starting offset is 0 (or a multiple of [8 * 512]), otherwise there will be leftover blocks that will not be wiped. This can be checked with something like hexdump after the TRIM.

 

https://superuser.com/questions/1060831/triming-as-alternative-to-securely-erasing-a-ssd