Ability to remove drive from array without having to destroy parity

[jonp]

(Implies a wipe feature as well)

[Isolgrac]

Would this also include the ability to replace a drive without having to rebuild parity?

 

For example, if I want to replace an existing drive with a bigger drive (including the parity drive) it would be great if we could just add the new drive and tell unRaid that it was to replace an existing drive. UnRaid could then clone the existing drive over to the new drive (whilst not destroying parity) and once complete, automatically remove the old drive from the array. The key thing here is that while the clone operation takes place, any parity calculations include the clone pair as a single drive rather than as two separate drives.

 

Hopefully the above makes sense. If this isn't included in this feature, please can you move this post to a new feature request?

[itimpi]

Would this also include the ability to replace a drive without having to rebuild parity?

 

For example, if I want to replace an existing drive with a bigger drive (including the parity drive) it would be great if we could just add the new drive and tell unRaid that it was to replace an existing drive. UnRaid could then clone the existing drive over to the new drive (whilst not destroying parity) and once complete, automatically remove the old drive from the array. The key thing here is that while the clone operation takes place, any parity calculations include the clone pair as a single drive rather than as two separate drives.

 

Hopefully the above makes sense. If this isn't included in this feature, please can you move this post to a new feature request?

Replacing a drive with a bigger drive without breaking parity is already available and has been for a very long time.  During this process the array is actually unprotected, but as long as you still have the old drive you can still revert to the protected state if anything goes wrong.  I am not sure that your idea of having the old drive and the new one both plugged in and cloning actually adds any significant value, and it seems it could add a lot of unnecessary complexity.

[SSD]

Would this also include the ability to replace a drive without having to rebuild parity?

 

For example, if I want to replace an existing drive with a bigger drive (including the parity drive) it would be great if we could just add the new drive and tell unRaid that it was to replace an existing drive. UnRaid could then clone the existing drive over to the new drive (whilst not destroying parity) and once complete, automatically remove the old drive from the array. The key thing here is that while the clone operation takes place, any parity calculations include the clone pair as a single drive rather than as two separate drives.

 

Hopefully the above makes sense. If this isn't included in this feature, please can you move this post to a new feature request?

Replacing a drive with a bigger drive without breaking parity is already available and has been for a very long time.  During this process the array is actually unprotected, but as long as you still have the old drive you can still revert to the protected state if anything goes wrong.  I am not sure that your idea of having the old drive and the new one both plugged in and cloning actually adds any significant value, and it seems it could add a lot of unnecessary complexity.

(Commenting on bolded section] So long as you refrain from writing to the disk being rebuilt during its reconstruction process.

[Isolgrac]

My understanding is that to replace a drive you remove the old one and insert the new one and allow unraid to rebuild it from the parity calculations. Please let me know if this isn't the correct way of doing it currently because I'm about to do it and I don't want to make mistakes! :-)

 

If my understanding above is correct (and from your comment 'the array is actually unprotected' I assume it is) then I think what I'm looking for is a way to do it without leaving the array unprotected during the process. Not being familiar with the internals on unRaid I have no idea if this would add a significant amount of complexity over and above what is already planned for this feature (removing a drive) and so I asked if it would be possible to include it. I know that it is not an essential feature (or else it would be in already), but being able to upgrade drives without leaving the array unprotected for a period of time would be a big plus for users such as myself who are less skilled/confident in such matters.

[NAS]

You are talking about replacing a drive. We are talking about removing a drive (i.e. reducing the drive count in the server by one).

 

You have a right to be confused though as we touched on a lot of things namely my opinion that to upgrade  a drive you should not just pull one and have it rebuilt. For me this is a bad thing as it puts the system into a failed state for potentially a whole day+ in which time any real drive failure will result in data loss.

[Isolgrac]

You are talking about replacing a drive. We are talking about removing a drive (i.e. reducing the drive count in the server by one).

 

Ok, I'll raise this as a separate feature request then. Thanks for the clarification :-)

[NAS]

Just curious if this is being worked on or at least actively considered.

 

I raise it again as I am currently embarking on a new disk refresh and upgrade (3rd since we started discussing this) and it still maintain that this would both be a genuinely useful / time saving feature but also a Unique Selling Point that your competitors simply cant offer (due to the nature of the RAID they use) should be grabbed with both hands IMHO

[chickensoup]

Bumping this thread again to see if there are any plans to implement in the near future. 

 

I've had my server running extremely well for around a decade now (pre-4.7) and as a result, have some smaller disks I would prefer to phase out completely, reducing my disk count, rather than replace. I'm aware there are ways to do this but I'd currently require some assistance from the forums to make sure I don't break anything (i.e. rsync/move data on to alternate disk, remove, rebuild parity). 

 

The idea of being able to 'decomission' a disk seems like a great idea. 

[mgranger]

On 5/21/2019 at 2:00 AM, chickensoup said:

Bumping this thread again to see if there are any plans to implement in the near future. 

 

I've had my server running extremely well for around a decade now (pre-4.7) and as a result, have some smaller disks I would prefer to phase out completely, reducing my disk count, rather than replace. I'm aware there are ways to do this but I'd currently require some assistance from the forums to make sure I don't break anything (i.e. rsync/move data on to alternate disk, remove, rebuild parity). 

 

The idea of being able to 'decomission' a disk seems like a great idea. 

I would also like this.  I am fearful to add more disks to my array because they will be in my array forever (unless i break my parity to remove them)  Then I will have that many more disks in my array that could fail at some point.  Nice feature to have.

[andyd]

+1

 

But in general I think removing a drive should not require a wiki that has multi steps that can lead to loss of data if mishandled as in...

 

https://wiki.unraid.net/Shrink_array

 

I would be ok with rebuilding the array for now if there was a way to abstract the guide to a button in the OS itself that safeguards against any potential issues. Otherwise, it's seems odd that wanting to remove a drive that I have already moved data off of can still lead to a significant amount of data loss

Edited March 22, 2020 by andyd

[Squid]

51 minutes ago, andyd said:

Otherwise, it's seems odd that wanting to remove a drive that I have already moved data off of can still lead to a significant amount of data loss

Think of it this way.  No OS in existence (today or the past) ever actually "deletes" anything off of a drive when you move / delete a file from it.  Rather it simply clears out the directory entry.  (This is why when you delete a file (or in Windows perform a quick format) you can quite easily recover the files that were contained on it with zero data loss)

 

Because nothing is ever actually deleted, the contents of those files (or more precisely the sectors on the hard drive) are still contributing to the parity system.  Only if you actually zero out the drive is it safe to remove without affecting a parity operation (Parity 1 at least)

 

Although I've never had a need to actually remove a drive (and don't foresee ever doing that in the future), if I was going to actually do it (knowing that my array is 100% healthy with no problems at all), I would simply issue a New Config and let the parity drives rebuild their contents accordingly.

 

All in all though, either way its far easier to remove a drive from unRaid than say something that runs RAID-5 where the process would basically be to copy the entire contents of the entire array somewhere else and then recreate the array and copy everything back.

[andyd]

That makes sense. Thanks for explaining! 

 

But I think there are legitimate reasons that people will want to swap drives out eventually - no sense in keeping older smaller drives when you start to run out of space and don't have enough ports. That is the issue with my current array.

 

I was going to post it as a separate feature request but it seemed like it belonged here. In general, I don't really care if the parity needs to be rebuilt etc. My main concern is that removing a drive requires several steps where it seems like a button that says "Remove Drive" should abstract those steps away

 

Edited March 22, 2020 by andyd

[Squid]

1 minute ago, andyd said:

But I think there are legitimate reasons that people will want to swap drives out eventually - no sense in keeping older smaller drives when you start to run out of space and don't have enough ports. That is the issue with my current array.

 

Completely agree with that.  I just swap out the smaller drives for larger ones and let the system rebuild the drive and then expand it to the size of the replacement.   A feature to handle this all automatically would be nice for the very rare times that users require it.  I'm not really sure though if the development effort is worth it.  But, if enough people ask, Limetech does listen if it fits in with their development plans.

[Opawesome]

On 3/22/2020 at 8:06 PM, Squid said:

But, if enough people ask, Limetech does listen if it fits in with their development plans.

I would definitely enjoy such a feature @limetech

[eagle470]

On 3/22/2020 at 1:46 PM, Squid said:

Think of it this way.  No OS in existence (today or the past) ever actually "deletes" anything off of a drive when you move / delete a file from it.  Rather it simply clears out the directory entry.  (This is why when you delete a file (or in Windows perform a quick format) you can quite easily recover the files that were contained on it with zero data loss)

 

Because nothing is ever actually deleted, the contents of those files (or more precisely the sectors on the hard drive) are still contributing to the parity system.  Only if you actually zero out the drive is it safe to remove without affecting a parity operation (Parity 1 at least)

 

Although I've never had a need to actually remove a drive (and don't foresee ever doing that in the future), if I was going to actually do it (knowing that my array is 100% healthy with no problems at all), I would simply issue a New Config and let the parity drives rebuild their contents accordingly.

 

All in all though, either way its far easier to remove a drive from unRaid than say something that runs RAID-5 where the process would basically be to copy the entire contents of the entire array somewhere else and then recreate the array and copy everything back.

I'm about to yank a drive out of my array tonight myself do to poor performance and the magic of tdarr/unmanic. 

I don't think you need to NOT have to rebuild parity, but I should be able to 1) stop array (2) unassign disk (3) start array. When you start the array it would be like coming back from an unclean shut down, parity would need to be rebuilt. 

 

This would allow me minimum reconfiguration time when I yank the drive. 

[Squid]

If you simply unassign the disk, then the system still carries on it's merry way emulating the removed drive, and you wind up running the array in a degraded state.  If you're removing and not replacing it with another, then you need to issue a Tools - New Config and have it rebuild the parity data

[eagle470]

1 minute ago, Squid said:

If you simply unassign the disk, then the system still carries on it's merry way emulating the removed drive, and you wind up running the array in a degraded state.  If you're removing and not replacing it with another, then you need to issue a Tools - New Config and have it rebuild the parity data

I know, I've done it multiple times, but maybe it would be simpler to have the UI make the disk slots removable. You add the number you need, then if you fail out a drive, you add the new one in the same way.

 

If you want to remove a drive from the array, you remove the slot, a prompt comes up, you type in "Yes, I know what I am doing" and then your root password (disable auto fill for this.)

 

This is just one idea on how to do it, another would be to have a different section in the array where you have a 'Disk Remove' section under 'Tools', go in there, select the disk to remove from the array, fill out the same acknowledgements and move forward. 

[ChatNoir]

8 hours ago, eagle470 said:

I know, I've done it multiple times, but maybe it would be simpler to have the UI make the disk slots removable. You add the number you need, then if you fail out a drive, you add the new one in the same way.

 

If you want to remove a drive from the array, you remove the slot, a prompt comes up, you type in "Yes, I know what I am doing" and then your root password (disable auto fill for this.)

 

This is just one idea on how to do it, another would be to have a different section in the array where you have a 'Disk Remove' section under 'Tools', go in there, select the disk to remove from the array, fill out the same acknowledgements and move forward. 

I see so many ways this might be used improperly and people loosing their Parity.

 

In general, I am in favor of simplicity, but in that case, I think that having a separate step to go through is a good way to reduce user error.

[eagle470]

On 4/13/2021 at 12:57 AM, ChatNoir said:

I see so many ways this might be used improperly and people loosing their Parity.

 

In general, I am in favor of simplicity, but in that case, I think that having a separate step to go through is a good way to reduce user error.

People lose data all the time with the current method. I've seen it in the month I've been in the Unraid user group on facebook. 

The method I'm proposing at least removes the possibility of mistaking the cache drive for a data drive or vice versa.

The second method I mention seems like the best of the two options, though more limiting. 

[daniel.boone]

I equate this as a "fast" replace option and fast is actually the wrong word here. Anyone with a large array or large disks knows replacing drives takes considerable time. Nowadays that's probably most of us.

 

I understand this new function like this. It would allow one to copy an existing drive with the sole purpose of replacing the disk with the same or larger drive. This to be done without performing a read of every sector on every disk in the array. 

 

Right now I'm planning to replace my 2 parity drives. To do so the simplest method is a new configuration, replace the drives and recreate the parity. The other option is me removing a drive at a time and running multiple rebuilds. This is before I even introduce the old drives to participate in the array. Both options presents risks and additional wear on all the entire array. An improved option would be copy disk and replace option. Same should be possible with any drive participating in unRAID. Say if I wanted to upgrade the old 1TB  disk in the array or even a well used cache drive. I should be able to clear a replacement disk, copy, swap the disk while only reading the disk being replaced. This takes the concept of preclearing a drive to the next level. 

 

The process for rebuilding a failed drive works great. It has saved my data on more than one occasion. I feel no change is needed there. The process for good disk replacement could benefit from a more streamlined approach.