I've always pondered that since unRAID can rebuild the data from a removed/missing disk, why only to a replacement disk?  Why not, assuming there is enough free space, rebuild to an existing disk?  For example, when I built my first server, it was all IDE drives.  Eventually I replaced all but one.  Now, I have one 500GB IDE drive left and I recently purchased a 2TB drive and added it to my server because I was getting low on space.  I would like to remove the IDE drive to improve parity check/sync performance and the new drive has plenty of space to accommodate that data.  I think it would be cool if I could remove that drive from the array and unRAID would ask me where to rebuild that data.  I know I can manually copy the data from the one drive to the other, but then you still have to remove the old drive and rebuild parity.  It would be far more convenient if it would allow you to rebuild the data to an existing drive with enough free space.  Not only would this save time, but it would feel safer to me (maybe falsely) since you would be rebuilding from known good parity data rather than rebuilding parity from freshly copied/moved data that may now be incorrect/corrupt.

 

Are there technical reasons why this isn't possible or would that not be a good way to do it?

Not sure if this is impossible, but certainly not easy. Parity allows a complete disk to be rebuilt. It does this by rebuilding each sector with absolutely no knowledge of the files or even filesystem. Doing what you are suggesting would require a completely different type of redundancy model than unRaid's parity model to implement effectively.

Not sure if this is impossible, but certainly not easy. Parity allows a complete disk to be rebuilt. It does this by rebuilding each sector with absolutely no knowledge of the files or even filesystem. Doing what you are suggesting would require a completely different type of redundancy model than unRaid's parity model to implement effectively.

I will echo what bjp999 said.  And also add that this would have to be an opt in thing.  I personally would NOT want my server rebuilding data "by itself" onto another drive or other drives in the array.

And also add that this would have to be an opt in thing.  I personally would NOT want my server rebuilding data "by itself" onto another drive or other drives in the array.

 

I never said for unRAID to do that on it's own.  I said it would ask where to rebuild the data.

Not sure if this is impossible, but certainly not easy. Parity allows a complete disk to be rebuilt. It does this by rebuilding each sector with absolutely no knowledge of the files or even filesystem. Doing what you are suggesting would require a completely different type of redundancy model than unRaid's parity model to implement effectively.

 

I figure it was probably not feasible.  Thanks for the info.

And also add that this would have to be an opt in thing.  I personally would NOT want my server rebuilding data "by itself" onto another drive or other drives in the array.

 

I never said for unRAID to do that on it's own.  I said it would ask where to rebuild the data.

You can do that yourself...  If, for example disk5 were to fail, you can copy all the files on it to free space elsewhere (on a specific physical disk with space) by typing

cp -r /mnt/disk5 /mnt/disk6

You don't really want to rebuild data, rather you would need to move the data to other disks and then zero the disk so it could be removed. Even if the drive failed you would not do a rebuild but rather you would still move the simulated data. You actually can already do this, it's just not an automated task. So, this would actually be a new function, no an option when rebuilding a disk.

 

The basic steps would be something along these lines;

1. Rename each share directory to some temporary name, for example "Movies-temp" and "Music-temp".

2. Exclude the disk from the shares so it's not used again.

3. Copy the data to the proper share, for example a command like cp -r /mnt/user/Movies-temp /mnt/user/Movies would be used. Actually, using rsync would likely be better I suppose. Note that this puts the data back into the share which is better than disk to disk copying.

4. Zero the partition on the disk to be removed, either actually zero it if it's working or virtually zero it if it's not working.

5. Remove the disk while keeping the parity valid.

 

This could be automated if there were the required hooks in the unRAID engine to do a few of the required tasks. But at this moment I don't believe the hooks are all there. For example, to remove the zero'd disk you have to stop the array, unassign the disk, and then re-validate parity (mdcmd set invalidslot 99 in the older unRAID versions) before starting again. I don't believe you could do this via a script. If these steps could be peformed in a script then anyone with the capabilities and motive could do this.

 

In the future with the 5 series unRAID, I could see this being possible via a plug-in which would even appear on the web interface.

 

Peter

 

But you're still left with a disk being simulated which requires a parity rebuild to remove.

And also add that this would have to be an opt in thing.  I personally would NOT want my server rebuilding data "by itself" onto another drive or other drives in the array.

 

I never said for unRAID to do that on it's own.  I said it would ask where to rebuild the data.

You can do that yourself...  If, for example disk5 were to fail, you can copy all the files on it to free space elsewhere (on a specific physical disk with space) by typing

cp -r /mnt/disk5 /mnt/disk6

And after you are sure you have the data backed up, follow this http://lime-technology.com/forum/index.php?topic=2591.msg20919#msg20919 post to remove the old disk.

 

The whole procedure is time consuming but quite doable already without adding a feature to automate it.

You don't really want to rebuild data, rather you would need to move the data to other disks and then zero the disk so it could be removed. Even if the drive failed you would not do a rebuild but rather you would still move the simulated data. You actually can already do this, it's just not an automated task. So, this would actually be a new function, no an option when rebuilding a disk.

 

The basic steps would be something along these lines;

1. Rename each share directory to some temporary name, for example "Movies-temp" and "Music-temp".

2. Exclude the disk from the shares so it's not used again.

3. Copy the data to the proper share, for example a command like cp -r /mnt/user/Movies-temp /mnt/user/Movies would be used. Actually, using rsync would likely be better I suppose. Note that this puts the data back into the share which is better than disk to disk copying.

4. Zero the partition on the disk to be removed, either actually zero it if it's working or virtually zero it if it's not working.

5. Remove the disk while keeping the parity valid.

 

This could be automated if there were the required hooks in the unRAID engine to do a few of the required tasks. But at this moment I don't believe the hooks are all there. For example, to remove the zero'd disk you have to stop the array, unassign the disk, and then re-validate parity (mdcmd set invalidslot 99 in the older unRAID versions) before starting again. I don't believe you could do this via a script. If these steps could be peformed in a script then anyone with the capabilities and motive could do this.

 

In the future with the 5 series unRAID, I could see this being possible via a plug-in which would even appear on the web interface.

 

Peter

 

 

I figured this thread would die after my last post and I hadn't checked back until now.  I'm surprised to see further discussion on this. 

 

Anyway, I am a little confused by steps 1-3.  You want me to rename my shares and then recreate the old share directories (empty) from scratch excluding the disk I wish to remove?  Then copy the data from the temp shares to the new shares?  Won't this duplicate my data?

Anyway, I am a little confused by steps 1-3.  You want me to rename my shares and then recreate the old share directories (empty) from scratch excluding the disk I wish to remove?

The directories that contain shares or portions of shares are renamed to facilitate moving the data to its new location. The shares are not recreated. The disk that is to be removed is added to the Excluded disk(s) field of the existing shares. The remaining disk or disks that are to contain the data must be included either explicitly or by default.

Then copy the data from the temp shares to the new shares?  Won't this duplicate my data?

Only the shares or portions thereof that lie on the disk to be removed is temporarily duplicated. The real or simulated disk that is to be removed is then zeroed for removal and only a single copy remains.

 

Anyway, I am a little confused by steps 1-3.  You want me to rename my shares and then recreate the old share directories (empty) from scratch excluding the disk I wish to remove?

The directories that contain shares or portions of shares are renamed to facilitate moving the data to its new location. The shares are not recreated. The disk that is to be removed is added to the Excluded disk(s) field of the existing shares. The remaining disk or disks that are to contain the data must be included either explicitly or by default.

Then copy the data from the temp shares to the new shares?  Won't this duplicate my data?

Only the shares or portions thereof that lie on the disk to be removed is temporarily duplicated. The real or simulated disk that is to be removed is then zeroed for removal and only a single copy remains.

 

 

I think I understand all that, but I have one last question.  You say rename the directories.  That means to rename the shares within the unraid GUI, right?  Because if I remember correctly (I'm at work so I can't check it), unraid won't allow you to just rename the share directory on the disk.  If that's the case, then when I run the cp command wouldn't it fail because /mnt/user/Movies doesn't exist since I renamed it to /mnt/user/Movies-temp?  That's what's confusing me.

Anyway, I am a little confused by steps 1-3.  You want me to rename my shares and then recreate the old share directories (empty) from scratch excluding the disk I wish to remove?

The directories that contain shares or portions of shares are renamed to facilitate moving the data to its new location. The shares are not recreated. The disk that is to be removed is added to the Excluded disk(s) field of the existing shares. The remaining disk or disks that are to contain the data must be included either explicitly or by default.

Then copy the data from the temp shares to the new shares?  Won't this duplicate my data?

Only the shares or portions thereof that lie on the disk to be removed is temporarily duplicated. The real or simulated disk that is to be removed is then zeroed for removal and only a single copy remains.

 

 

I think I understand all that, but I have one last question.  You say rename the directories.  That means to rename the shares within the unraid GUI, right?  Because if I remember correctly (I'm at work so I can't check it), unraid won't allow you to just rename the share directory on the disk.  If that's the case, then when I run the cp command wouldn't it fail because /mnt/user/Movies doesn't exist since I renamed it to /mnt/user/Movies-temp?  That's what's confusing me.

 

Your right. I was thinking of a slightly different procedure. This one should copy only what is required.

 

The renaming must be done on the command line, e.g., "mv /mnt/disk1/Movies /mnt/disk1/Movies-temp". Then copy to user share "cp -r /mnt/disk1/Movies-temp/* /mnt/user/Movies"

 

Following the original procedure does require renaming the existing share and then recreating that share exactly the same except for excluding the source (to be removed) disk.

In case it's still not clear, I meant to rename the user share directories only on the disk to be removed and to exclude that disk from the share. Then ,when you move or copy the files back to the user share they wil be moved to whichever disks have the appropriate room for the files according to the user share settings.

 

About the only steps that can be done using the unRAID interface is the excluding of the disk and the unassigning of the disk when it's ready to be removed. Everything else basically has to be done via the command line.

 

Peter