I know how mdadm RAID works, and I know why you can't replace a disk with a smaller disk in that system. I'm still very new to unRAID, but I would love it if V7 would allow the following:

 

2TB disk fails

Has 900GB of data on it

Replace with 1.5TB

Good-to-go.

 

I haven't tested this functionality on unRAID yet, but the Wiki makes it sound pretty clear that replacing data drives with smaller disks is not allowed.

 

Can anyone explain why this is so, and whether it would be possible to add in V7? (BTRFS, for example, does permit down-sizing of disks at the filesystem level and it may be more stable by the time URv7 is released).

What you ask is IMPOSSIBLE.

 

unRAID does not work at the filesystem level. It works at the device level. Therefore it is impossible for unraid to know where the data is placed on those disks. It only knows you had 2TB drive and somewhere in that entire range of 2TBs you had some data. Therefore it is impossible to restore a range of 2TB onto any smaller of a range.

Btrfs is a file system, so could manage to manipulate the files into less space.  It sounds risky and slow to me though.  It would have to perform a free space defrag to move all file data to the bottom of drive, then shrink the file system, then shrink the partition, then clone the partition to the replacement drive, and expand the partition and file system to fit.  That's doable, but would be faster and safer to prepare the second drive and copy all files to it (which will automatically defrag them).  Then you only have to remove the original drive, which is still a little tricky but doable.

Well, it used to be impossible. Rockstor, for instance, uses BTRFS as its sole filesystem and it allows downgrading disks while the array is live. Since unRAID now supports BTRFS, this could theoretically be possible here, too. BTRFS is a filesystem and so it does know where data resides on each individual disk, thus allowing all kinds of crazy-cool new things to be done.

 

Heck, you can even change between RAID levels while the array is live using BTRFS. I've tested Rockstor pretty seriously lately and switching between RAID levels was so easy ... it blew my mind. That said, they don't have the kind of support and polish that unRAID has. So, maybe in the future they'll pickup some steam, or unRAID will head that direction.

 

If this kind of functionality remains impossible with unRAID, so be it. But there are others afoot who are bringing this bag of tricks with them and I, for one, am excited to see what the future holds for the NAS market  I'm a huge fan of both unRAID and this community since discovering them, and I would like to stick with LT in the future. That's the only reason I started this thread.

Once again, you need to understand the difference between raid operating at a device level and raid operating at a filesystem level.

Btrfs is a file system, so could manage to manipulate the files into less space.  It sounds risky and slow to me though.  It would have to perform a free space defrag to move all file data to the bottom of drive, then shrink the file system, then shrink the partition, then clone the partition to the replacement drive, and expand the partition and file system to fit.  That's doable, but would be faster and safer to prepare the second drive and copy all files to it (which will automatically defrag them).  Then you only have to remove the original drive, which is still a little tricky but doable.

 

Ha! It seems that you knew where I was going with this.

 

I don't have the tech know-how to understand what this would mean for LT and unRAID, I just wanted to put it up for consideration. Whatever implementation these guys choose is a-okay with me. I'll defer to the experts on the best way to accomplish this goal, I would just like to see it as an option. I know on more than one occasion I've over-planned for data accumulation and it's just nice to have a bit of flexibility in the future.

 

Another scenario:

2TB disk fails

Your array has 5TB free space on it.

Rather than replace the 2TB disk, why not just fail it and downsize the array by one disk? Brilliant!

 

Does unRAID support this? BTRFS does.

Once again, you need to understand the difference between raid operating at a device level and raid operating at a filesystem level.

 

I do. Thanks for your helpful response.

 

I think you're missing the point of my post a little bit. If unRAID works at the device level and that prevents all of these features from being available, then unRAID v7 probably shouldn't work at the device level. That's kind-of where I'm going here. Are these things possible now? If not, we should try to make them possible in the future. Whatever needs to be done to make that happen is just part of the ride.

 

So, maybe you meant "it's impossible" as in "this is not possible in unRAID today." That may be a valid response to my feature request. However, that doesn't mean it is impossible, objectively, and unRAID should at least CONSIDER moving in that direction in the next two to three years.

 

Just my two cents. I think this is a pretty darn good feature request.

Automated downgrading or reactions on array failure can be bad. I think somewhere burried in the long history of posts here, it was mentioned that most typical red-ball array failures are temporary and due to bad data cables or power cables followed by bad disk controllers. Any automated system response may actually make matters worse.

 

I woukd like to see a feature where upon any disc failure the entire array of all drives goes into read only mode and requires an informed decision by an intelligent human before the array is able to be modified. I want it to go into a full lockdown to protect what data is on the array at that moment in time.

 

Automated downgrading or reactions on array failure can be bad. I think somewhere burried in the long history of posts here, it was mentioned that most typical red-ball array failures are temporary and due to bad data cables or power cables followed by bad disk controllers. Any automated system response may actually make matters worse.

 

I woukd like to see a feature where upon any disc failure the entire array of all drives goes into read only mode and requires an informed decision by an intelligent human before the array is able to be modified. I want it to go into a full lockdown to protect what data is on the array at that moment in time.

 

I completely agree. The downgrade process that I've seen elsewhere isn't automated. You have to click a button to make it happen in the GUI if that's your decision, but it doesn't require any CLI work whatsoever. Quick and painless, it is.

 

Frankly, automated anything gives me pause. I'm waiting for the robot overlords to come make me breakfast. What do you think, ten years?

...

I'm waiting for the robot overlords to come make me breakfast. What do you think, ten years?

 

I think you have the overlord role confused... who is making who breakfast lol.

 

I, for one, welcome our new robot overlords

 

 

If unRAID works at the device level and that prevents all of these features from being available, then unRAID v7 probably shouldn't work at the device level.

 

Why would you want to take a step back and introduce more risk for data loss?

 

Also there is a way to shrink the array currently. You just can't rebuild a disk into a smaller disk.

I'm waiting for the robot overlords to come make me breakfast. What do you think, ten years?

 

The one thing I always think about is when the robot overlords come why would they need us...

Power.  Didn't you see the matrix???

I personally think what Noob is requesting is kind of a cool idea. When I 4Wheel in my Jeep in a group and one of us break something its a Mad Scramble for all of us to figure out how to get each other home. Sure its not normally the best fix, but it is at that time.

 

It was kind of the same issue I had when I was cloning discs for my HTPC. I tried using Clonezilla and Acronis and I found out the Hard way that Clonezilla wouldn't allow me to clone a larger disc and install that image on to a smaller disc. Keep in mind the image I was using was only 4GB on a 120GB drive. I wanted to install that image on to a 30GB SSD. Took me some time to figure out I was using the wrong Imaging app.

 

Keep in mind I'm not saying unRAID is the wrong tool for the Job, but for what he is asking it "Currently" doesn't support it and honestly I'm not sure, but I can't imagine LimeTech would invest much time into it since everybody has been asking for larger drives not smaller drives. lol

There is a way to shrink the array currently. You just can't rebuild a disk into a smaller disk.

 

O_o

 

Well, that is a longer, round-about way of doing the same thing, but I'll take that. What is this procedure?

 

Also, if the drive I want to shrink/remove from the array has some data on it, how can I tell unRAID to migrate that data to other disks in preparation for the array to be shrunk?

Keep in mind I'm not saying unRAID is the wrong tool for the Job, but for what he is asking it "Currently" doesn't support it and honestly I'm not sure, but I can't imagine LimeTech would invest much time into it since everybody has been asking for larger drives not smaller drives. lol

 

That's a valid point. This only comes in to play under pretty narrow circumstances. When you're juggling disks among multiple servers to triage a bad situation, when you go to make a replacement purchase and the smaller drive is buy one, get one (or some similarly impossible to pass on deal), or when you are phasing out a server/migrating data and want to start pulling drives offline.

 

That said, even though this comes up once-in-a-blue-moon, it would be a very useful feature for when those times do pop up. You would have never needed to leave unRAID to move that little bit of data if it were aware of what you were trying to do. It just isn't, yet. I think it's something I would be thinking about as a dev team because the whole set of features available on some of the latest gen storage projects kick-ass. unRAID just has them beat by a mile in features, support, and reputation. Those are great advantages, match the other guys in features and Lime-Tech will continue to grow into the future.

 

Again, just my two-cents, but I think they're worth taking to the bank.

There is a way to shrink the array currently. You just can't rebuild a disk into a smaller disk.

 

O_o

 

Well, that is a longer, round-about way of doing the same thing, but I'll take that. What is this procedure?

 

Also, if the drive I want to shrink/remove from the array has some data on it, how can I tell unRAID to migrate that data to other disks in preparation for the array to be shrunk?

unRAID never moves any data for you except for the cached user share data. You will have to move it yourself. The way you remove a disk from the array is new config without it and rebuild parity. Take a look at the unbalance plugin and see if it has anything for you, I have never used it.

Freaking awesome.

 

Unbalance is exactly the tool to migrate data from one disk to another. I don't know if it was designed with shrinking an array in mind, but it will do just fine for that purpose.

 

Consider this question/thread resolved!

One flaw in the "New Config without it" process ==>  the premise of this question was that you could replace a failed drive that has less data than the size of the drive with a smaller drive that could hold all the data.

 

If you do a New Config without the failed drive, you've just lost the ability to rebuild that drive or to access the data on it via emulation from the other disks in the array.    The FIRST thing you should do if you want to replace the drive with a smaller drive is copy ALL of the data from the drive to another location [or, if you have good backups, be sure you have a current directory of the failed drive so you know which files you need to copy back to the array].    THEN you can do a New Config without the failed drive; and then simply copy all of the data back to the array.

 

If you do a New Config without the failed drive, you've just lost the ability to rebuild that drive

 

Right, that makes sense. What I thought I would do is use the unbalance plugin to unload, for example, sdc. Then I would pull sdc from the array and replace it with a smaller disk. I would then assign the new disk to the array as sdd, and fail/remove sdc entirely. As new data is written to the array, unRAID will use the new space on sdd for the writes and eventually the drives will return to equilibrium.

 

Will this work, or no?

So, are you're saying is unRAID should become a front end for a BTRFS array/pool?

 

The current unRAID array of stand-alone disks protected by a parity drive does not lend itself to any kind of easy downgrading or disk removal while maintaining the parity protection during the whole process.

Well, based on what Gary said, it seems like it might be possible to drop a disk entirely and replace it with a smaller one (which is fine, that's the same function even if it uses different means to obtain the same end). If I understood Gary and trurl correctly, I think unRAID already has the ability to do what I want with just a simple plugin.

 

I'm presently awaiting clarification on that front.

If a drive has failed, you should be able to use unbalance or some other method to copy or move the data from the emulated disk to other disks in the array. Then when you no longer need the data on the emulated disk, you can new config without the failed disk.

 

However, note that running with one disk failed/emulated means the array effectively no longer has parity protection. Usually the fastest way to get back to a parity-protected state is to rebuild the failed disk to another disk the same size or larger.

Well, based on what Gary said, it seems like it might be possible to drop a disk entirely and replace it with a smaller one (which is fine, that's the same function even if it uses different means to obtain the same end). If I understood Gary and trurl correctly, I think unRAID already has the ability to do what I want with just a simple plugin.

 

I'm presently awaiting clarification on that front.

 

You could do it, but it's not like you can do it with a simple command. Also, your array would be unprotected until the parity build operation was completed on the replacement downsized drive or the array with a drive removed. I'm not positive, but I would expect a BTRFS array would maintain protection throughout the process.

 

Remember the very first line in the assumptions in your original post:

 

2TB disk fails

 

So the most important thing you want to do at that point SHOULD BE to get the system back to a protected state.  Once a disk fails, you're running "at risk" ... i.e. any other failure will potentially result in loss of all the data on all failed disks.

 

The quickest way to achieve that is to replace the failed disk and let UnRAID rebuild the disk.  But this does indeed require a disk at least as large as the disk that failed (and no larger than the parity disk).

 

Your premise then went on to assume there was only 900GB of data on the disk, and you wanted to know if you could use a 1.5TB disk for the rebuild.  As we've all noted, the answer to that is no.

 

You COULD, as I noted above, copy all of the data from the failed (but now emulated) disk to another location [another UnRAID array; an external drive with enough space; another PC with enough space; etc.] and THEN do a New Config, replacing the failed disk with a smaller disk if you wanted to (1.5TB, 1TB, etc.); and then when you started the array it would do a new parity sync.    Note, however, that in this scenario you're running "at risk" for the entire time it takes to copy the data, and then to do the new parity sync -- which is longer than a simple disk rebuild would take.

 

I would NOT recommend trying to move the data "within" the array by using the UnBalance plugin ... as this increases the risk of data loss.  If anything was to go wrong during this move (i.e. another drive failure), the emulated disk would have already been modified as a result of the moves, and it would no longer be possible to do a successful rebuild of the failed disk.

 

Bottom line:  By FAR the best approach is to use a replacement drive that's at least as large as the failed drive [Personally I never replace a failed drive with anything smaller than the current parity drive].    If you don't have one handy, shut down the server for a couple days until you get one.  Especially if you don't have a complete set of backups.