I've been thinking for a long time about this, kinda slow about presenting it.  These ideas are things Tom has probably been thinking about too, and may already have a road map in mind, and nothing I say is to be construed in any way as trying to push him in any particular direction, just provide some ideas, and foster the discussion and feedback that may help him to determine future directions and features of unRAID.  Many of these ideas are not new, have been discussed at some length in earlier threads.

 

There's a lot of competition for unRAID out there, and in some markets Windows Home Server is the leader.  I've been impressed with some of its capabilities, but have thought that unRAID with some improvements could compete very well.  I have not had access to it, so won't discuss it directly.

 

The unRAID system is currently a single array, with an optional parity drive.  It is currently set to allow up to 16 drives, although the super.dat currently has room for up to 31 drives.  I am sure that Tom has already, or is now working on, or is planning to, retool the unRAID core driver, to change from single array management to multi-array management.  It has been discussed before, and he at that time had ideas as to how he wanted to do it.  Once the ability to have multiple arrays/pools/zones/arenas is available, then all kinds of features become possible.  Here are some ideas of mine, for array types:

 

1. Classic unRAID array

    the current unRAID array becomes Array 1

    optional parity drive                                                      (as in current release)

    each array may have 1 to 15 data drives                          (as in current release)

    optional, configurable cache space on a cache drive          (as in current release)

    optional use of warm spare

    automatically mounted, configurable export and sharing    (as in current release)

    can be multiple (initially 1 - N classic arrays, future 0 - N)

 

2. RAID 1 array (mirroring)

    array of 2 or more drives [of equal size?]

    drives may be removable (eSATA, USB, Firewire, etc)

    writes are simultaneous, to each drive

    no parity drive (may initially be internally a 2 drive classic array, 1 data & 1 parity)

    automatically mounted, configurable export and sharing    (as in current release)

    can be multiple (0 - N RAID 1 arrays)

    [future:  optimize reads]

 

3. RAID 0 array [future]

    drive(s) assigned here are all part of one volume

    no parity protection

    automatically mounted, configurable export and sharing    (as in current release)

    can be multiple (0 - N RAID 0 arrays)

    [future:  optimize reads and writes]

 

4. Transfer zone

    Cache drive(s) are here, not removable

    non-cache drives may be removable (eSATA, USB, Firewire, etc)

    non-cache drives may be any file system, FAT, FAT32, NTFS, ReiserFS, ext2, ext3, HFS, etc

    could include hardware RAID arrays, if driver support available

    optional mounting, optional export and sharing, all configurable per drive

    not multiple (0 or 1 Transfer zone)

 

5. No spin zone

    drive(s) assigned here must not be accessed at all

    for protecting drives used in a dual boot system

    for installed but unused drives, spares, etc

    automatic spindown at boot

    no mounting, no export or sharing

    not multiple (0 or 1 No spin zone)

 

6. Warm spare zone

    drive(s) assigned here are initially tested, cleared, and formatted

    on drive failure in any classic array, may be configured for automatic drive replacement

    automatic spindown at boot

    no mounting, no export or sharing

    not multiple (0 or 1 Warm spare zone)

 

7. Future types of arrays

    to keep up with, and stay ahead of competition

    to keep up with unRAID user's ideas

    perhaps add back the standard md driver, with all of the other RAID types, 0, 1, 5, 6, 10, etc

 

8. Un-assigned disks

    not really an array or zone, just disks that have not been assigned anywhere

 

 

Development work:  Most of this is straightforward, and easily broken into small tasks, that can be phased in, in different releases.  But several are major, and the biggest is the first, the conversion of unRAID from the current single array to a multi-array program.  It's not hard, but usually means walking through a lot of code, looking for variables that need to be modified for multi-array support.  It will require extensive testing, because a lot of i's have to be exactly dotted, and t's crossed.  Things like parity.currentBlock may change to array[n].parity.currentBlock, and disk[n].size might become array[a].disk[n].size.  Corresponding changes have to be made to the Web Management interface.  I envisage an Arrays tab being added, with initially only one default array existing, a classic array corresponding to the current unRAID array, call it Array 1.  There will also be an Add Array button, that will bring up a dialog requesting Array Type, and then the appropriate configuration questions for that type of array, such as an optional array name, cache path, Export (No, Hidden, Read Only, Read/Write), etc.  The arrays will be displayed at the top, like the disks are now, with some configuration and status info.  The Devices tab will change a little, with separations and headings to show each configured array, and optional disk slots to be assigned under them.  The Main status page will change similarly.  Only arrays set up on the Arrays tab will be available on the Devices tab.  To add a disk to a new array, you would first go to the Arrays tab to setup the array, then go to Devices tab to assign the drive to that array.  To move a disk from one array to another, it would have to be un-assigned first, then various rules may apply, and finally it can be assigned to another, existing array.

 

The next biggest job (I think) is expanding User Shares to work on multiple arrays, and other types of arrays, along with the associated security controls.  Sounds very tricky to me, and would also require much testing.  But it doesn't have to be done initially.  I would start with the first multi-array release only supporting User Shares on the default classic Array 1, basically carrying forward only what is already working.  The only other task that seems like a challenge is setting up the RAID 0 striping, and that can be left to the future.  Most of the rest of the tasks won't even require that much testing, as they will either work or they won't.

 

So my plan would be to release unRAID Multi, as v5.0 beta 1, with possibly only multiple classic arrays implemented.  Then in beta 2, add a simple RAID 1 (just a 2 drive classic array internally) and a simple Transfer zone with one Cache drive, and possibly a simple No spin zone (because it's easy).  Then release 5.1 with support for removable drives, with the corresponding improvements to the RAID 1 and Transfer arrays.  Then release 5.2 with expanded support for User Shares and security across multiple arrays, and other types of arrays.  Then release 5.3 with the Warm Spare support, and optimized RAID 1 arrays.  Then release, well you get the idea...

 

It's possible that Tom may want to change the product and pricing structure, perhaps introduce unRAID Multi as a new product with an upgrade fee for existing customers, or change it so that unRAID Pro becomes unRAID Multi (with up to 32 drives), and unRAID Plus becomes unRAID Classic or Single, same as the current Pro license with up to 15 drives.  I think he's worth an additional fee for this upgrade, and I want to see further development fully funded.  Others may have other thoughts.

 

Now there are various other requested features, such as NFS support, UPS support, individual drive spin down control, dedicated directory caching, etc.  But it strikes me that all of these are almost completely independent of my designs above.  That is, they could be scheduled before, during, or after all of the multi-array support.  I'm in no way trying to push one above another.  Most of us would probably like to see them sooner rather than later though.  [my apologies for length of post]

Good Post. We must have been thinking along the same lines.

 

My recent thoughts have been to add more "arrays" to the mix using "personalities" as you mention

Personalities being

"Protected JBOD"

"RAID 1"

"RAID 0"

 

The concept of zones is a good one.

 

In that concept I would like to see a "removable" / "shareable" zone.

 

These are removable drives that when plugged in are brought up into view and a share is make to export them.

My readyNAS has this which is why I have not gotten rid of it.

I plug a USB device in (card reader, camera, hard drive) and it gets mounted and loaded to the network.

On one special port it does an automatic copy to a dated directory on the raid arry.

This is really handy for my camera. Plug it in and files are copied to a dated directory.

 

 

Getting back to the personalities,

I would love to see RAID1 and RAID0 brought back into the md device driver.

Where I see it being extremely useful is in a SAFE50 or SAFE33 environment whereby a set of two drives are joined together in use.

The drives are partitioned into two partitions of specified sizes. (50% , 33%... perhaps even configurable).

One one set of partitions a RAID1 array is created, On another set of partitions a RAID0 array is created.

The Cache drive (or transfer zone) is assigned to the RAID1 array.

The Parity  drive is assigned to the RAID0 array.

I see the "SAFE" arrangement being a big plus to unRAID because most home users are looking to use spindles as effectively as possible.

If you have a parity drive and a cache drive, it makes sends to set them up in a SAFE arrangement.

You get the benefit of cache & parity without needing an extra spindle.

In addition, you get protection on the cache drive, plus speed on the parity drive.

Win Win all the way around.

 

 

As much as like the idea of multiple arrays and near infinite expandability.... I'd like to voice my opinion that there are basic features that are needed WAY before all these new multi-array classifications.

 

- UPS Support.  Yes, some of us can add our own packages, but most cannot.

- E-mail alerts when a failure occurs.  Yes, we can add our own, but many cannot.

- NFS File System support.

- Performance...  (there are too many people who are experiencing poor performance.)  From what I've seen, the "background" process that should be writing the disk blocks in memory to the physical disks is blocking all network IO while active.  This is just not how it is supposed to work.  It is supposed to happen in the background, not block all foreground network activity.  The "Cache" drive, in my opinion, is a band-aid, put over a major underlying problem of process locks and synchronization.

- Support for other file systems. NTFS, ext2, ext3, etc.

 

All of these are on Tom's laundry list, plus these 4 items:

 

- support more than 1 array (parity group) in same system

- permit some disks to be outside the array. This will let you select devices which will not be included in the parity-protected array.

- concatenated parity drives

- hot spares

 

Do these last 4 sound a lot like your ideas... They sure do to me.  In my opinion however, other items are higher in priority.

 

Joe L.

- UPS Support.  Yes, some of us can add our own packages, but most cannot.

- E-mail alerts when a failure occurs.  Yes, we can add our own, but many cannot.

 

These can be added by the community, all it requires is working together.

As far as UPS support. It's a very hard thing to do as there are so many variances.

We can easily add apcupsd but the user will still have to configure the hardware side.

We can easily add NUT support, but again, the user will still have to configure the hardware side.

 

As far as email alert support, I posted something to start the ball rolling, no one responded.

http://lime-technology.com/forum/index.php?topic=2048.0

29 views and no responses...

In that sense, it seems that maybe people are not that interested.

 

One of the biggest hindrances we have is the user interface is not programmable/hookable/pliugin-able... for lack of better words.

If the interface were a standard http server and the control program was a daemon whereby CGI or PHP communicated to it then we could be adding features much more easily.

Or at least if emhttp could serve pages we write and run CGI programs then we could add our own hooks.

 

  This is just not how it is supposed to work.  It is supposed to happen in the background, not block all foreground network activity.  The "Cache" drive, in my opinion, is a band-aid, put over a major underlying problem of process locks and synchronization.

I'm going to have to agree with you here.. Then again, the cache drive is only 1 drive spinning all day instead of two or more.

In some sense I really like the idea of it. For our backups, 30 servers can dump the backups as fast as possible on a mostly clear drive.

Then over time, the drive is mv'ed (rsync'ed) to resting spots on the array.

In my situation, the mover runs every 6 hours and moves files with rsync at a slow reasonable rate to the array.

It only moves files not modified in the last 6 hours.

 

Tom did say he was going to introduce RAID1 support on cache drive. I think this opens us up to using the cache drive for protected local program storage too.

If Tom were to add RAID0 support for parity and allow us to use the dual drive SAFE arrangement we would gain allot.

I think this would be a precursor to multiple arrays and providing a gain and a way to work out thhe RAID1/RAID0 additions.

 

As far as cache flush performance, I remember someone mentioning that if the priority was adjusted then the flushes ran smoother.

I bet if SMP support was added, we would see this issue become less of an issue.

 

 

WeeboTech:  Much of what you mentioned is what I was thinking about, with the 'Transfer zone', especially the SAFE33 and SAFE50 you had mentioned awhile back (I couldn't remember their names).  'Transfer' is just a starter for a name, could be better.  'Hodge-podge' is unlikely, but descriptive.  Perhaps JBOD zone.  I envisioned this zone as one that is a collection of miscellaneous drives and arrays, each with its own configuration, fixed or removable, auto mount or manual, and individually configured for shares, exporting, and security options.

 

JoeL:  You're right, and my last paragraph includes some of the ones you mention.  I'm in no way trying to influence the priorities.  As to the blocking issue, I've wondered if Tom had been waiting to include the recently released 2.6.25 kernel, as I believe it may have some improvements in this area.  He may have been wanting to see if it would make a difference.

If altering the priority of pdflush made a difference during a test, then SMP will probably make a difference.

However, I do believe that this may require more ram so that data could still be received while buffers are still being flushed.

I agree with Joe. I think there are other things that should be programmed first into unRAID before a multi-raid system is developed. I do like the idea of running RAID 1 on the cache drive though. My main concern with using the cache drive is that if that drive fails I lose everything on the drive that didn't get transfered to the array. If you only copy cache to your array once a day, on average every time you write to unRAID your data will be unprotected for the first 12 hours. Adding a RAID 1 setup to the cache drive would be a very nice addition.

 

Once some of the more basic features are added then I'd welcome the idea of multi arrays. Of course, if adding multi-arrays would be a simple thing for Tom to do then I'd welcome that right now as well. But if it's a time consuming task to complete, I'd rather that time be spent on other things.

 

I like your idea's though Rob. I think creating a new product called unRAID Multi would be a good idea as well. Increase the price a bit and offer current users a lower upgrade fee. That way those who can use the feature can buy it if they'd like, and Tom won't be developing this just to get new users... his old(er) users that already run unRAID may be will to buy an upgrade to run unRAID multi.

 

I think offering a multi raid environment would definitely set unRAID ahead of it's competition though.

Very interesting post.  I like all of the ideas you presented.

 

Your post got me thinking that enhancement requests seem to fall into the following categories:

 

1 - Adjust and refine:  These enhancements have to do with cosmetic / usability enhancements, GUI changes to show more info, buttons to easily gather technical information, grow the number of disks in the array, etc.

 

2 - Core enhancements:  These enhancements represent truly new capabilities. I'd include the cache drive in this category, support for multiple arrays/zone (as RobJ articulates very well), allowing removal of a disk without losing parity

 

3 - Linux "techee" enhancements - Like NFS support

 

4 - Non-RAID enhancements - Like ability to run VMware, BNC (music software), support UPSes, etc.

 

As I started thinking about these, I came to the conclusion that Tom could satisfy most of 3 and 4 by allowing unRAID to run in a full Slackware (or perhaps more mainstream flavor of Linux like Ubutu).  He could make the installable module still look for the special USB .KEY file so as to preserve his protection against piracy.  By decoupling unRAID from the OS as a installable product, people could run any other apps on their server that they wanted to run.  Tom could continue to offer the Slackware USB OS option for those interested in a pure "unRAID appliance" model.

 

The OS provides many RAID type features already.  By "playing nice" inside other OS environments, unRAID users could take advantage of those other options based on their own personal needs.  This would address many of RobJ's "zones".

 

There really are not than many "pure unRAID" enhancements to satisfy most users here.  They are mostly in the "adjust and refine" category. The only "core enhancement" that I think is a high priority is the ability to remove a disk without losing parity, and maybe supporting multple arrays (low on my list but others may find this useful).  Once these are done, I would see requests for enhancements going way down.

I would be the first to reply to your post and then I thought of first reading others' opinion.

 

Well to be honest, your "design" seems a little far fetched.

I mean it will make things immensely more complicated, something I know for a fact Tom will never agree.

Add to that that I am with Joe, that a few things HAVE to be added much before such a plan starts to take shape; and no, not by the community, some things need to be "official" (also syslog and config management).

 

On the other hand, I am sure Tom (now that Lime Tech is his full-time job), NEEDS to think of things that will keep his product in front and competitive enough.

So what you say may is a little far fetched as I characterized it, but not "out of the question" (I am not to decide of course).

 

My first priority would be (1) retain and enhance stability even further, (2) add "basic" features at least to the point the competition goes today (like mail reports, error management, much better front end interface, ability to add USB disks etc. without a geek factor etc.) and only then go on to add ideas that would in fact mean (because what you list this is what it means) a total rewrite.

 

In fact if such a product ever comes (from this company) I would name it unRAID "something", that would clearly state that it is a DIFFERENT product (and believe me, for long, many will choose "unRAID classic", because again believe me, for quite some time this unRAID "something" will be WAY back in stability compared to what unRAID has achieved now.

 

BTW, where is unRAID 4.3 release?