I hope this isn't against the rules or something but here it goes....

 

Stumbled on this while cruising the AVSForum and deciding if unRaid was the way to go for me.... http://www.vilett.com/disParity/

 

So basically since windows 7 has a library mode which allows u to easily group multiple folders, spanned over multiple disks, into one share.... using disParity you could achieve something very similar to unRaid with one computer instead of 2.... HTPC and Media Storage in one. ( If you don't mind the noise from the clicking hard drives or they're in a quiet enclosure like the Antec p182/3 )

 

This does not calculate parity live as you write to all the drives, instead you run a command line to create the parity drive and then just update it with an update command.... this can be added to a scheduled task to run every night or be part of a shutdown script... rebuilding a failed drive is just as easy, just run the rebuild command. You also don't have to calculate parity for whole disk.. you can just pick specific folders to protect.

 

Check out the forum, some dude on there has a 20+ disk setup with some impressive stats on parity calcs and rebuilds. Power usage wise, you can still spin down the drives. The exception being that every time you look at a share it spins up all the hard drives included in that share. Havn't Figured out a way to get around that yet....

 

Hope this helps someone who's on a tight budget and wants to protect their media files.

 

Sounds very similar to FlexRaid.

Sounds very similar to FlexRaid.

 

It does sound very similar to FlexRaid.

 

The problem I see with something like this is that it does not solve the problem I set out to cure initially.  Granted it will allow a level of protection but I wanted all my media storage to be moved out of my HTPC.  I guess you could set up a box running Windows and then run this on top of it but you don't get the real time protection that unRAID provides.

 

It seems slightly more flexible then unRAID but then it also opens up your data until you run it again to recalc parity.

Sounds very similar to FlexRaid.

 

It does sound very similar to FlexRaid.

 

The problem I see with something like this is that it does not solve the problem I set out to cure initially.  Granted it will allow a level of protection but I wanted all my media storage to be moved out of my HTPC.  I guess you could set up a box running Windows and then run this on top of it but you don't get the real time protection that unRAID provides.

 

It seems slightly more flexible then unRAID but then it also opens up your data until you run it again to recalc parity.

 

I agree 100%

After reading up on flexraid it does sound almost exactly like it, except flexraid seems to be more developed with a gui and all.  I think flexraid or disparityt is is more than adequate for media storage in my case... if u run a recalc every night than, if anything did go wrong you would only lose what you RIPPED or DOWNLOADED that day... not a big deal in MY CASE as its easily recoverable. Thanx for pointing me into flexraid as it seems like a better developed program.

Disparity was created because FlexRAID's developer was MIA for a few months, so someone created disparity as a reaction. (HINT TOM, HINT)

 

FlexRAID's dev returned to active development (more or less) while disparity's dev is not that active.

I know that the guy that makes FlexRAID does have big plans for his product (including a live parity version - not snapshot), but only time will show.

 

For me having the flexibility of... erm... FlexRAID is a great bonus. More or less what people manage to do by hacking unRAID on their own Linux build.

Since the current status of unRAID is not exactly giving a choice (either use the beta and use my extra disks or NOT use the extra disks and have the additional services of Bubba's experiment)... a FlexRAID + WHS mix is currently giving me way more services from a machine 24/7 than unRAID ever did.

 

This might change in the future - but life is sort.

 

FlexRAID I have to say, is still sorting out problems. It's way less mature product than unRAID - it is usable though.

 

Anyway, I hope Tom really has something big in his mind for 5.X and that 5.X is really in this homo sapiens sapiensis lifetime.

 

It seems slightly more flexible then unRAID but then it also opens up your data until you run it again to recalc parity.

 

This was always my concern regarding flexraid. However realistically if you ever intend to use a cache drive with unraid to bolster speeds - you hit exactly the same issue with unraid.

 

Slightly different mechanism but the end result is the same.

Exactly boof.

 

This is not as bad as it sounds though. You see not ALL your data are "exposed" until you recalculate parity, just your NEW data.

Most of our data are "static" ...they are just there to stay. A daily flexRAID resync schedule is more than enough for most. Worst case scenario you loose data from the last 24h - statistically your last 12h.

 

I can "bear" this personally.

 

My current concern about flexRAID is not its snapshot philosophy. It's how fast you can actually do the resync. If it was just a matter of having already marked the differences and just port them to parity would be great. Not the case exactly, currently. Even resync takes time.

 

 

 

Exactly boof.

 

This is not as bad as it sounds though. You see not ALL your data are "exposed" until you recalculate parity, just your NEW data.

Most of our data are "static" ...they are just there to stay. A daily flexRAID resync schedule is more than enough for most. Worst case scenario you loose data from the last 24h - statistically your last 12h.

 

I can "bear" this personally.

 

My current concern about flexRAID is not its snapshot philosophy. It's how fast you can actually do the resync. If it was just a matter of having already marked the differences and just port them to parity would be great. Not the case exactly, currently. Even resync takes time.

 

And while I generally agree with this statement, some of use run a torrent client on out unRAID machine.  This torrenting will completely screw up the parity if not done in real time.  I guess you could move the torrenting to a separate drive outside of parity protection but I would still prefer to have it always being done.  This does mean that 2 of the drives in my system do not ever spin down, but I am willing to put up with that for the sake of always on parity protection.

disParity has problems with fragmentation. I know of someone who had to delete and rebuild parity because of fragmentation on the parity drive. He's now buying a 2TB drive as parity for the data drives that are max 1.5TB

 

The problem I have with offline RAID is that you have to keep building parity. I can't trust that, so I prefer unRAID.

Sure, it's good to have choices.

 

 

Here's something I could not figure out with Flexraid and even the developer on the AVS Forums never really answered.

 

Say I protect complete drives like unRAID does - I know Flexraid can protect directories too but I'd have a "movie" drive so I want it all protected anyways.

 

So, I'm using Flexraid to parity protect 3 data drives. From what I understand, it basically does a stripe parity calculation. I then add a few big files to one drive and remove a few more on one drive. Before I rebuild the parity another drive fails. Here's the big question - would I not get data corruption on the rebuild drive in the same physical locations as the new files and possibly the deleted files on the other drive???

 

There were posts that this isn't possible yet no explanation of how it could be avoided. The claim was that you could change the data all you wanted on one drive and then recover another drive if it failed and that makes no sense to me.

 

I can understand that it would be possible to change the data on one drive and then recover that drive if it failed and it would go back to the old data.

 

Peter

 

I can answer that.

 

1) FlexRAID currently uses a form of RAID4, exactly like unRAID does. i.e. data are data and parity is separate

 

2) you are confusing what FlexRAID does - it protects files, not disks... so in your example:

 

Restore doesn't have to happen "on location". So if a whole disk failed, FlexRAID will in fact "care" (and understand) that some of the files in parity (the failed and yes also the ones you deleted before refreshing parity), are now missing. That's all FlexRAID cares.

 

So if you go and "restore" (somewhere you know you have space, even on the array itself), it will just recover the lost files. Be it because a disk failed or you deleted them on purpose. Or of course if you edited a file after your last parity snapshot... then you get the old version.

 

Of course if you know that DRU (risk unit - in your case a whole disk) 2 failed and you deleted data manually in DRU3, you don't need to restore DRU3 (or DRU1 or whatever) too. You only restore the failed disk contents (in your case DRU2).

 

Think of it a smarter (and way more compact) way to backup (or a backup that depends on your remaining data).

 

Clear?

 

 

Nope, not clear at all. Still not a technical explanation of how changing data without updating the parity doesn't break the protection for some files.

 

Of course if you know that DRU (risk unit - in your case a whole disk) 2 failed and you deleted data manually in DRU3, you don't need to restore DRU3 (or DRU1 or whatever) too. You only restore the failed disk contents (in your case DRU2).

 

Restoring the data on DRU2 would use both the parity and DRU3 but DRU3 was changed without updating parity. How does it get around the fact that DRU3 and parity are broken for some of the data?

 

It's not exactly RAID4 since RAID4 has striped data but a fixed parity location. If unRAID was RAID4 then the data would be striped across all data drives and the parity would be on the parity drive.

 

unRAID reads a "Stripe" from multiple drives and then builds parity data from that.

 

If Flexraid is protecting files (it can protect files, folders or drives according to it's web site) then it must read a "Stripe" from multiple files and then build parity from that. So, Flexraid must put a bunch of data sources in "parallel" and then build parity as it reads across the "stripe" created by these data sources.

 

I'm going to use a simple example. Say I have 4 files, each 5gig in size. I use Flexraid to create parity. So, flexraid puts the 4 files in parallel and reads the "stripe" data from them to create the parity. Now, I change file 1 and before I rebuild parity I lose file 4. How can file 4 be rebuilt???

 

I guess my point is that any form of parity protection relies on reading multiple blocks of data and then creating a parity block that matches. You can lose any one block in that "stripe" without losing any data. So, how can changing one file not affect the parity protection for another file?

 

The responses saying that you can change any data you want and still recover any other unchanged data due to a failure just makes no sense to me. It's the equivalent of turning off the unRAID parity and changing the data on disk1 and then saying that disk2 can still be recovered.

 

Peter

 

I'm going to use a simple example. Say I have 4 files, each 5gig in size. I use Flexraid to create parity. So, flexraid puts the 4 files in parallel and reads the "stripe" data from them to create the parity. Now, I change file 1 and before I rebuild parity I lose file 4. How can file 4 be rebuilt???

 

I guess my point is that any form of parity protection relies on reading multiple blocks of data and then creating a parity block that matches. You can lose any one block in that "stripe" without losing any data. So, how can changing one file not affect the parity protection for another file?

 

The responses saying that you can change any data you want and still recover any other unchanged data due to a failure just makes no sense to me. It's the equivalent of turning off the unRAID parity and changing the data on disk1 and then saying that disk2 can still be recovered.

 

Why not try it yourself and see what happens? It won't take you long to get your scenario above up and running, there are no dependencies beyond the flexraid server binary and a client executable.

Yes I understand what you are saying.

 

I think this is going a bit too far for unRAID forum. I would take these questions to FlexRAID forum directly.

 

Also as boof says, FlexRAID doesn't really involve any serious dependencies on your system, so can just experiment yourself with let say three folders full of files (that you also safely have somewhere else) plus a parity folder.

 

 

Why not try it yourself and see what happens? It won't take you long to get your scenario above up and running, there are no dependencies beyond the flexraid server binary and a client executable.

 

Because I really can't be bothered and I'm not installing some extra software I don't want to use. I had posted this question on the AVS Forum where the developer was posting and never got a suitable answer there either.

 

I don't take anyone's word that it "works" and will recover the data. If this is true then show me how. Both unRAID and Flexraid are claimed to do the same type of parity protection. Well, if unRAID only updated parity once a day and I changed block 18,352 on disk1 right after the parity calculation then block 18,352 on any other disk would be recovered wrong if I had a failure of any other disk. That type of random data corruption is not really acceptable to me.

 

I just ran some scripts on my movie metadata files one disk1 of my server. These very small files get added as movies are so I'd bet they are spread out over the physical drive capacity. I can imagine how much data damage that'd do if the parity was not updated real-time and I had to rebuild one of the other drives after a failure. :'(

 

Just giving anyone who considers this as a data protection solution some food for thought.

 

Peter

 

I tall depends on data moves and changes which in turn depends on what access you give to user to make those changes.

 

I could easily make 10 disks in one of my unRAIDs read only since the data never changes. Never is perhaps to strong it changes perhaps once a year. By next year 15 disks could be made read only easily.

 

Done right and organized and restricted correctly i can see real advantages in a "line in the sand" approach.

Trying not to get any offense by your reply (English not being my native language certainly doesn't help), I have to say one simple thing: We don't HAVE to show you, you don't HAVE to take our word for it and you don't want to install a simple piece of software (and then remove) to test it yourself.

 

I guess it's a deadlock.

Tough luck.

 

I agree on the two and a half paragraphs, but as I said you could just answer those questions yourself.

The guy that makes FlexRAID, makes it perfectly clear: It works on "archived" data. I guess dynamic data (like databases or other files that rapidly get added or removed) can be kept outside the virtual array (for example I do exactly that with my actively downloading torrents and some indexing files).

 

Nobody said it fits you or that it has to fit you.

 

 

Because I really can't be bothered and I'm not installing some extra software I don't want to use. I had posted this question on the AVS Forum where the developer was posting and never got a suitable answer there either.

 

It's hard to help you if you won't help yourself! It would take you less time to answer this yourself than it would posting to forums. Regardless, why not try the official flexraid forums instead of AVS?

 

I don't take anyone's word that it "works" and will recover the data. If this is true then show me how. Both unRAID and Flexraid are claimed to do the same type of parity protection. Well, if unRAID only updated parity once a day and I changed block 18,352 on disk1 right after the parity calculation then block 18,352 on any other disk would be recovered wrong if I had a failure of any other disk. That type of random data corruption is not really acceptable to me.

 

If you won't take anyones word on it, how can we help you? As you're not willing to take 10 minutes and try it yourself you're going to have to take someone elses word for it!!. No, flexraid does not claim to be the same as unraid. In fact flexraid specifically says that it is not ideal for the scenario you've given. And as above I don't want to take my time showing you how when you're not willing to take the time to answer it yourself.

 

FlexRAID™ Basic is suited for data that you only change a few times during the day.

If you continuously change your data during the day, every day, then FlexRAID™ Basic is not a fit for you.

 

Note that we are not talking about having an issue with continuously reading the data.

FlexRAID™ Basic will work just fine even if you read your data continuously 24/7.

It is just that FlexRAID™ Basic needs break periods where no data change is occurring to re-synchronize the RAID.

 

When I say "data change", I am not talking about adding new data files to the RAID as that action needs no break period at all.

 

I've snipped alot of that out and english isn't the authors primary language but the impression I get from the above and supporting statements around that on the website and from the faqs is that constantly *changing* existing data is not a good use case for flexraid. Adding additional data is fine but changing existing data can cause issues. I would not be surprised if these issues are the same ones you're musing over.

 

I just ran some scripts on my movie metadata files one disk1 of my server. These very small files get added as movies are so I'd bet they are spread out over the physical drive capacity. I can imagine how much data damage that'd do if the parity was not updated real-time and I had to rebuild one of the other drives after a failure. :'(

 

See above. Although if you are not *changing* existing files, only adding new files I'm not sure this will be an issue. Worse case if you restore parity you rebuild all the old files the parity does now know about (i.e your data on the failed disk) and I doubt it would remove the new ones in the process. Of course you would lose new data on the drive that failed but this is in herent to the 'snapshot' non realtime nature of flexraid. You would face similar issues with the unraid cache drive were you to use it.

 

Just giving anyone who considers this as a data protection solution some food for thought.

 

You're bashing a product for a use case it itself doesn't claim to fit and without being willing to try it to gather empirical evidence to support your claims. It's bad tasting food.

 

As slightly tastier food for thought the promise of flexraid live and flexraid NAS is quite exciting and could be the first real commpetitor to unraid in terms of feature set. However there is the issue that the autho has been promising both these things 'in a few weeks' for at least the two years I've been keeping on eye on the flexraid website and forums. When or if they ever appear is anyones guess - but flexraid live in particular could be extremely attractive if done sensibly.

 

A further two minutes on the flexraid website answers your precise question - although I leave you to interpret the answer due to the translation.

 

With FlexRAID™ Basic, what happens if I lose a drive while my RAID is un-synchronized?

If the RAID is un-synchronized because you've only added new files, you are fine.

The failed drive will be fully recovered.

In the next release, re-named and moved files will also be a non-issue.

 

However, if you edited or deleted data, only the un-synchronized portion will not be recoverable on the failed drive.

The remainder of the data will be recoverable.

So, if you edited or deleted only 1GB of data, only 1GB would not be recoverable.

 

With FlexRAID™ Basic, you should ensure to re-synchronize the RAID after data changes have occurred.

If you re-synchronize the RAID often, the probability of a drive failing while the RAID is un-synchronized will be extremely low.

 

Again, this is only a limitation of FlexRAID™ Basic.

FlexRAID™ Live! and FlexRAID™ NAS won't have such limitation.

Well, as I have already stated, I posted this question over on the AVS forum where the AUTHOR of Flexraid was posting and did not get an answer. I already know it's for archived data but even archived data can be changed every now and then.

 

I ask the question because I want everyone who's reading the thread to know the limitation and any possible "gotcha's" of the software. So having an actual answer would be much much more helpful.

 

You're bashing a product for a use case it itself doesn't claim to fit and without being willing to try it to gather empirical evidence to support your claims. It's bad tasting food.

 

Huh, the example was me deciding to do an edit on a drive of archived movies. The metadata files are small xml files describing the movies, one for each movie. They get added as each movie gets added which means they are physically spread out over the drive media. I was changing the files to modify the movie genres that were in the files, not adding these files all at once. I don't see how I could do something like this without basically losing my parity protection until parity was rebuilt. If this breaks parity more than the Flexraid web site lets on then I feel people should know. It might not seem like a big deal but if I lost another drive while doing something like that the data on that other drive would end up badly corrupted.

 

Another point is I'd have no idea what files were corrupted. Basically, I have to go through the data and figure out what is good and what is bad. Sucks to do on a 2T drive.

 

Peter

Well, as I have already stated, I posted this question over on the AVS forum where the AUTHOR of Flexraid was posting and did not get an answer. I already know it's for archived data but even archived data can be changed every now and then.

 

As before I strongly suggest you post your question on the official flexraid forums. Logic dicatates you're far more likely to get an answer there than an unaffiliated forum, regardless of where the author may or may not appear to be posting.

 

If nothing else there are many actual users of flexraid on the official forums who can answer you in leiu of the author himself.

 

I ask the question because I want everyone who's reading the thread to know the limitation and any possible "gotcha's" of the software. So having an actual answer would be much much more helpful.

 

Your question is answered on the flexraid website and I've also replicated it in this thread for you.

 

Huh, the example was me deciding to do an edit on a drive of archived movies The metadata files are small xml files describing the movies, one for each movie. They get added as each movie gets added which means they are physically spread out over the drive media. I don't see how I could do something like this without basically losing my parity protection until parity was rebuilt. If this breaks parity more than the developer lets on then I feel people should know.

 

In this instance - new additional xml files (note you are *not* editing any files, you are creating new ones this is a critical distinction in terms of flexraid) - you would not lose parity protection of your already existing data. You would also not 'lose' parity of the new xml files as you have not yet created any parity for them at this point.

 

Your worst case at this point is a reconstruct of protected data from parity - which would not include details of any xml files given they're not protected at this point. Thus some may be lost depending on the failure scenario. To avoid this you can, of course, incrementally update your parity to include the new xml files. You can do this manually, or more likely, via a scheduled tasks using cron or windows task scheduler.

 

The author is quite clear via his website about what flexraid does and doesn't do and isn't misleading anyone. You are not breaking your existing parity in any way in the above scenario.

 

It's becoming clear that you simply don't understand how flexraid works. I re-iterate that the absolute best way for you to do this would be to install it and try it for yourself. It's very easy to do so non destructively by using test data. Although it would still be quite safe to use your live data if you wanted.

 

You do not need to install this on your unraid server you could install it on any old windows (or linux) workstation for a quick try out. It's very light weight and easy to install. Configuration is a bit trickier but the web gui makes it fairly straightforward and this would be the best way to learn it's concepts.

 

If you're not interested in doing this but still insist on throwing out wild conjectures about what the program does or doesn't do as well as insinuating the author is not telling the whole truth about the mechanics of the program - then theres little I or anyone else can do to help you 

Another point is I'd have no idea what files were corrupted. Basically, I have to go through the data and figure out what is good and what is bad. Sucks to do on a 2T drive.

 

I think this snuck past as an edit.

 

You can verify parity at any point - I believe flexraid will inform you of any corrupted files (or at least what it assumes are corrupted files - i.e differences between a file on disk and the data it contains for the same file in parity).

 

I havn't tested that exact scenario, but I can say that if you remove a file and verify flexraid will flag the 'raid' as being degraded and list the problem files.

right so