Is it OK to create a folder that acts as a placeholder? At the moment, using Split level 2, there is no folder for this share on the disks that data hasn't "spread" to yet.

 

As part of my backup process, I'd like to use the disk shares. This way if a disk fails, and I am unable to rebuild from Parity, I can get a new disk, restore to replacement disk from backup, and get back to where I was.

 

Will creating a folder manually cause a problem with Parity or UnRaid's algorithm in any way? I'm using Highwater and split-level2.

 

 

There is no problem with using the disk shares to directly place a file on a specific disk (and is the way to do it when you want to control file placement exactly).  It will still show up in the correct place under the user shares.

Is it OK to create a folder that acts as a placeholder?

 

Sure, that's no problem at all.  Just create a new folder on the disk with the name of the share.

 

Will creating a folder manually cause a problem with Parity or UnRaid's algorithm in any way?

 

Absolutely no problem.  Writing to any disk will automatically update the parity.  In fact, many of us write a lot of stuff directly to the disks instead of a user share just to control file placement.

 

 

As part of my backup process, I'd like to use the disk shares. This way if a disk fails, and I am unable to rebuild from Parity, I can get a new disk, restore to replacement disk from backup, and get back to where I was.

 

While this concept works if you're simply backing up the individual array disks; whether it's the most efficient process depends on what your backup target is.    If, for example, you maintain a 2nd UnRAID server for your backups, it's probably better to just backup from the share -- so that ALL of the share is backed up, regardless of which disk(s) are involved at any given instant.    You can always restore the "pieces" that are missing from a failed disk by simply using a synchronization utility to sync your backups to the array ==> only the "missing" files will be copied in that process;  so you don't have to track what's on which disk.

 

Thanks Gary.

 

On the topic of backups I am just replicating over slowly changing data like family videos, photos etc. The other files on each disk are VMDKs from the VMs. But those are backed-up via other mechanisms.

 

I am tempted to do as you say, have a second UnRaid server, since I have a spare key/license combo. Perhaps take a Norco case that can handle 24 drives, and split 12 @ Primary and 12@ backup. that would be nutz! if money flowed freely, that would be possible.

For most folks, a complete set of backups will fit on a single 2-4TB disk.    Pictures, music, documents, e-mail, etc. simply don't use the space the movies, VMDKs, system images, etc. use.

 

If that's the case, you can use a single external drive (may as well make it a 4TB unit); and run a synchronization utility to sync it with your user shares -- that way there's no concern about WHICH specific UnRAID disk contains the files.

 

I've thought about just creating a 2nd (it would actually be my 3rd) UnRAID box for backups;  but what I do instead is this:  (a)  I have an external eSATA caddy [http://www.newegg.com/Product/Product.aspx?Item=N82E16817153071 ]  that I keep my current backup disk in;  (b)  Whenever I write a new file to the UnRAID server (DVD, VMDK, etc.) I also copy it to my backup disk;  ©  When the backup disk gets full, I replace it with a new one, label the full one, and store it in a disk box [http://www.amazon.com/WiebeTech-DriveBox-Anti-Static-Hard-Disk/dp/B004UALLPE/ref=lh_ni_t?ie=UTF8&psc=1&smid=A1XBPHGHAXLHDG ]

 

I store my backup disks in a fireproof, waterproof safe  [This is the main reason I don't just use an extra UnRAID server].    Note that in the event of a disk failure in UnRAID, the restore process would be like this:  (a)  Put Bakup#01 in the caddy; run a SyncBack profile to backup the backup disk to the UnRAID array => note that this will only copy anything from the backup disk to the array IF it's not already on the array;  (b)  Repeat (a) for the rest of the backup disks (in order).

 

Note that the process I use works perfectly for static data (DVDs, TV recordings, music, etc.).    If you have large, frequently changing files it's not as perfect -- you'd end up with multiple copies of the same file on different backup disks => but the restore process still works fine, as what will happen is each subsequent version encountered will replace the previous one ... so as long as you process the backup disks in order, everything will be perfect when done.  [but in reality what I do for my system images and virtual machine VMDKs is keep those backups in their own shares; and back them up to a single 4TB drive]

 

for a simpleton like me, your process seems to require discipline. i'm trying to do something that is as set-it-and-forget-it as possible (with some occasionally tuning)...I used to store my physical media in the basement. When Sandy hit, I lost a large part of my disks that were at my parent's house. I do have digital versions, but now, this is my last good copy. If something were to happen to my digital 

 

I'm almost tempted to go get bd-blanks and just make physicals of my digitals... but the time involved in doing that is not something i'm willing to spend.

 

thanks for your input around here :-)

for a simpleton like me, your process seems to require discipline

 

:)    ... I think any good backup plan requires discipline !!  Mine's not onerous at all ... but it does require copying new data to 2 locations.

 

If something were to happen to my digital 

 

Thus the reason for backups !!  NO data you don't want to lose should EVER only be in one place !!  I always tell folks you should always assume that your hard drive is going to catastrophically fail every night at midnight ... if there's anything you don't want to lose on it, be sure it's backed up !!

 

I'm almost tempted to go get bd-blanks and just make physicals of my digitals... but the time involved in doing that is not something i'm willing to spend.

 

A few years ago I did essentially that.  I still have about 2,000 DVDs of backup data I burned over a period of a few years !!  [stored in 300-disc binders]    Bur THAT was an onerous backup process (as you know).    With modern high-capacity hard drives, it's much simpler to just keep all the backups on hard drives.    If you want really simple, just build a 2nd UnRAID server and use an automated utility to manage your backups.  I had considered doing this;  but decided against it for two reasons:  (a)  It's really not much trouble to implement the process I use (as I noted, the only real "discipline" you need is to make the 2nd copy whenever you're writing to the array); and -- probably the main reason -- (b) you can store the disks in a fireproof, waterproof safe.  While not as secure as a good off-site backup, I'm very comfortable that they would survive just about any foreseeable catastrophe [The safe is bolted to my concrete slab, so it would even survive something like the tornado that just hit Oklahoma].

 

well the way i figure it either crashplan or carbonite ... can back up the critical data. sure the restore process can be slow, but the beauty is, the data is within reach, in the event of a natural disaster. If i can get to a location where there is power/internet, I can produce any documentation needed (be it a scanned passport, license, marriage certificate, etc etc)... whereas if it's in a safe at the house that just got demolished, i'd have to wait for the clean-up to happen before i can get to said safe.

 

hell if the cell towers are still up, both providers have apps that'll let me pull stuff up on my phone. i'm OK with a slow rebuild of my data, as in a scenario like this, other issues are going to be more pressing.

 

I've created my placeholder folder, and have now added it to the backup set. this way i don't have to remember to keep an eye on it. All I need to check is that the backup is running.

 

funny thing, somehow I hosed a 500GB VMDK... i'm a bit busy, but soon, i'm going to try to run a scan on it... if scan doesn't fix, then I will be going thru my first semi-large restore. let's see if it take s a couple of weeks on my 50-mbit download or a month+

Carbonite works very well ... I don't use it, as I've got far too many TB of data to backup via the cloud; but I have considered using it for our pictures, documents, etc.    As you also apparently do, I have scanned copies of virtually every document/bill/receipt/etc. for a very long time ... and these simply don't use all that much space by modern standards.    I DO have what I consider VERY safe backups of them all ... but despite that they're all in the same physical house - so Carbonite is a cheap way to eliminate that point of failure [While I'm fairly confident my safe would survive just about any foreseeable event, you never really know].

 

How did you back up a 500GB VMDK to the cloud??

 

i'm using crashplan... i have a carbonite subscription that expires november, that i am not using at all anymore.

 

the VM itself is backed up by CP (i'm on the family plan) ... so i'd really be downloading the 500GB of contents of the VMDK, and not an actual 500GB vmdk. if that makes sense.

the VM itself is backed up by CP (i'm on the family plan) ... so i'd really be downloading the 500GB of contents of the VMDK, and not an actual 500GB vmdk. if that makes sense.

 

Understand -- that's still a lot of data to upload to the cloud => do you have fiber ??

 

actually have almost 4TB of data on the cloud. I used to have fiber with a 25mbit down/25mbit up connection. since moving, the best available to me now is 50mbit down/6mbit up.

 

The initial upload took several months. now, it's just incrementals, so no more than 4-5GB any given day. When I go on a week long family vacation, between photos and videos, we rack up about 20GB. CP is the only 3rd party I rely on. I don't use flicker/fb/instagram/picassa or any of the other photo-sharing apps. IF I want to share photos with family, I host them on my own server (WHS/SBS) or email a one-off attachment. Photos and videos taken on my phone are synced nightly to my server, which eventually shoots them over the wire to CP. Camera media is sometimes uploaded while on vacation, other times after returning. The upload to CP averages anywhere from maxing my connection to about 1/2 the speed. Either way, I keep an eye on the backup, and when CP has it, then I clear out the memory card, to have ready for the next round.

 

 

lol could be worse! check this guy out:

 

http://arstechnica.com/information-technology/2013/05/fios-customer-discovers-the-limits-of-unlimited-data-77-tb-in-month/

 

 

lol could be worse! check this guy out:

 

http://arstechnica.com/information-technology/2013/05/fios-customer-discovers-the-limits-of-unlimited-data-77-tb-in-month/

 

"... a rack of seven servers with 209TB of raw storage in his house ..."  !!!!!!!!  :)

 

I have to admit he's got you beat 

 

 

"... a rack of seven servers with 209TB of raw storage in his house ..."  !!!!!!!!  :)

 

I have to admit he's got you beat 

 

it's obvi that i'm totes jelly.

 

lol he's literally got 200TB more than me. i only have a lowly 9TB in my array, with the potential of another 2TB drive. wow.

 

 

"... a rack of seven servers with 209TB of raw storage in his house ..."  !!!!!!!!  :)

 

I have to admit he's got you beat 

 

it's obvi that i'm totes jelly.

 

lol he's literally got 200TB more than me. i only have a lowly 9TB in my array, with the potential of another 2TB drive. wow.

 

Don't feel alone -- I suspect he's got just about everyone beat !!    I've got two UnRAID servers with a total of 39TB, plus perhaps another 20TB on my various PCs ... and he's still got more than 3 times what I have !!

 

But I'm less jealous of his storage capacity than I am of his upload BANDWIDTH => he's averaging about 100Mb/s 24/7 !!!!

 

I have three unRAID servers: 1 with ~16x3TB (60-70% full), 1 with 4x3TB and ~16x2TB(50-60% full), last with 6x2TB (40-50% full).  Then 4 SageTV servers with recording drives: 1 with 6x3TB (70% full), 2 with 5x3TB (80% and 50% full) and the last with 10x1TB & 11x2TB (50% full).  So ~176TB total storage which still falls short - certainly on the usage front I assume.  Also because I don't like to fill the drives completely full (only to 92-95%) and some drives have specific shares on them with as little as 30% used I have ~18TB of unusable space.  When the 5TB WD Reds come out I may replace the drives above.  I could even consolidate down to one unRAID server again.  But I doubt it.  I definately don't have the bandwith.  Am supose to get 15/5 but am lucky if I get 10/1.

I have three unRAID servers: 1 with ~16x3TB (60-70% full), 1 with 4x3TB and ~16x2TB(50-60% full), last with 6x2TB (40-50% full).  Then 4 SageTV servers with recording drives: 1 with 6x3TB (70% full), 2 with 5x3TB (80% and 50% full) and the last with 10x1TB & 11x2TB (50% full).  So ~176TB total storage which still falls short - certainly on the usage front I assume.  Also because I don't like to fill the drives completely full (only to 92-95%) and some drives have specific shares on them with as little as 30% used I have ~18TB of unusable space.  When the 5TB WD Reds come out I may replace the drives above.  I could even consolidate down to one unRAID server again.  But I doubt it.  I definately don't have the bandwith.  Am supose to get 15/5 but am lucky if I get 10/1.

 

holy smokes! how many people live in your household to consume that much entertainment? I'm using lowly 2TB drives @ the moment; the 3TBs are still not cost effective, and the 4TBs are even less so. I'm guessing to use a 5TB, all the current gen hardware/software components will need an overhaul. Until I fill all of my current server, I will have to stick with the 2's. First step would be to upgrade the br10i to a card that can handle the higher TB drives. Then, carry on to either a bigger case, or bigger drives. time will tell.

 

 

holy smokes! how many people live in your household to consume that much entertainment? I'm using lowly 2TB drives @ the moment; the 3TBs are still not cost effective, and the 4TBs are even less so. I'm guessing to use a 5TB, all the current gen hardware/software components will need an overhaul. Until I fill all of my current server, I will have to stick with the 2's. First step would be to upgrade the br10i to a card that can handle the higher TB drives. Then, carry on to either a bigger case, or bigger drives. time will tell.

I find that 3TB drives have recently become more cost effective than 2TB ones.  I would not be surprised if that mantle does not get handed to 4TB drives in the quite near future. 

 

As far as I know, barring an unforeseen complication, when they arrive 5TB drives should work on unRAID without any need for change.

I have three unRAID servers: 1 with ~16x3TB (60-70% full), 1 with 4x3TB and ~16x2TB(50-60% full), last with 6x2TB (40-50% full).  Then 4 SageTV servers with recording drives: 1 with 6x3TB (70% full), 2 with 5x3TB (80% and 50% full) and the last with 10x1TB & 11x2TB (50% full).  So ~176TB total storage which still falls short - certainly on the usage front I assume.  Also because I don't like to fill the drives completely full (only to 92-95%) and some drives have specific shares on them with as little as 30% used I have ~18TB of unusable space.  When the 5TB WD Reds come out I may replace the drives above.  I could even consolidate down to one unRAID server again.  But I doubt it.  I definately don't have the bandwith.  Am supose to get 15/5 but am lucky if I get 10/1.

 

holy smokes! how many people live in your household to consume that much entertainment? I'm using lowly 2TB drives @ the moment; the 3TBs are still not cost effective, and the 4TBs are even less so. I'm guessing to use a 5TB, all the current gen hardware/software components will need an overhaul. Until I fill all of my current server, I will have to stick with the 2's. First step would be to upgrade the br10i to a card that can handle the higher TB drives. Then, carry on to either a bigger case, or bigger drives. time will tell.

I don't watch and delete.  I record and save.  So just me.  Do need to take some time and delete duplicates could drop the usage by another 25% or less.  HD-PVR recordings take 10-20GB each for 2-3 hour movies - SD are 4-6GB each for same movies.  OTA takes ~3+GB for HD and ~1+GB for SD and these are where the most duplicates are.  I have NO music and 250GB of pictures.

holy smokes! how many people live in your household to consume that much entertainment?

 

Consume?  I suspect he's just like many of us -- a COLLECTOR      My UnRAID storage figures don't count my Beyond TV server that has 10TB of recording space ... we DO delete things from time-to-time; but I don't worry about doing so;  if it starts to get full I just add another drive 

I have FAR more stuff than I'll ever watch !!!

 

I'm using lowly 2TB drives @ the moment; the 3TBs are still not cost effective, and the 4TBs are even less so.

 

Definitely don't agree with this comment.  Consider:  Using Newegg pricing (a reasonable supplier -- not always the cheapest, but certainly competitive) ....

 

At the moment, 2TB drives cost ~ $99.  If you go with high-quality WD Reds they're $120.   

 

3TB drives cost as little as $135.  A 3TB WD Red costs $151.

 

4TB drives start at $190 (a Seagate DM series).  There are no 4GB WD Reds (yet).

[Note that these are often on sale for ~ $150 ... see the "Good Deals" section (they are available for $150 today, for example) -- but since I'm using Newegg pricing, I'll use $190 for these comparisons) ]

 

So ...

 

2TB costs $49.50/TB, or $60/TB for WD Reds

 

3TB costs $45.00/TB, or $50.33/TB for WD Reds

 

4GB costs $47.50/TB  (n/a in WD Red)

[and if you use the non-Newegg sale pricing, they're $37.50/TB]

 

Thus BOTH 3TB and 4GB are more cost-effective than 2TB drives 

 

 

Thus BOTH 3TB and 4GB are more cost-effective than 2TB drives 

And that's just the raw media cost, the full cost is much more, as each additional SATA connection has a pretty much fixed cost, no matter what's attached to it. For grins, try pricing any combination of using 2TB drives to achieve a total data capacity of 20TB. If you use 4TB drives, you don't even need an extra controller card, you can use a cheap MB with 6 onboard SATA, a readily available consumer case, and you don't even really need a serious PSU. To get to 20TB using 2TB drives, you will have to have a controller card, a server case, and a high quality single rail PSU. Even if someone GIVES you half the 2TB drives, you are still going to be at a cost disadvantage to the 4TB drives. If you already have all the good hardware, there is even less incentive to buy 2TB drives, unless you are stubborn like me and haven't upgraded past version 4.7  If it ain't broke...

Thus BOTH 3TB and 4GB are more cost-effective than 2TB drives 

And that's just the raw media cost, the full cost is much more, as each additional SATA connection has a pretty much fixed cost, no matter what's attached to it. For grins, try pricing any combination of using 2TB drives to achieve a total data capacity of 20TB. If you use 4TB drives, you don't even need an extra controller card, you can use a cheap MB with 6 onboard SATA, a readily available consumer case, and you don't even really need a serious PSU. To get to 20TB using 2TB drives, you will have to have a controller card, a server case, and a high quality single rail PSU. Even if someone GIVES you half the 2TB drives, you are still going to be at a cost disadvantage to the 4TB drives. If you already have all the good hardware, there is even less incentive to buy 2TB drives, unless you are stubborn like me and haven't upgraded past version 4.7  If it ain't broke...

 

 

True -- the real cost advantage is even greater for the high capacity drives.  I was treating the "infrastructure" (motherboard/CPU/case/PSU) as "sunk costs" and just looking at the actual price of the raw storage.

 

But the large drives are really FAR more cost-effective when you look at the cost of starting from scratch and buying the system.    I suspect a lot of folks who haven't migrated beyond 2TB are using the very understandable "If it ain't broke, don't fix it" philosophy and not upgrading from v4.7 ... and THAT's the real reason to stay with 2TB drives.    I'm in both camps ... my media server still has ample space (24TB), so it's still running v4.7 and likely will continue to do so even after v5 final is out (at least until I need more space and migrate to 4 or 5 TB drives).    My backup server uses all 3TB WD Reds.

 

Note that not only does the physical hardware cost more to support an equivalent amount of storage with lower-capacity drives;  but your power bill will also be higher  .    My backup server idles at 20W, and still uses < 50W with all drives spun up and doing a parity check -- well below what my media server idles at, and 1/3 of what the media server uses when spun up.

 

hmm - last i checked the 2tbs go as low as $70ish, and the 3's are always at $100 to $150. i did upgrade to 5.0, so that i can use my br10i; but THAT's where i should've spent the extra, to get one of the others that support >2TB... if i had done that, i would want the higher capacity drives to achieve higher density. my case is filled to the gills at the moment, so anything beyond this is an uphill task