Is anyone else suffering from  Parkinson's Law of Data Storage?

 

When I filled out my unRAID server with 6 disks and oodles of free disk space about six months ago I thought I was set for at least a year.

 

Six months on, I'm hitting disk saturation. It seems my current banes are DSLR Raw files, HD home video files and bargain bin Blu-rays.

 

I really should start culling through all the photos and home video files. Maybe, start compressing some of the lower rated movies to a more manageable size. On the other hand, should I just wait for the next 2TB deal and expand or does this lead down the road to digital pack ratdom and digital data damnation. 

How much is your time worth to you?  If it takes you 4 hours to trim and compress your files so that you don't have to buy a new hard drive, is that worth the $65 - $80 you would save?

How much is your time worth to you?  If it takes you 4 hours to trim and compress your files so that you don't have to buy a new hard drive, is that worth the $65 - $80 you would save?

I think its a more interesting question than a quick cost benefit analysis. Hence, the reference to Parkinson's Law.

 

Edit: To clarify, there is no dilemma on my part on whether to expand the array or not, given the whole point of using unRAID is its easy expansion capability and the low Gb/$. Being on the threshold of an array expansion, just got me thinking about Parkinson's Law.

I would say trim and delete undesired files.  With unRAID there is a point that you can't build past.  Personally, I had a problem backing everything up to DVDR discs.  Where did that get me?  I now have over 3,000 that I have to go through and backup onto my server.  During this process I am finding that 65% of the content is out dated warez and stuff that I now have in better quality.  Save yourself the hours it would take once your server if full and start doing it now.  :'(

Also have a plan for what is considered acceptable compression. I back up blu-rays and compress them to h.264 using a constant quality that still yields good looking video. Average compression of films ends up being about 2.2:1, while compression of computer animated stuff(pixar style) at the same constant quality ends up being 4:1. To me the space savings is significant enough to spend an hour or two a week setting up a transcode queue to compress these videos.

 

Home video and Pictures take a bit more consideration. If you do a lot of editing, keeping them uncompressed is probably best. But if you're just storing the images and plan to print out an occasional 5x7 or make a scrapbook, Jpeg is probably fine.

 

I guess what I'm saying is: know in advance how you will use your storage and how you're going to store your data. It will save you time, frustration and headaches later on.

Is anyone else suffering from  Parkinson's Law of Data Storage?

 

When I filled out my unRAID server with 6 disks and oodles of free disk space about six months ago I thought I was set for at least a year.

 

Six months on, I'm hitting disk saturation. It seems my current banes are DSLR Raw files, HD home video files and bargain bin Blu-rays.

 

I really should start culling through all the photos and home video files. Maybe, start compressing some of the lower rated movies to a more manageable size. On the other hand, should I just wait for the next 2TB deal and expand or does this lead down the road to digital pack ratdom and digital data damnation.   

 

 

YES!

 

I too started out about or seven months ago, with six disks. I figured that that should hold me at least a year (or longer).

 

I guess I didn't count on in the pent-up need that was happening in my house. Like many users, I had been "getting by" with my storage needs, a few USB drives here, storing some files over here, and there, etc. Also, I have (had?) a large number of TV shows captured and had the commercials edited out, and the results burned to a DVD.

 

However, once I installed a brand new shiny unRAID server, and we became enamored with having a home server, well things got out of hand. Don't get me wrong, I had built other servers before, with various operating systems, but they were so unreliable that my household was unwilling to use them, or ended up storing files locally, AND on the server, defeating the point of the server.

 

Now when I built this server (unRAID), I thought I would be 'smart' and build it out to accommodate 15 drives. I figured 15 drives would be WAY more than I would need anytime soon. It appears that my assumption may have been wrong. First, my DVD library is way more that I had figured. Videos, that were mine, started to appear from nowhere, and needed converting to be stored on the server. They were always here, just not stored all together, because of space concerns. Now the household desire is to have them stored, and organized, on the server for general access to everyone. Second, once vast amounts of (reliable) storage was available, the immediate desire was to store more stuff. Things that might not have been saved are now being saved. Lastly, now that there is remote (server based) storage available, local storage is now looked upon as 'unreliable' and the desire has been to push files that were once stored locally to the server, to keep them 'safe'. Everything that once was just stored locally has now been pushed to the server, including a centralized 'music' library for the house.

 

This is the price of success I imagine.

 

I am even the worst culprit, so I guess I shouldn't complain.

 

I convinced a friend to install an unRAID server, and he is going through much the same thing. The need, or at least the desire, was there for a long time, but not the availability. So far I have been purchasing HDD's on sale, one and two at a time, and so have been able to keep up with the demand. I have not reached my 15 drive capacity, yet, but sometimes I get worried. I will be looking to the future, and 3 and 4 TB drive support to keep me in my current box. Otherwise I may need to start thinking big, like a Norco.

 

Bruce

I guess what I'm saying is: know in advance how you will use your storage and how you're going to store your data. It will save you time, frustration and headaches later on.

This is the point, I think Bruce in the post above and I did exactly that, plan. Hence, becoming a victim of Parkinson's Law of Data Storage. 

Well, if you already purchased the Blu-Rays, you have an uncompressed backup copy for archive already.

 

I'd say compress those to 1080p/720p MKVs.

 

Then figure out about compressing your Home Videos (if that's necessary or not)

 

Finally, upgrade your hard drives to 2TB if they're not already.  I think you can still get the Hitachi 2TB CoolSpin from Newegg for $70 after MIR.

Oh I know the feeling. 4tb of storage and 2tb parity when I started backing up all my dvds etc around 2 months ago. I though yeah that should get me over the back up hump and leave me plenty of spare...oh wow. Was I wrong, I keep backing up everything. it's amazing having it all in reach with little effort. Digital Glutony lol.

I made the mistake of backing up a few of my PC's with Clonezilla and Acronis now I have family members that are asking if I can store images of their machines. I thought how bad could it be. Well now I'm being asked what if I do upgrades can I reclone it again? LOL

 

I guess people even without space want space which cracks me up.

 

I don't mind helping, but good lord I don't want to be somebody else's unpaid "Digital Safety Deposit Box"

I made the mistake of backing up a few of my PC's with Clonezilla and Acronis now I have family members that are asking if I can store images of their machines. I thought how bad could it be. Well now I'm being asked what if I do upgrades can I reclone it again? LOL

 

I guess people even without space want space which cracks me up.

 

I don't mind helping, but good lord I don't want to be somebody else's unpaid "Digital Safety Deposit Box"

 

All of my family/friends come to me when its time for a new computer. The way it works is I buy it, then they pay me back. I have it shipped to me, I strip all the crap off, install any software they have provided, completely update everything, then make a backup with Acronis.  I have about 30 different TIB files of images ready to quickly restore someones computer. I figure, storage is cheap and in the event they totally mess up their computer (as they are likely to do), i can restore it in 15 mins or so.

 

I lost the SSD in my netbook  a couple of weeks ago. I was back up in 15 minutes after putting in a new drive since I had an Acronis image that I took after installing and activating Windows 7 and Office 2010.  Didnt even have to reactivate.  I also have the original factory image so when I put this thing on CraigsList in a couple of months, I can restore it to "factory fresh."

All of this talk of imaging has gotten me thinking that I should probably start imaging my computers too.  As it is now, all my data is backed up on either the unRAID server or via Crashplan (looking into SpiderOak and Wuala as well).  All my bookmarks are also backed up via Google (I love the Chrome/Google account integration).  So I'm able to reformat on a whim at any time and not have to worry about losing any data.  However, I do still have to reinstall the OS and all my programs.  Still, it is maybe an hour's worth of time at most:

 

- Windows 7 takes 25 minutes to install onto a HDD from a DVD (about about 10 minutes onto an SSD from a flash drive)

- I then run all the Windows updates, which probably takes 10 - 15 minutes, though often requires many rounds of restarts and downloading new updates.

- I then use the wonderful free software Ninite to install all my commonly used freeware apps (Just a couple: Chrome, Firefox, VLC, MediaMonkey, Audacity, CCCP, Flash, Java, Picasa, GIMP, FoxIt, Spybot, uTorrent, ImgBurn, TeraCopy, CDBurnerXP, 7-Zip, WinRAR, FileZilla, Notepad++, and PuTTY).  Takes maybe 5 minutes to download and another 10 to install.

- I'll then install my paid-for apps (Office, Adobe Suite, DJ and audio editing software, etc.).  This takes probably about 20 minutes of install time.

- Finally I'll let Crashplan restore my data into its original location.  If the backup machine is local, then the restore takes maybe an hour of inactive time.  If the machine is remote, then it can take a day or more.

 

So I'm looking at maybe 30 minutes to an hour of active time spent, and several hours of inactive time.  Most of these stages are just click, click, wait, so I can go off and work on other stuff.  The paid-for apps is the most annoying bit.  That's the part that I would benefit the most from by using images instead of my current method.

 

The only benefit I can think of to my method over images is that I know I'm always getting the latest version of the freeware software I use, since Ninite takes care of that for me.  Some of those apps have built in updaters, but many of them have no updating function but instead make you uninstall the old version of the software and reinstall the new one.  A bit of an inconvenience, but I suppose that is mitigated by the overall convenience of restoring from an image.  Perhaps the best of both worlds would be to create images of Windows, my data, and all my paid for apps but still use Ninite to reinstall the freeware apps each time.

 

Here's a question for you all - when using a HDD as a boot drive, I was always careful to install all Windows stuff (including updates) first, then my most used apps and data, and finally my least used apps and data.  I figured that I always wanted Windows and my most used apps and data on the outer portion of the HDD platter so that access time would be as fast as possible.  I was also careful to install things one at a time so that there was no fragmentation.  Now that I'm using SSDs as boot drives, this should no longer be a concern, right?  As I understand it, all sectors on an SSD have the same access speed, and file fragmentation is not an issue.  This means that I should be able to install multiple apps at once, or install apps while Windows updates are running and not worry about it.  Is this correct?

 

 

 

I normally just use Disfraggler to help out with fragmentation. Does it move things to the inner or outter is beyond me.

 

For me the biggest reason for imaging has always been I seriously hate to rebuild a machine when the machine says rebuild me not when I took the time. I can think of at least 3 times in my life when my machine was hit with a virus, driver update or simple stupidity of the operator that sent things screwy. So I image at least with the best build I can of the system and go from there. I can easily restore if/when needed to that point. I've never trusted the built in windows restore because I've been burnt at least twice with that.

 

Heck my last restore was because something glitched in a Linux install of XBMC on my HTPC. I was watching a flick and all of the sudden it shut down and went into a boot loop. I could of spent hours tinkering with the syslogs, messing with configuration files and god forbid fighting with getting my remote working perfectly. I instead just re-installed the image, let it grab the newest flicks/tv shows off my unRAID server and I was back up and running. The only thing that was missing was some of the check marks saying what I've watched and when I viewed the most recent movies/tv shows they where out of order. Its been a few weeks now and I've added some here and there and you would have no clue that anything has happened. My 5year doesn't care as long as his Scoobie Doo is available for his viewing pleasure.

Windows Restore is definitely crap, disabling it is one of the first things I do when installing Windows.

 

I can definitely see the appeal in imaging an HTPC, since a fresh install would require re-indexing the library from your server, which can take days (and you would still lose your 'already watched' checkmarks).

 

Maybe I'll mess around with Clonezilla when I get home tonight.  I always start with the freeware, then only upgrade to the premium software if I feel it is necessary.

Here's a question for you all - when using a HDD as a boot drive, I was always careful to install all Windows stuff (including updates) first, then my most used apps and data, and finally my least used apps and data.  I figured that I always wanted Windows and my most used apps and data on the outer portion of the HDD platter so that access time would be as fast as possible.  I was also careful to install things one at a time so that there was no fragmentation.  Now that I'm using SSDs as boot drives, this should no longer be a concern, right?  As I understand it, all sectors on an SSD have the same access speed, and file fragmentation is not an issue.  This means that I should be able to install multiple apps at once, or install apps while Windows updates are running and not worry about it.  Is this correct?

 

Hey Rajahal,

  With an SSD, you are right, no more fragmentation and you can access all data at the same speed. In fact, Windows 7 disables a bunch of things when it detects a SSD, including the weekly defrag. As for installing things, even with my SSD drives, I still only do one install at a time. If you have multiple installs going at the same time, some of them updating Windows files, on a rare occasion you can mess up Windows or the apps you installed. So I just stick "safe".

 

Shawn

Rajahal,

 

For imaging of my real windows systems I use Acronis True Image Home which fits all my backup needs almost perfectly ... it lets me do a full back up every X days with incremental images in between ... and also will allow you to set a limit on the number, total size or age of those backups.

 

For my windows virtual machines I use Paragon-Software Backup & Recovery 2011 (Advanced) Free (for personal use)

 

and as for parkinson's law ... I've got two 12 TB unraid systems running currently with a 2TB drive dedicated to system images and 7797 recordings from my HTPC ...

(easier to buy new hard drives than delete the old TV shows sometimes ...)

 

for defragging on windows systems I use MyDefrag by Jeroen Kessels

 

Bobby

Is anyone else suffering from  Parkinson's Law of Data Storage?

 

When I filled out my unRAID server with 6 disks and oodles of free disk space about six months ago I thought I was set for at least a year.

 

Six months on, I'm hitting disk saturation. It seems my current banes are DSLR Raw files, HD home video files and bargain bin Blu-rays.

 

I really should start culling through all the photos and home video files. Maybe, start compressing some of the lower rated movies to a more manageable size. On the other hand, should I just wait for the next 2TB deal and expand or does this lead down the road to digital pack ratdom and digital data damnation.   

 

I suffer from the same problem.  With 2TB's coming/going on sale for a song,  I have to ask myself how much of my time is worth $85 (or less) bucks.

 

And that always has me buying another drive and not worrying about it

Thank you for the introduction to Ninite.  Seems like an awesome app.  Raj if you feel like doing a fresh install everytime you should consider http://www.autopatcher.com/ being that I know you have a server to host the files. 

Thank you for the introduction to Ninite.  Seems like an awesome app.  Raj if you feel like doing a fresh install everytime you should consider http://www.autopatcher.com/ being that I know you have a server to host the files. 

 

Coolness I have a few machines I'd like to keep up to date and that seems like a pretty cool idea.

Thank you for the introduction to Ninite.  Seems like an awesome app.  Raj if you feel like doing a fresh install everytime you should consider http://www.autopatcher.com/ being that I know you have a server to host the files. 

 

Nice, thank you!  Sounds like I can just create a CD image of a bunch of Windows 7 updates and install them all with one click.  Sounds great!

Rajahal,

 

for offline windows patching, I use WSUS offline update myself. then again i do a lot of rebuilds at work. just map a drive to the patch folder and hit install with auto reboots and logon.

 

For imaging I use ghostcast server. I'm old school i guess.

 

 

the other backup i highly suggest for windows boxes.. WHS2011 (or WHSv1).

You can do an entire bare-metal re-install in just a few min by booting from a usb stick to the WHS2011 client. then restoring your PC to what it looked like lastnight... or say you picked up a virus... rebuild, but roll back 4 days earlier... or just get one document you deleted or overwrote weeks ago and now need.

 

I have a small atom box similar to your zbox running WHS2011.

it is backing up 4 win 7 boxes (mix of 32&64) and 2 XP boxes. all my backups going back 6 months (on the whsv1 i just migrated last week to 2011) look about 800 megs thanks to de-duplication.

 

It is one of the few things Microsoft got right... $250 server plus $100 license = well spent piece of mind.

you don't have to think about it, it just backs up every night, waking the PC if it is off..

 

Edit: I should mention it has saved my hide and few people i that know.

people that know nothing about pc's and would never have bothered with backups otherwise.

 

 

 

 

At least we know we all have a future as television stars.......

 

When they finally decide to come out with the new series

 

 

DIGITAL HORDING BURIED ALIVE 

Yup, I'm adding a new 2TB HD as I type.  I'm up to 11TB!  Who would have guessed?  I was going to replace an old 250GB IDE drive with the 2TB, but after I opened up my box, I realized I've used up all my SATA heads (4 on the mother board, 4 on two PCI cards)!!!  Damn, now I have to get a PCI card with 4 heads?  So, I decided to leave it be and swap it with a 750.  At least I'm picking up an extra 1.25TB.  BTW, all these backup strategies are interesting, what about the Mac crowd?  I've been using Carbon Copy (free) to clone my boot disc, but I just expanded that to 2TB too and put my iTunes library on it.  So I think I'll keep using Carbon Copy for everything but iTunes and then backup iTunes to the unRAID.  Media files for my editing projects are tricky.  Some of that stuff comes and goes and I don't have to worry about it.  But some personal projects and multiple use media I've just kept on an external FW drive and backup to the unRAID.