I nearly posted this in the other thread I have, but I figured since the topic was totally different, it would make more sense to start a new thread.

 

I currently have a MD-1500 purchased from Limetech.  I am using this as a backup for over 10,000,000 image files.  I am currently trying to optimize the system.  I've had trouble with it running out of RAM in writing errors to log files, and my nightly backups can do hundreds of thousands to millions of files, depending on how much work has been done in a day.

 

1.)  I got more ram yesterday, 3 1GB sticks, to try to bump my RAM up to 3GB.  When booting up, the system only recognizes 2GB.  I pulled one of the 1GB sticks out, and put the 2 512's back in, still only saw 2GB.  I moved the ram around, same result.  The puzzling part is that I know it worked with the 2 512MB sticks before.  Now it's working with at least one 1GB stick, and the 2 512's, or both 1GB sticks.  On bootup, it only sees 2GB.  This is the motherboard - http://reviews.cnet.com/motherboards/asus-p5b-vm-do/4507-3049_7-32310394.html?tag=mncol;psum - so I know it supports 4GB.  Any ideas on what I can do to get it to see 4GB?  Should I get another 1GB stick, and see if that works? 

 

2.)  When I first got the limebox, I my main share to high water.  What I didn't think about at the time is that it would be best to try to group my different jobs on each disk.  I would rather lose a full 3-4 jobs than lose part of 25 jobs if I lose a disk.  This may seem nitpicky since the lime box is just a backup, but we have had one hard drive failure that cost us thousands of man hours to recover.  Being able to group our jobs on a disk level would limit our exposure to losing two or three jobs vs parts of dozens of jobs.  Is there a way to re-configure this where UnRaid starts moving things in a share around during set periods to try to consolidate directories to disks?  I know this is an open ended question, but I'm just fishing for anything that might help.

 

3.)  Lastly, I was wondering if in my situation, a cache drive would increase the speed of my backups and more importantly, the speed of a parity check, (this usually takes 10-15 hours).  I looked around some to try to figure out how the cache drive works, but I didn't find much.  Maybe I didn't look hard enough.  Is the cache drive's primary purpose to speed up huge writes, or is it to cache stuff that has been used recently, like movies and music?

I nearly posted this in the other thread I have, but I figured since the topic was totally different, it would make more sense to start a new thread.

 

I currently have a MD-1500 purchased from Limetech.  I am using this as a backup for over 10,000,000 image files.  I am currently trying to optimize the system.  I've had trouble with it running out of RAM in writing errors to log files, and my nightly backups can do hundreds of thousands to millions of files, depending on how much work has been done in a day.

 

1.)  I got more ram yesterday, 3 1GB sticks, to try to bump my RAM up to 3GB.  When booting up, the system only recognizes 2GB.  I pulled one of the 1GB sticks out, and put the 2 512's back in, still only saw 2GB.  I moved the ram around, same result.  The puzzling part is that I know it worked with the 2 512MB sticks before.  Now it's working with at least one 1GB stick, and the 2 512's, or both 1GB sticks.  On bootup, it only sees 2GB.  This is the motherboard - http://reviews.cnet.com/motherboards/asus-p5b-vm-do/4507-3049_7-32310394.html?tag=mncol;psum - so I know it supports 4GB.  Any ideas on what I can do to get it to see 4GB?  Should I get another 1GB stick, and see if that works? 

I think that is it, you probably need to put pairs of like sized memory in the two banks.  Get another 1GB Stick... Try to get one with the same CAS latency and timing.  Preferably the same brand.

2.)  When I first got the limebox, I my main share to high water.  What I didn't think about at the time is that it would be best to try to group my different jobs on each disk.  I would rather lose a full 3-4 jobs than lose part of 25 jobs if I lose a disk.  This may seem nitpicky since the lime box is just a backup, but we have had one hard drive failure that cost us thousands of man hours to recover.  Being able to group our jobs on a disk level would limit our exposure to losing two or three jobs vs parts of dozens of jobs.  Is there a way to re-configure this where UnRaid starts moving things in a share around during set periods to try to consolidate directories to disks?  I know this is an open ended question, but I'm just fishing for anything that might help.

unRAID has no such feature to move data around once it has been stored. It never re-balances the files on its own across physical disks.

3.)  Lastly, I was wondering if in my situation, a cache drive would increase the speed of my backups and more importantly, the speed of a parity check, (this usually takes 10-15 hours). 

Ouch.. thousands of hours of recovery is not something you ever want to have to explain to the boss.

As far as parity... monthly parity checks are recommended (and only so all the blocks on all the disks get read so the SMART disk firmware can find and deal with bad sectors), other than that, the parity should take care of itself unless the system loses power or crashes.

I looked around some to try to figure out how the cache drive works, but I didn't find much.  Maybe I didn't look hard enough.  Is the cache drive's primary purpose to speed up huge writes, or is it to cache stuff that has been used recently, like movies and music?

Writing directly to a "disk share" without having a cache drive enabled involves two reads, and two writes for each block written.  4 I/O operations, this cannot be made more efficient.  unRAID must read the existing data block, the the corresponding parity block, compute the new parity based on an "exclusive or" of the new data being written and the old parity data, then write the newly calculated parity to the parity drive, and also the new data to the data drive.

 

Writing to a "cache drive" involves one write, directly to the cache drive. Later, typically in the middle of the night, data is read from the "cache drive" and written to the data drives, with those same 4 I/O operations... in fact, there are 5 I/O operations, as there is an additional one to read the data from the cache drive.

 

While data is on the cache drive it is not protected from a drive failure.  If the cache drive were to crash, your data is gone.  Once your data is moved in the middle of the night to the data drives from the cache drive, it is protected by the parity drive from any single drive failure.

 

The cache drive does nothing to make reading your files from the server faster.  It is there to allow programs that must store large streaming data in real-time to be able to function. (Some DVR and video capture applications on PCs on the LAN need to be able to save high-def transport streams, and writing to parity protected drives is not always fast enough to keep up with the data)

 

In your case, the cache drive would need to be big enough to store ALL the data written to the array between its scheduled moves of that data to the parity protected drives.  If it were to run out of space you would either get an out-of-space error, or, if the writing program opened the file at its full size, end up writing directly to the parity protected array if it knew there was not enough space on the cache drive initially.

 

Basically, the cache drive still writes the data as slowly as before to the parity protected drives, perhaps even a tad slower, but does it without having to worry about the need for speed... it has as long as it needs during the night.  The cache drive can just be written to faster, as it is not protected by parity.  I'm not sure a parity drive will work very well with millions of files on a given night... It might, but it will sure take its time. (but then, millions of anything take time)

 

The cache drive will speed up your backups, as long as it is big enough to hold all the data you are writing.  It will leave that same data unprotected by parity until it is subsequently moved by the "mover" script from the cache drive to the parity protected data drives.  Since the "mover" script is scheduled by "cron" you can schedule it to run more frequently than once a night in case you do some backups during the business day.

 

Joe L.

1.)  I got more ram yesterday, 3 1GB sticks...

 

I could be wrong here, but I believe you have to match the current gig of RAM, because most modern boards are designed to take advantage of dual channel access.  You need to install 2 pairs of matched RAM.  You probably need one more - exactly like the 3 you just bought.  Although generally, it is best to buy them already in matched pair bundles.  If you can return these, pick up a pair of 2gig sticks.  That leaves 2 slots free, you may want more RAM later.  Your motherboard manual should be helpful.

 

3.)  Lastly, I was wondering if in my situation, a cache drive would increase the speed of my backups and more importantly, the speed of a parity check, (this usually takes 10-15 hours).  I looked around some to try to figure out how the cache drive works, but I didn't find much.  Maybe I didn't look hard enough.  Is the cache drive's primary purpose to speed up huge writes, or is it to cache stuff that has been used recently, like movies and music?

 

The unRAID Cache drive is strictly a write cache, has no read caching capabilities.  unRAID doesn't really need any read caching, and Linux does an adequate caching job.  In case you haven't seen it, see the FAQ entries about the Cache drive.

 

[edit: apologies to JoeL, for some reason, the forum did not inform me of his post]

Ouch.. thousands of hours of recovery is not something you ever want to have to explain to the boss.

  Yea, well I'd been the one hollering for the backup and kept getting told we couldn't afford one.  After the crash, it was finally recognized that we couldn't not afford to have it.  If I'd been more specific in my clamoring, and insisted that we didn't need a 30K solution, maybe we would've had one before.  Anyway, you can't afford not to have a backup.

 

Thanks for the explanation on the cache drive.  If at some point in time, I can't run my backups during off hours because of the speed, I may get a cache drive and run the backups at night to the cache drive, then run the sync during the day in order to free up production machines.  As long as the backup runs during the set period of time, I see no advantage to increasing the complexity of the backup.

Ouch.. thousands of hours of recovery is not something you ever want to have to explain to the boss.

  Yea, well I'd been the one hollering for the backup and kept getting told we couldn't afford one.  After the crash, it was finally recognized that we couldn't not afford to have it.  If I'd been more specific in my clamoring, and insisted that we didn't need a 30K solution, maybe we would've had one before.  Anyway, you can't afford not to have a backup.

Back when I worked for AT&T I was on a software development project for the NYSE, yup, the Stock-exchange in NY City...  We were on such a tight schedule I could afford NO downtime because of a crashed disk, or accidentally deleted files.

 

I wrote a script to do hourly incremental backups of the development server.  Now, even with 30 or so developers and analysts, there were not too many files changed in an hour, but it was still critical to make sure we had copies of all work, just in case. 

 

It saved our butts at least twice during that development cycle when somebody needed an older version of what they were editing.  (yes, we were using also version control, but the incomplete files were not checked in daily)

 

 

Joe L.