Cache drive question

[BLKMGK]

I've not ever used a cache drive in my systems as mostly I do reading with not so many writes. However I have a project I'm working on that is requiring HUGE amounts of write and th ewrite speed is getting in my way.... The deal is this - my DVD were ripped with DVD Shrink into a VOB/IFO format, turns out ISO format works better with my chosen media software - XBMC. So I am having to convert, using Shrink as it happens, all of my DVD to ISO format. Not a big deal except that I have almost 800DVD images and over 2/3rds are in the format I need to convert FROM.

 

Right now I am using Shrink to convert them to a local drive - I see 30-38megs a second transfer on READS. Writes are not bad IF I transfer one image at a time. But I'm transferring 16-20 at a time into different dirs - things slow to a CRAWL with huge gas where nothing transfers at all.

 

My question - Currently I rip from already created directories leaving behind metadata, delete the source once the rip is confirmed, and then copy back to that same dir using a top level share that spans all of my drives. IF I have a cache drive in the loop will the images goto that drive and THEN get copied to the intended directories when the Cron job kicks off? How will I know when that job has taken place? I guess what I want to know is does this cache drive act as a seamless go between or will my already having dirs on drives cause issues?

 

I can keep doing it the way I am but man the writes sure slow me down to the point that I can only do a batch or two a day :'( At this rate it will be a very long time before I finish!

 

P.S. I have considered trying to batch this with some manner of ISO creation tool but the moving of the metadata etc. and the risk of ending up with a bad ISO image or losing data due to a burp REALLY scares me!

[SSD]

The cache drive only delays the pain.  It allows you to copy data to the server onto an unprotected disk, and then overnight unRAID will move it to the array.  If this would help you, go for it.  But if you are copying chunks of images overnight anyway, the extra trip to and from the cache disk will actually slow you down.

 

 

[BLKMGK]

The "pain" in this case is the unresponsiveness of the Vista GUI. Getting drive listings to refresh and doing anything like oh say ripping the next disk causes things to all but lockup and I see network traffic come to a halt for seconds at a time as the array scrambles to handle my request. If I am writing and try to rip a DVD I see speeds go from 30+megs to a fraction of that. Having the copies occur internal on the array while I'm sleeping without me having to spend a ton of time dealing with the GUI would be fine by me!

 

So, would the cache drive still copy the data to my ultimate intended location and would I have to do anything special to employ the cache drive other than trying to do a copy to a location on the array? Does it somehow get in the middle of the copy on it's own and know where I was going to put the data? Something seamless is what I need, this is hard enough as it is. Figuring out a relaible way to script this would rock but I have too much data to easily backup so I fear a Journalspace incident if I am not careful.

 

Heh, you just know those idiots are going to become a Verb!

[SSD]

The "pain" in this case is the unresponsiveness of the Vista GUI. Getting drive listings to refresh and doing anything like oh say ripping the next disk causes things to all but lockup and I see network traffic come to a halt for seconds at a time as the array scrambles to handle my request. If I am writing and try to rip a DVD I see speeds go from 30+megs to a fraction of that. Having the copies occur internal on the array while I'm sleeping without me having to spend a ton of time dealing with the GUI would be fine by me!

 

So, would the cache drive still copy the data to my ultimate intended location and would I have to do anything special to employ the cache drive other than trying to do a copy to a location on the array? Does it somehow get in the middle of the copy on it's own and know where I was going to put the data? Something seamless is what I need, this is hard enough as it is. Figuring out a relaible way to script this would rock but I have too much data to easily backup so I fear a Journalspace incident if I am not careful.

 

Heh, you just know those idiots are going to become a Verb!

 

The cache drive was designed to work in concert with user shares.  You would copy new data to the user share (or directly to the cache drive into the user share directory) and at the appointed time (middle of the night) the mover script would move it from the cache disk to the user share using the user share allocation method.  I do not have a cache disk per se, I have a mounted drive that acts like a cache disk but without the mover script.  I copy things to it, and then before bed issue some "mv" statements to move the files to specific disks / directories.  Works for me, but YMMV.

 

Under heavy load I do experience reduced performance with unRAID too, but not nearly as bad as you are reporting (but I am using XP not Vista).  I recently posed a link in the "Best of the Forums" (see link in my sig) related to Vista network performance issues.  If you haven't seen it, might want to take a look.

 

BTW, the mover script is literally that.  You can overwrite it with your own (on each boot) or even just replace with a null script and do your moves manually.

 

Good luck!

[BLKMGK]

Well, this data is destined to a User Share but it's a directory already setup. From the sounds of it this might be worth a shot as I'm moving the files to a user level share with dirs under it.

 

Something like this \\unraid\DVD\moviename\isogoeshere.iso and in that directory will be folder thumbnails, XML descriptors, and fan art for XBMC. Sounding like it's worth a shot since I have at least one IDE drive sitting on the shelf that's plent big enough. Maybe I'll try that tonight and see how it goes.

 

As for performance - it's the unRAId. Network data halts when things are being processed and then spikes, then halts, then spikes. This is all during writes, during single thread reads it FLIES and degrades pretty nicely when I do multiple reads. <shrug> I do see in the log that one of my drives is apparently setup for just SATA 1.5 instead of 3 but I don't think that's an issue - I'll look for a forgotten jumper when the system is down tonight I guess...

[jimwhite]

Don't forget that writing multiple images to the same folder simultaneously is a recipe for massive file fragmentation...

[RobJ]

That is another advantage of using a Cache drive.  You can simultaneously write multiple files to the User Share, and using the Cache drive they will write much faster, but possibly fragmented.  Then later, the mover script will move them to their final resting place one at a time, essentially defragged.

[erikatcuse]

Since you have two sets of the same data you could just take the parity drive offline.  Do your rips/transfers and then rebuild parity.  This would allow you to have full speed on the server and if you need to access other files you would be able to do so.  The only drawback is you would be unprotected for the amount of time it took you to rip/transfer those files.

 

another option if you have the space is mount an empty drive and copy the files to that without protection and then add it to the array.

[BLKMGK]

Yup, possibly fragmenting but I do not need high speed access to this data - it's SD DVDs heh. Not sure I'd be real comfortable taking Parity offline for the duration since this is a going to take me awhile. So far though it sounds like the Parity drive is the most straightforward way to go with this and pretty much what it was intended for. So long as I have an open IDE connector on that box I think I will go for it and see how it works out, I have spare drives just sitting that are as big as 500gigs <shrug>

[SSD]

BLKMGK - Have you found a command-line tool or other batch way to build .ISOs for a series of DVD subdirectory structures?  I am thinking of doing the same and would benefit from your experience.

[flambot]

BLKMGK - Have you found a command-line tool or other batch way to build .ISOs for a series of DVD subdirectory structures?  I am thinking of doing the same and would benefit from your experience.

 

You can set-up multiple sessions of DVDshrink and activate then sequentially with a small program called Multishrink (1.4).  Works like a charm.

[BLKMGK]

Had not heard of multishrink - that could be QUITE the lifesaver here, THANK YOU!

 

Sadly it looks like my IDE is already full so bummer I am going to have to buy soem more SATA drives Think I will snag soem 1.5s and swap out the IDE - use one of those for cache. Perfect!

[flambot]

Had not heard of multishrink - that could be QUITE the lifesaver here, THANK YOU!

 

It's particularly good for ripping episodic TV DVD's into single file episodes for use on XBMC. Once all your multi-sessions are set-up - click and forget.

[SSD]

Thanks flambot!

[SSD]

I did some searching and read about a linux tool called "mkisofs".

 

It can create an ISO from a subdirectory structure directly in Linux.

 

Here is the download ... LINK

 

Instructions (from HERE)

 

Example:

mkisofs -o /home/jdoe/myiso.iso /home/jdoe/myMovie

 

This will make an iso image of the files in /home/jdoe/myMovie and output myiso.iso in /home/jdoe

 

I downloaded it and tried it out.  Works geat!

[BLKMGK]

I might try the ISO creator, honestly though messing with this data makes me VERY nervous <sigh> MultiShrink is actually quite the help!

 

 

Now for the issue still at hand - cache drive. It turns out one of my existing drives had no real videos on it but was instead a backup of XBOX games. Kewl, I copied them all off to my other server which is dedicated to things like that and turned that drive into my new spiffy CACHE drive! Yay!

 

So I have a handful of ISO I am dying to copy thanks to MultiShrink, I navigate through a TOP LEVEL SHARE (not a disk share) find the target directory, and begin a copy of ONE ISO image. gee, NO FASTER It's doing the exact thing it did previously - slow write speed in bursts. I am averaging 8meg a second - about normal considering the machine is doing a parity check from the drive removal.

 

I navigate to the DISK SHARE and sure enough my file is being created there. Thinking maybe I am being spoofed I telnet into the box and navigate to the cache mount. Clean as a whistle - nothing being created there.

 

I checked the Wiki and I see a link to -> http://lime-technology.com/forum/index.php?topic=1731.0 and I find out that I need to enable the cache drive for each of the shares I want it to function on - DOH! NOW I see directories and files being created on the cache drive <smack> Not sure I am seeing quite the speed boost I had hoped but it is certainly WAY more smooth and multiple copies seem to be continuing without pausing and starting. I have 4 multi-gig copies going right now and each is pushing about 3.5-4megs a sec. My HOPE is that when Parity is completed that the computer will have more resources available for the cache drive and speeds will rise. I really need to move this data faster if possible. If nothing this is more consistent and keeps chugging with many more concurrent copies, that's an improvement I needed as having everything slow to a crawl was awful and I was starting to look for ways to copy files in a serial fashion one after another <sigh>

 

So anyway, maybe not the dramatic improvement I had hoped but good enough to help me keep my data moving faster while I do this big conversion. Short of running an ISO converter ON the unRAID box I am not sure how I could go any quicker...

[BLKMGK]

Okay, mover script doesn't appear to have kicked off. Mover "schedule" is listed as "40 3 * * *" umm what does that mean in English? Does it need to be modified? Default was supposed to be 3am thereabouts but that sure doesn't look like 3am to me. Does having a parity check running interfere with the mover?

[Joe L.]

Okay, mover script doesn't appear to have kicked off. Mover "schedule" is listed as "40 3 * * *" umm what does that mean in English? Does it need to be modified? Default was supposed to be 3am thereabouts but that sure doesn't look like 3am to me. Does having a parity check running interfere with the mover?

First field = minutes past the hour, second = hour, in 24 hour time, remaining fields match all months, day of month, and day of week.. 

 

Scheduled time = 3:40 AM, every day, every month.

 

The mover script does not care if parity is running or not.  It will not move a file if it is in the process of being written.

 

Joe L.

[SSD]

Short of running an ISO converter ON the unRAID box I am not sure how I could go any quicker...

 

Is running the ISO convertion on the unRAID box a bad thing?  I posted instructions for doing exactly that, and ran a successful test.  See my prior post.

[BLKMGK]

Short of running an ISO converter ON the unRAID box I am not sure how I could go any quicker...

 

Is running the ISO convertion on the unRAID box a bad thing?  I posted instructions for doing exactly that, and ran a successful test.  See my prior post.

 

Bad thing? No but I am nervous about it. For one thing I do not have space on the drive to double up the numbers of DVD so I would have to convert and delete as I went. If I screw up then I am going to have a mess on my hands. I AM tempted to try it and that might ultimately be the best way to go but reripping all of these DVDs would be a nightmare. Also, each DVD has meta data associated with it - this data is buried in the Video_ts folder. I have to copy that up one folder and rename it using the name of the parent folder as input. Doable with a script yes but I have not attempted it and that complicates things for me raising my level of discomfort. The worst would be to do the conversion and then find out that soemthing got hosed while watching a movie - I'd scream

 

That said, the cache drive just sucked down a file copy at over 50megs a second! This was a process that would have taken FAR longer before so this is indeed a big improvement! I did notice one odd thing, the parent folder isn't deleted by the script after the files get moved - I kicked it off manually earlier today. <shrug>

[hawihoney]

BLKMGK - Have you found a command-line tool or other batch way to build .ISOs for a series of DVD subdirectory structures?  I am thinking of doing the same and would benefit from your experience.

 

Sorry for butting in late. I used ImgBurn to create ISOs automatically. It's not fully autmated but if you are still searching I can help you to make it better. I had to transform the following struct where every film had it's own directory below V:\S (all films starting with S):

 

...
V:\S\Seven\VIDEO_TS\...
V:\S\Stargate\VIDEO_TS\...
...

 

 

Put the resulting ISOs into \\Tower\disk1\Filme\S\. This script is called MakeFilmeISO.bat and it was located in X:\:

 

REM *** For example
REM *** v:
REM *** cd s
REM *** dir /B /O:N /A:D > X:\Batch.bat
REM *** put call MakeFilmeISO.bat "foldername" around the directory names
REM *** x:
REM *** call batch.bat

@ECHO OFF

SETLOCAL

SET MI_SRC="V:\S"
SET MI_DEST="\\Tower\disk1\Filme\S\"
SET MI_FILE="%1"
SET PathToImgBurn="C:\Programme\ImgBurn"

IF EXIST "%MI_DEST%\%MI_FILE%.iso" EXIT /B

REM *** for BLURAYs "%PathToImgBurn%\ImgBurn.exe" /MODE BUILD /BUILDMODE IMAGEFILE /SRC "%MI_SRC%\%MI_FILE%\" /DEST "%MI_DEST%\%MI_FILE%.iso" /FILESYSTEM "UDF" /UDFREVISION "2.50" /VOLUMELABEL "%MI_FILE%" /CLOSESUCCESS /NOIMAGEDETAILS /ROOTFOLDER "YES" /START

"%PathToImgBurn%\ImgBurn.exe" /MODE BUILD /BUILDMODE IMAGEFILE /SRC "%MI_SRC%\%MI_FILE%\" /DEST "%MI_DEST%\%MI_FILE%.iso" /FILESYSTEM "ISO9660 + UDF" /UDFREVISION "1.02" /VOLUMELABEL "%MI_FILE%" /CLOSESUCCESS /NOIMAGEDETAILS /ROOTFOLDER "YES" /START

COPY "%MI_SRC%\%MI_FILE%\folder.jpg" "%MI_DEST%\%MI_FILE%.jpg"

DEL "%MI_DEST%\%MI_FILE%.mds"

ENDLOCAL

 

Regards

Harald

 

 

[SSD]

I wanted to make sure you had seen my prior note about converting to .ISOs on the unRAID server.  I think I will likely go that way.

 

You can write the .ISOs to your cache disk just the same as if you had done it from Windows - it just means you don't have to keep your Windows box online, they don't have to go across your network, and you can set up a batch of them to process.

 

I am with you on being careful. It would be easy to mess up something.  Doing it with near zero empty space would be scary. 

 

I'll likely do them in batches, and test well before deleting the source files.

 

I do not use the mover script, but believe it is supposed to delete the source folder when it is done.  I'd be careful about deleting something it was supposed to have deleted.  I HAVE used the "mv" command to move files, and it works on a directory basis and not a file basis.  So if you were moving a movie directory structure to another drive, it would copy each file in the folder and only then delete the source files and folder structure.  This is different from how Windows does it - but better IMO.

 

Good luck!

[BLKMGK]

Appreciate the suggestions guys. Yeah so far the mover script has worked fine manually but NOT deleted any directories. I am out of town right now so when I get back it will have had a chance to run a few times and I can see WTF it has been up to

 

I am going to checkout that script some and see if I can bend it to my needs. In each of my video_ts directories I have a TBN file, an NFO file, and sometimes a JPEG that is art. I have to move them up one dir and rename then based on the parent directory name - I also have to go down a BIG list of directories, soe of which won't even have the video_ts sub so while I know this can be scripted I am fearful of trying (lol). I am slugging along manually using the Multishrink and cache drive, so far not too bad! Am up to the Hs hehe.

 

One thing pointed out by a friend tonight though... he too has a program creating XML descriptor files and he used top level shares like I do. He has found that when he digs into the disk shares and look at his movie folders that the XML isn't always there. Duh, unRAID using a top level share will put it where space best permits which is NOT always with the directory you are trying to tag. Whoops!! Not a bad thing I guess but if you are moving stuff at disk level for whatever reason you might not get all of the files you expect. I do not YET know if this is also effecting me but it seems likely since I too use top level shares and a 3rd party tool <sigh> Will look - when I am done converting!

[optiman]

My cache Mover didn't kickoff, meaning it didn't move anything.  What can I check to investigate?

 

Also, I can manually hit the Move Now button and it works fine, but I noticed that it leaves the directories on the cache drive.  Shouldn't that move Everything off?

 

Thanks

[Biggy2872]

My cache Mover didn't kickoff, meaning it didn't move anything.  What can I check to investigate?

 

Also, I can manually hit the Move Now button and it works fine, but I noticed that it leaves the directories on the cache drive.  Shouldn't that move Everything off?

 

Thanks

 

A quick serach of the forum shows that the mover does infact leave the folder sturcture on the cache drive (maybe by design?).  They are just empty folders and are not taking up any space.  If you don't want them on the cache drive, just delete them every once in a while.

 

Cheers,

Matt