To Cache drive or not to Cache drive?

[defected07]

I'm using 0 split level, and created a FLAC directory at the top of my cache drive. In my share, my FLAC directory is 3 levels deep. It copied my FLAC directory to one of my drives at the top, but it was also in the 3 deep level directory. How exactly does the mover script, and the 0 split level work? i thought it allows for a top level directory among many drives and then second level directories among any number of those drives so you can evenly distribute your categories. Sorry. I read then Unofficial doc but it's still not 100% clear

 

Can someone help with my questions, please?

 

I'd have to create the same structure each time, if my cache drive were to mirror my top-level user shares?  Or am I not setting up the cache drive correctly? 

 

I'm just trying to figure out why created my FLAC directory as a top level on a seemingly random drive (not quite random, as it's an included disk (/mnt/disk4) in my Media user share, which FLAC is a part of--only a few levels beneath Media.) 

 

It also appears to have placed the FLAC where my FLAC actually resides (/mnt/disk1/Media/Music/FLAC).  So does the mover script just search for the folder name that's on the Cache drive, within the other Disk/User shares, and place contents there?  Because why would it also place the FLAC folder on disk4, as a top-level folder?

[lionelhutz]

I'm using 0 split level, and created a FLAC directory at the top of my cache drive. In my share, my FLAC directory is 3 levels deep. It copied my FLAC directory to one of my drives at the top, but it was also in the 3 deep level directory. How exactly does the mover script, and the 0 split level work? i thought it allows for a top level directory among many drives and then second level directories among any number of those drives so you can evenly distribute your categories. Sorry. I read then Unofficial doc but it's still not 100% clear

 

Can someone help with my questions, please?

 

I'd have to create the same structure each time, if my cache drive were to mirror my top-level user shares?  Or am I not setting up the cache drive correctly? 

 

I'm just trying to figure out why created my FLAC directory as a top level on a seemingly random drive (not quite random, as it's an included disk (/mnt/disk4) in my Media user share, which FLAC is a part of--only a few levels beneath Media.) 

 

It also appears to have placed the FLAC where my FLAC actually resides (/mnt/disk1/Media/Music/FLAC).  So does the mover script just search for the folder name that's on the Cache drive, within the other Disk/User shares, and place contents there?  Because why would it also place the FLAC folder on disk4, as a top-level folder?

 

If you actually gave an example with some clear directory structrues illustrated then you might get an answer. Right now, I have no idea what really happened so I can't give an answer.

 

Split Level 0 - if Media/Music/FLAC exists only on disk1 then any data written to Media/Music/FLAC must go to disk1.

 

It possibly maybe sounds like you might have created a top directory on the cache disk called FLAC and then the mover would just moved that diretory to one of your disks. The cache disk MUST mirror the data disk folder or directory structure. If Media/Music/FLAC is on the data disk then you have to create Media/Music/FLAC on the cache. However, if you go to your network shares and write files to //Tower/Media/Music/FLAC then the directories and files are created automatically on the cache disk.

 

Peter

 

[defected07]

So if I enable Cache disk for my Media share, it'll populate the Cache disk with the files FIRST, then the mover script will run and put them on my data disk?

 

Sorry for the convoluted example above.

 

What happened was I moved a bunch of FLAC albums to /mnt/cache/FLAC.

 

When the mover script ran, it seems it moved the albums to both /mnt/disk1/Media/Music/FLAC (which is where it should go), as well as /mnt/disk4/FLAC (that FLAC directory did not exist prior to this move).

 

This was also before I enabled the Cache disk for the user share Media.

 

So I would have to create a folder-structure mirror of the User shares whenever I move files to Cache disk, correct? Because if I create empty folders on the Cache disk, it would just move those empty folders to the User share (and thus, delete the file structure on my Cache disk)

 

I suppose I could create another script that scans /mnt/user for folders, and creates that folder structure on Cache after the mover script, runs.

 

Just trying to clear up some confusion and understand how the Cache drive and mover script works.  Thanks.

[lionelhutz]

You should just move the files to /mnt/Media/Music/FLAC and the cache drive would then just be used to temporarily store the files. I'm not sure why are finding it necessary to be accessing the cache disk directly and manually creating the directory structure on the cache disk.

 

I'm not sure what you did but I highly doubt that files stored in /mnt/cache/FLAC were moved to /mnt/disk1/Media/Music/FLAC. They would simply be moved to /mnt/diskX/FLAC. This would also create a new user share, and would likely be put on the first disk called out by the high-water allocation method. The mover does not "seek out" the FLAC folder to put the files there. It just picks the disk following the share rules and then does a direct root-level to root-level move.

 

Peter

 

[defected07]

OK, thanks Peter.  Not sure why it happened, but looks like all the FLAC files were placed there--just going to verify everything is one to one.

 

I wasn't aware the Cache drive was used "implicitly"--I thought I would have to copy over to it, but now it makes sense to copy to the User share, and it'll store on the Cache.

 

So the /mnt/user0/ directory is for the Cache drive, right?

[lionelhutz]

Yes, the share location is mnt/usr/Media, not /mnt/Media. I believe the usr0 is the shares without the cache drive included.

 

Peter

[voldak]

I have not been happier since I added my cache drive. My transfer speeds went from about 30mb/s to this picture below..

 

http://i.imgur.com/IhSco.png

[defected07]

Wow, I don't know what I'm doing wrong.  I need to optimize my network or drives somehow.  I have a 7200RPM drive, but def. don't see speeds like that.

 

Any suggestions?

[voldak]

I picked up a Kingston SSDnow v100 64gb ssd recently and threw it in my laptop. Before I was running a 320gb 7200 rpm 2.5" drive (my speeds were about 60-70mbps to cache)

 

I'm running a caviar blue 7200 for my cache drive.  Also, I was practically doing nothing else on my network.

[defected07]

Yeah, I don't always factor the read speed of local drives...

 

I think I've been sending from my LaCie Big Drive (?) 1TB, which I believe are 2 x 500GB 7200RPM drives (Samsungs, I think) through FW800 interface.

 

Speeds never seem to get over 45 MB/s, over Gigabit ethernet.

 

But I'm plugged into an Airport Extreme, which is extending the Wireless network via Ethernet of our AT&T router that's in the basement.

 

That cable running from the basement is 100ft of Cat5.  So I wonder if it goes all the way back to the AT&T router instead of using the Airport Extreme as a switch.  I'll run a 50 foot Cat6 this weekend, since that's all the length I need, and see if that improves matters.

[voldak]

I'm also using an airport extreme for my few gigabit ports (unraid, laptop) and my other port goes to an additional 10/100mbs switch.

 

It shouldn't go all the way back to your router unless your server is on a different network or subnet. Also, it's doubtful that a 100 foot vs 50 foot would make any difference unless one cable was bad. Good luck with your testing. Definitely post if you see any improvements.

 

Edit: I was reading up a little bit on your setup and this is probably your issue...

 

"FireWire 800 offers impressive results, with speeds up to

100MB/s, though current drive technology limits this to 55MB/s (maximum sustained

throughput) for a single drive, and up to 100MB/s (maximum sustained throughput) per

bus in RAID 0 configurations. "

 

I read this from this link - http://www.lacie.com/download/more/WhitePaper_FireWire_800.pdf

[defected07]

Ahh, interesting.  Well it is a Big Disk 1TB Lacie, using RAID 0 (striping).

 

Really hoping to get rid of it as it spins down very frequently, and whenever I open Finder, or save a file, it has to spin up the disks first--temporarily freezing Finder and Save As dialogs--annoying!

[Rajahal]

I have not been happier since I added my cache drive. My transfer speeds went from about 30mb/s to this picture below..

 

http://i.imgur.com/IhSco.png

 

Kudos, that's the fastest I've ever seen!

[aaronwt]

I'm also using an airport extreme for my few gigabit ports (unraid, laptop) and my other port goes to an additional 10/100mbs switch.

 

It shouldn't go all the way back to your router unless your server is on a different network or subnet. Also, it's doubtful that a 100 foot vs 50 foot would make any difference unless one cable was bad. Good luck with your testing. Definitely post if you see any improvements.

 

Edit: I was reading up a little bit on your setup and this is probably your issue...

 

"FireWire 800 offers impressive results, with speeds up to

100MB/s, though current drive technology limits this to 55MB/s (maximum sustained

throughput) for a single drive, and up to 100MB/s (maximum sustained throughput) per

bus in RAID 0 configurations. "

 

I read this from this link - http://www.lacie.com/download/more/WhitePaper_FireWire_800.pdf

 

Was that written in the 20th century? Even back in 2001 when I started recording HD shows to my IDE hard drives(using HiPix tuner cards), I got sustained throughput greater than 55MB/s(440mb/s) when I transferred the recordings from my HTPC to my storage server over my gigabit network.

[mrklaw]

can you partition a drive and use part of it as cache?

 

Eg for running apps like sabnzbd it will want working space for temp downloads, NZB archives etc. I don't want these moved to the data array while in the middle of a download. When a download is finished, sabnzbd or sickbeard can move the final version to a desired location - I'm happy with that being direct to the data array.

 

So can you simply have a 'working' drive that doesn't get moved?

[WeeboTech]

can you partition a drive and use part of it as cache?

 

Eg for running apps like sabnzbd it will want working space for temp downloads, NZB archives etc. I don't want these moved to the data array while in the middle of a download. When a download is finished, sabnzbd or sickbeard can move the final version to a desired location - I'm happy with that being direct to the data array.

 

So can you simply have a 'working' drive that doesn't get moved?

 

You can partition a drive and use partition 1 for cache and another partition for a work area. (or even swap).

Parition 1 must always be the cache drive.

 

You can also use a hidden folder on the cache drive for files you want to keep there.

I.E. a folder with a . in front of it.

These hidden files/folders do not get moved when the mover runs.

 

 

[prostuff1]

can you partition a drive and use part of it as cache?

 

Eg for running apps like sabnzbd it will want working space for temp downloads, NZB archives etc. I don't want these moved to the data array while in the middle of a download. When a download is finished, sabnzbd or sickbeard can move the final version to a desired location - I'm happy with that being direct to the data array.

 

So can you simply have a 'working' drive that doesn't get moved?

 

Yes, this can be done, thought he current beta wipes the partitioning out by accident.

[Mailman74]

I seem to have problems recording multiple HD shows to my unraid server using Sagetv. The recordings will have stuttering, pixelationa and some audio issues. Wondering if adding a cache drive to my server would help this problem.

[BRiT]

It may help.

 

Though to be honest you may have other issues. Typical HD streams are slightly under 20mbit/sec. Typical local writes to recent unRAID systems using Green drives are 200-280mbit/sec (25-35 MB/sec). The system and drives themselves should easily handle multiple HD streams being recorded at the same time.

[mrklaw]

stupid question, but do I format in the unRAID webserver gui or directly through telnet? I've partitioned the 500GB drive into 2x250 (btw, should they both be primary partitions?), so the first half can be used for cache and the second half as working space for apps.

 

and should I preclear for cache or don't I need to bother?

[theone]

I would like to use the cache drive as a data drive for applications like transmission and such...

 

How can I set it up and use it so:

1. The applications' data doesn't get copied to the array data disks?

2. Data copied to the array shares is copied straight to the data disks and not the cache drive (I will also disable the mover script).

 

I am currently doing it via S.N.A.P but would prefer not to run it if it can be done using the cache drive slot (i have an unRAID PRO license).

 

 

[lionelhutz]

How can I set it up and use it so:

1. The applications' data doesn't get copied to the array data disks?

2. Data copied to the array shares is copied straight to the data disks and not the cache drive (I will also disable the mover script).

 

 

1. Reply 65, 5 posts before yours.

 

2. Just set the shares to not use the cache disk, which I know has also been discussed in this thread.

 

 

[mrklaw]

quick question - just setting up now and I've put a cache drive on for future use of apps etc, but left parity off while I populate with files for the first time.

 

Just realised the cache won't be big enough to hold everything - transferring 2TB across and my cache is 250GB. Thats plenty for day to day usage, but not for first time use.

 

But I'm just accessing the user share, because I want unRAID to distribute the files across my disks evenly. Can I go into user shares and turn the cache off for the TV and Movies shares, then copy across (which will go direct to the disks, following the water mark rules), and then turn the cache support on for those shares afterwards? No ill effects?

 

Then once is all up and running I'll add the parity drive.

[Rajahal]

But I'm just accessing the user share, because I want unRAID to distribute the files across my disks evenly. Can I go into user shares and turn the cache off for the TV and Movies shares, then copy across (which will go direct to the disks, following the water mark rules), and then turn the cache support on for those shares afterwards? No ill effects?

 

That will work fine, no ill effects.

[tunetyme]

Let me see if I understand this correctly. 

 

If I make a modification to the tag on an audio file it will store the file on the existing disk and it will not be copied to the cache drive.  Is this correct?

 

Tunetyme