I've been troubleshooting (successfully, that was a different thread) my unRAID box the last few days, and working more intimately with it than I usually do has brought some curiosities of mine to the forefront.  I suppose in the end it comes down to performance, but I've wondered about how the "tidiness" of your drives affects performance.

 

To back up a bit, I jumped into unRAID a while ago, I set it up, and off I went.  I'm lazy, so I setup everything to write to user shares.  In the beginning, while I read up on allocation methods and split levels, and tried to configure them logically, between lack of full comprehension and additional laziness I don't think I've done a very good job of it.

 

What I'm left with, I think, is an unRAID server that has a lot of content nearly randomly thrown all over it.  This leads me to wonder, despite Tom's awesome pooling algorithm, would there be any benefit to attempting to consolidate "like" or "near" data onto the same drives?

 

For example, all my digital pictures which is a somewhat substantial quantity, but nothing approaching 3-4TB drive capacity (yet ) are spread out across (it looks like) 5 of my 7 drives.  Would it be beneficial to consolidate those onto one disk? 

 

Off the top of my head it would seem to be good because it would only require spinning up one disk when I access them.  Would access times be improved at all, either accessing the files or just browsing the share?  Or does the unRAID driver+cache_dirs do that sufficiently?

 

Another thing I've been curious about, is there an "easy" way to tell what disk a given file is on?  Blue-sky thought is right click on the file in Windows Explorer and look at properties, but obviously that's not practical.  Is there some way better (faster) than running "find /mnt/disk* -name foo" from a command line?  It seems like it should be possible to query unraid with a user share path and get the "physical" path back.

 

Just some questions that have been bouncing around in my head for a while that I couldn't find answers to yet.

Same here (confused massively about split levels).  I spent a few MONTHS with test unraid configs and played with split levels until I found something I liked.  And posted noob questions here.  A lot.

 

I have a metric ton of content, and its spread all over the place, every drive in the array is a hodgepodge of whatever.

 

I did it because if I have a double disk failure (parity + member), at least I have some family pics on the rest of the drives, some music on the rest of the drives, etc.  I know "there is no substitute for a backup" but when you find a 12TB backup solution that doesn't require me to sell a kidney, let me know.  I am in the process of building a second unraid server to use as a backup (I know, the horror).  If my ISP didn't have a monthly quota I'd store the second server at a family member's house.  I've thought about differential backups (easier on bandwidth) but I don't have the time right now to read/hack with rsync.

 

Secondarily, I chose unraid because I wanted Drobo-like functionality without the Drobo.  Most of my files are a hugely unorganized mess (don't judge) and I just needed a massive depot where I could throw files and organize them as I have spare time (rare).  So having all data written across all disks works for me because I don't have to worry if my "movie" disk(s) is/are filling up, etc.  I just send data to a specified share and off it goes into binary oblivion and unraid handles where to put it.

 

I had directory caching enabled, which kept all my disks spun up all the time.  I disabled it, and now when I go to watch a movie, or open Windows explorer to look for a file, there is a slight delay.  I can live with that.

 

Don't know too much about your second question. 

I had directory caching enabled, which kept all my disks spun up all the time.  I disabled it, and now when I go to watch a movie, or open Windows explorer to look for a file, there is a slight delay.  I can live with that.

The whole idea of the cache_dirs script is to AVOID having to keep all disks spun up.  If running it keeps disks spun up then that suggests the directory information is NOT being cached - probably due to insufficient free RAM.  Do you have other add-ons running that are RAM hungry?

Don't want to hijack the OP's thread... but yeah.

 

8GB RAM, nothing else running.  Haven't seen a spun down disk in six months.  Turned off dir caching, all disks spin down now.

 

Cache_Dirs shouldn't keep disks spinning UNLESS you've got so much memory allocated for disk buffers and add-ons that the dentry cache is reduced to a very small size.    With reasonable disk tunables it should NOT maintain disks spinning.  I have over 200,000 files, 4GB of RAM, and disk tunables set at 2048/768/1024 ... and my disks are always spun down unless I'm actually access a file.

 

As for whether or not you want all files of a type together => that's purely an organizational choice on your part.    Clearly it's useful if, for example, all VOB's associated with a movie are on the same disk -- that way only one disk has to spin up to play that movie.    Similarly, you may want all pictures associated with a specific event or date (e.g. "Our Alaskan Cruise")  together, as you'll probably look at those at the same time.    But there's no real reason to have ALL movies, or ALL pictures, etc. on the same disk.  The whole concept of shares is that UnRAID will manage that for you ... and all you need to do is access the share to see everything associated with it.

 

Turned it back on, all disks are spinning now.

 

Set to defaults as far as I know, I don't know how to tune it or made any other changes.

 

What are your tunable disk parameters set at?  [settings - Disk]

 

... and what plugins/addons do you have running?

 

What are your tunable disk parameters set at?  [settings - Disk]

 

... and what plugins/addons do you have running?

 

 

Yes/15 mins/yes/no/mbr: 4k/1280/768/384

 

I'm running simple features.

Turned it back on, all disks are spinning now.

How many files are on your array, rounded to the nearest 100,000? 

 

I'm betting you just have WAY too many files to use cache-dirs effectively.

Turned it back on, all disks are spinning now.

How many files are on your array, rounded to the nearest 100,000? 

 

I'm betting you just have WAY too many files to use cache-dirs effectively.

 

 

Is there an easy way to find out?

Turned it back on, all disks are spinning now.

How many files are on your array, rounded to the nearest 100,000? 

 

I'm betting you just have WAY too many files to use cache-dirs effectively.

 

 

Is there an easy way to find out?

 

In Windows Explorer "point" to each of your shares;  then "Select All" of the content;  right-click, and do Properties.    This will show the number of files and folders.

 

Repeat for each of your shares, and add up the numbers 

Repeat for each of your shares, and add up the numbers 

Just for reference sake, I have about 400,000 in one share, and 550,000 in another. No cache-dirs for me.

Repeat for each of your shares, and add up the numbers 

Just for reference sake, I have about 400,000 in one share, and 550,000 in another. No cache-dirs for me.

 

Yes, a million files is probably a bit much for Cache_Dirs 

I presume you DO know you can easily limit which shares it caches ... so, for example, you could just cache the "Movies" share (or whatever you call it).

 

LOL, I was hoping there was a better way than right-click-properties on each of my shares (I have about a dozen).

 

I'm easily at half a million files, and probably well over.

 

How can I tell it which shares to cache?

I dont see why 1 million or even 10 million files should be a problem. Sure they are big numbers for humans but they are tiny for computers.

 

I accept there may be a problem now but there shouldnt be.

How can I tell it which shares to cache?

 

What do you WANT to cache?    For example, if you have a Movies share that spans a lot of disks, that's a good candidate to cache, since you don't want the system to have to spin up a lot of drives to determine which drive a specific movie that you want to play is on => if the directory is cached, it will KNOW which drive to spin up, and the others will remain spun down.

 

Just think about how often you access your various shares, and whether or not it's a hassle to wait for the drives to spin up when you access them.    If, for example, you have shares that are restricted to a single drive, that's probably not worth caching.

 

The issue is simple:  EVERY file you cache requires a certain amount of RAM to hold the path and directory name.  Not sure what that is in Linux, but it's probably ~ 128 bytes.  So if you cache a million files, the cache alone will occupy 128MB.    Not sure how much space UnRAID allocates for the Dentry cache, but if your cache requirements exceed that, then there will always be some files that aren't cached, so Cache_Dirs will continuously be updating it (thus drives will keep spinning).    I can confirm that with ~ 300,000 files there's no problem ... but not sure how high that number can go (and also not sure if 128 bytes is enough to keep the necessary info for each file ... I've very familiar with Windows directory structures, but not with Linux).

 

I have 8GB RAM so that's not an issue.

 

Under Settings -> Folder Caching, what should "Minimum Interval," "Maximum Interval," and "Maximum scan level" be set at?

 

I just turned off caching and, again, all my drives spun down...

I have 8GB RAM so that's not an issue.

 

Under Settings -> Folder Caching, what should "Minimum Interval," "Maximum Interval," and "Maximum scan level" be set at?

 

I just turned off caching and, again, all my drives spun down...

I would set the min to 1, max to 10, and max scan level to 5

Then include only the top level "share" directories used by your media player.

 

You are on your own figuring out how to do this through the simple-features plugin.  I have absolutely nothing to do with it, and have never even installed the plugin.

cache_dirs is a command line utility, and person who wrapped it with a simple-features plugin never coordinated their work with me.

 

On the command line you can run it in the foreground to see what it is doing with the "-F -v" options.

 

As far as enough memory... it has nothing at all to do with that.  It is the number of "directory entries" configured in the kernel, and you cannot change that.  They are in low memory.

 

Joe L.

As far as enough memory... it has nothing at all to do with that.  It is the number of "directory entries" configured in the kernel, and you cannot change that.  They are in low memory.

 

Joe, do you know how large the Dentry cache is as a function of installed RAM?  Also, in Linux, how many bytes/entry are required to store the names?  [in Windows it's 128-192 depending on the version, but not sure what Linux requires]    And is there a parameter that can be added to the Go script to adjust the Dentry cache size?

 

LOL, I was hoping there was a better way than right-click-properties on each of my shares (I have about a dozen).

 

I'm easily at half a million files, and probably well over.

 

How can I tell it which shares to cache?

 

You could try this on /mnt/disk*:

http://stackoverflow.com/questions/9157138/how-to-recursively-count-files-in-a-linux-directory