Confusing problems with disk caching

[Rizlaw]

I have unRAID 6.0.0 installed. I have created 3 shares (Bluray, DVD, TV). Initially, I created "Bluray" to include disks 1, 2, 3 and 4. I then changed my mind and excluded disk 1 and also deleted the "Bluray" folder created by the share on disk1. My /config/share/bluray.cfg file shows that disk 1 is excluded from the share

 

# Generated settings:
shareComment="All Bluray Movies and Media Content"
shareInclude="disk2,disk3,disk4"
shareExclude="disk1"
shareUseCache="yes"
shareCOW="auto"
shareAllocator="fillup"
shareSplitLevel="0"
shareFloor="15000000"
shareExport="e"
shareSecurity="public"
shareReadList=""
shareWriteList=""
shareExportNFS="-"
shareExportNFSFsid="0"
shareSecurityNFS="public"
shareHostListNFS=""
shareExportAFP="-"
shareSecurityAFP="public"
shareReadListAFP=""
shareWriteListAFP=""
shareVolsizelimitAFP=""
shareVoldbpathAFP=""

 

The "Bluray" share is set to "Fill-up" and split level is set to  "Manual:do not automatically split levels". It also has disk caching enabled.

 

The other two shares (DVD and TV) are set to use disk 1 only and exclude disks 2,3 and 4.

 

From my Linux workstation I connected to the "Cache" share and copied 500GB of files. When the copy completed, I saw all of the files on the Cache drive. I then manually ran the "Mover" script via the GUI button on the "Main" tab.

 

As I understand things (and apparently I don't), the "mover" script should be copying these files to disk 2's "Bluray" share which has 750GB free space left. It should copy to the first "Bluray" share disk that has sufficient free space. It's not doing that! It's copying all the files to disk 1 which is excluded from the "Bluray" share.

 

Question 1: What am I doing wrong?

 

Question 2: (proper use of cached shares)

 

Based on this thread: http://lime-technology.com/forum/index.php?topic=5754.msg240860#msg240860

 

If sharing and caching are setup correctly: I gather that you are supposed to copy files from your workstation to the unRAID server share location you ultimately want the files copied to (in my case the "Bluray" share. If the share is cached, this actually copies the files to the cache drive and NOT to the array containing the "Bluray" share. Thus, files sit on the cache drive until they are either manually moved via the "mover" button in the GUI or automatically moved at 3:40am to the correct share (again, in my case "Bluray").

 

This doesn't happen for me.

 

Firstly, if I copy a file from my workstation to my "Bluray" share and then look in the "Cache" folder, the file is not displayed. Nor, is the file displayed on any of the three disk drives which have a "Bluray" share (disks 2,3 and 4). Why can't I see the file anywhere. Where has it gone?

 

If, on the other hand, I copy the file or files directly from the workstation to the "Cache" share (presumably the wrong way to do it), I can see the file(s) in the "Cache" drive's directory. The problem doing it this way, as I described at the beginning of this post, is that the files do not ultimately get "moved" to where I expect them to be moved even though I only have one share (Bluray) cached.

 

So, I am totally confused how this is all supposed to work in version 6.  I need a better, clearer explanation  how to set this up correctly (assuming I haven't already done so) and how to mechanically make the copy from computer A to computer B (unRAID 6).

 

Thanks.

 

 

[limetech]

For your config it's not necessary to specify both Included and Excluded (though what you did will work).  For instance, if you expect to add more disks for that BLURAY share, then it's sufficient to set 'Excluded' to disk1 for that share.  And set Included to disk1 for your DVD and TV shares.  Although... with split level to "Manual" that kinda goes out the window, because in this case when unRaid decides to write to the array it will only consider devices where the objects parent directory already exists.

 

So let's say you use 'Manual' for ALL your shares.  Then you go and manually create DVD and TV folders on disk1 only.  Since Manual says "do not automatically split' all objects written to DVD or TV share will only go to disk1.

 

Similarly you would create BLURAY directory on disk2, disk3, and disk4.  When an object needs to be created on the BLURAY share, unraid will select one of the disks where BLURAY already exists (disk2, disk3, or disk4 in this case).  How does it select the disk?  According to allocation method (highwater, mostfree, fillup).  Let's say you create a movie directory on BLURAY.  Ok unRaid will pick which of those drives based on allocation method.  But when you start creating objects in that movie directory, since it only exists on a single disk, all files for that directory will go to that one disk.  This is nice and it's the way I setup my own server.

 

What happens when you add disk5, will BLURAY start spilling into that disk?  Not until you manually create the BLURAY directory on that disk.  (And btw, if you created DVD on that disk, now you DVD share would have two disks to choose from).

 

Anyway that is how 'Manual' split level works, hope that make sense 

 

As for your cache issue... It sounds like you moved movie directories directly to the 'cache' share.  That's not right, you want to move them to the 'BLURAY' share - and since you have set 'Use cache' to Yes for that share, when you write the BLURAY share, the files will live on the cache disk until moved off by the mover.

[lionelhutz]

Could you post screen shots of the shares page, the share settings for the BLURAY share and the global share settings?

 

It'd be nice if you could go to \\TOWER\disk1, \\TOWER\disk2 etc and post screen shots of the top level file and directory content as well.

 

Can you also tell us what the directory structure is under the BLURAY share?

 

The files should just go to the cache disk when you write to the user share IF you have the settings correct. Also, the mover will copy the files from the cache to the array onto the appropriate disk if you have the settings correct.

 

You can put a file directly onto the cache disk IF you create a top level directory name that matches a user share. For example, I could write directly to the cache by creating a "Movies" directory and storing a movie in that directory since I have a share called Movies. But, I would be doing it wrong if I just put the movie file on the cache drive. There is no real need to do this though.

 

On a final note - have you tried a reboot since creating the cache drive settings?

[Rizlaw]

Limetech (Tom) thanks for the clarifications on sharing.    I really hope you can find the time to make a Linux version of the "make_bootable" file so that the two of us who have asked for it, don't always have to do it from a Windows machine.

 

lionelhutz Thank you for your help. I'm marking this post solved for now. I shut down my unRAID 6 server last night. I turned it on this AM and found all the unexpected shares had disappeared. I think I now understand what I (and I think unRAID 6.0, too) did that created the problem in the first place.

 

1. As I previously mentioned (I think), when I originally created the BLURAY share it was for ALL four of my new 4TB data disks. At that point I hadn't written any files to the share, but I did notice that unRAID had only created a "Bluray" folder on disk 1 and disk 2. It did not create a similar "Bluray" folder on disks 3 or disk 4. I don't know why and I didn't think much about it at the time.

 

2. I also created a two disk cache and enabled "caching" for the BLURAY share. I did not create a "Bluray" folder on the cache drive (two WD Greens) as limetech suggested I could.

 

3. I then changed my mind about the BLURAY share being on all four data disks; I excluded disk 1 and also manually deleted the "Bluray" folder created by unRAID 6 on disk 1. Again, nothing had been written to the share at this point.

 

4. I then began my first batch transfer of files from my workstation to the "Bluray" share. For reasons I still don't understand (and which haven't been explained to me by anyone else) they found there way to data disk 1 (which was excluded from the share). As far as I understand, removing disk 1 from the share and deleting the "Bluray" folder should prevent any files copied to the "Bluray" or "Cache" shares from winding up on disk 1.

 

5. Because of what happened in "4" above, I transferred the rest of my files directly to the CACHE drive and let the "mover" do it's work. A few of the files/folders continued to find their way to disk 1, but all the rest where spread out on disks 2, 3 and 4.

 

    Now, this is the point where I noticed that my GUI "SHARES" tab was filling up with lots of folder filenames as "shares" beyond the basic three shares I had originally created (Bluray, DVD and TV). At the time, I couldn't understand why this was happening, hence my original post.

 

    Last night I realized that this might have something to do with unRAID 6 not automatically creating "Bluray" share folders on disk 3 and disk 4 (which it did do on disks 1 and 2). So I manually created a "Bluray" folder on disk 3 and disk 4 and simply selected all the files that were placed in the root of those drives and moved them into the new "Bluray" folder on disks 3 and 4. I shut down the server and this morning turned it on and all the unwanted shares were gone from the GUI "Shares" section, leaving only the original three shares (Bluray, DVD and TV).

 

Until I have more files to transfer, I won't be absolutely sure if everything is back to normal. But, I think for now everything is OK and I will go back to copying files to the "Bluray" share and NOT to the "Cache" share.

 

Again, thanks for your help.

[RobJ]

Just one more point, the file system that is associated with all your User Shares is a virtual one, not permanent, created entirely at the moment the array is started and all disks have been mounted.  When the array is stopped, the User Share file system is destroyed, not the files and folders of course, but the entire /mnt/user folder tree.  If you ever see something that appears corrupted in the User Shares, you only have to stop the array and restart it, to have a completely new User Share file system, built entirely on what it sees at that moment.

[Rizlaw]

Robj,

 

Thanks, I didn't know that. It provides more insight how this is all supposed to work, or not.

[lionelhutz]

    Now, this is the point where I noticed that my GUI "SHARES" tab was filling up with lots of folder filenames as "shares" beyond the basic three shares I had originally created (Bluray, DVD and TV). At the time, I couldn't understand why this was happening, hence my original post.

 

Every top level directory is considered a share so you were creating top level directories on the cache drive which became shares and were moved to the array. You have to create or write into the share directory structure when you write directly to a disk.

 

I think your fill-up allocation method could still cause you heartache.

 

[Rizlaw]

    Now, this is the point where I noticed that my GUI "SHARES" tab was filling up with lots of folder filenames as "shares" beyond the basic three shares I had originally created (Bluray, DVD and TV). At the time, I couldn't understand why this was happening, hence my original post.

 

Every top level directory is considered a share so you were creating top level directories on the cache drive which became shares and were moved to the array. You have to create or write into the share directory structure when you write directly to a disk.

 

I think your fill-up allocation method could still cause you heartache.

 

Agreed: reverted all share split-levels to the default, "automatically split any directory as required".

[JonathanM]

    Now, this is the point where I noticed that my GUI "SHARES" tab was filling up with lots of folder filenames as "shares" beyond the basic three shares I had originally created (Bluray, DVD and TV). At the time, I couldn't understand why this was happening, hence my original post.

 

Every top level directory is considered a share so you were creating top level directories on the cache drive which became shares and were moved to the array. You have to create or write into the share directory structure when you write directly to a disk.

 

I think your fill-up allocation method could still cause you heartache.

 

Agreed: reverted all share split-levels to the default, "automatically split any directory as required".

allocation method != split level.

You may be better served by changing to high water, or most free.

[lionelhutz]

Agreed: reverted all share split-levels to the default, "automatically split any directory as required".

 

Besides that not being the allocation method as already pointed out, some people actually want a split level that keeps associated files for a Movie or TV Series on a single disk which that doesn't do.