Split Level 0 (special case) doesn't work [SOLVED]


Recommended Posts

I'm trying to use Split Level 0, which is supposed to be a special case, but it doesn't seem to be working.

 

I have 3 disks. On disk2 and disk3 I have /unRAID/Videos/, on disk2 I have /unRAID/Videos/Movies/ and on disk3 I have /unRAID/Videos/TV/.

 

So why did unRAID create /unRAID/Videos/Movies/ on disk 3 when I copied a new movie to /unRAID/Videos/Movies/ (user share, not disk share)?

 

unRAID is supposed to see that Movies exists only on disk2, and create new folders under Movies only on disk2.

 

I also had a /TimeMachine/ on disk1, and now I magically have /TimeMachine/ on disk2 and disk3 as well, that I never created. Split Level 0 is supposed to be a special case that does not create top-level folders.

 

I'm using 5.0b6a.

 

Update:

 

Moving a file that is already on the array to a new location will not take into account the allocation method, but will always keep the file on the disk that it is already on.

 

If you want to move a file that is already on the array to another location (e.g. from a downloads folder to a movies or TV shows folder), it is best to use the disk shares to ensure your media does not get split.

 

To avoid this issue when temporary storage is needed (e.g. download, unpack, rename, then move to final destination) you can download to and unpack/rename in a folder on your cache drive (if you have one) that does not get automatically moved to the array. For 5.0beta7 and earlier that means folders with a dot prefix. For 5.0beta8 and later you can use any top-level folder that does not also exist on the array, OR you can create another share that uses the cache disk only.

 

Update 2:

 

Just to clarify when I say "on the array" above I mean in a user share. Moving from one user share to another will always keep the file on the same disk and will ignore the allocation method. Moving a file from the cache disk (and probably a disk share) to a user share will work as expected.

 

Also be careful when using the new "cache only" option. If you create a "cache only" share for your temporary files, you must still configure your applications to move from /mnt/cache/cache-only-share-name instead of /mnt/user/cache-only-share-name. The only benefit to using a "cache only" share in this case is that your folder becomes visible on the network, and does not need to be hidden by prefixing with a dot.

 

Link to comment

The share is "unRAID". Inside there, I have folders for Videos, which has Movies, TV, Recorded TV. Inside "unRAID" I also have other folders for music, etc.

 

I don't want to have a bunch of different network shares, each with different hard coded split levels. I also don't want to rename all my files to include a special character like "[" to achieve different split levels on one share.

 

From what I've read, split level 0 should achieve what I want perfectly. I will manually create the folders on each disk that I want that particular folder to split across (via disk shares). Unfortunately, it doesn't seem to work reliably :(

 

I did some manual tests last night, and it does appear to work as described. I can't reliably reproduce the strange behaviour. However, this is not the first time that files have seemingly randomly ended up on the wrong disk.

 

Screenshots attached of the unRAID share settings, and the global system share settings.

unraid_share_settings.png.6a417c4cf0598e5ec7b72008e5161e2f.png

global_system_settings.png.c606db6555561fdcad85526d578450e6.png

Link to comment

i believe split level only works on the share not the folders in it.

since the share is present on all disks afaik it can use any of those disks to put files in

 

In addition, brought back new-and-improved "split level 0" functionality.  I will be posting a more comprehensive write-up on User Shares and Split Level on the web site and wiki soon, but in the meantime here's how it works.  If you explicitly set a share's Split level to 0 (that is, not blank, but actual numeric value 0), then when a new object (file or directory) is created, unRAID will form the set of disks where the parent directory of the object already exists; and, then the actual disk to use from this set will be selected according to the current share allocation method.  (Still subject to included/excluded masks.)

 

Example, assume you created this folder structure:

disk1/Video/Movies/Action

disk2/Video/Movies/Action

disk3/Video/Sports

 

Now suppose on the Video user share you create "Movies/Action/Braveheart" folder.  With split level 0, even though the Video share spans disks 1..3, since the parent of "Braveheart" only exists on disks 1 and 2, unRAID will create "Braveheart" folder on either disk 1 or 2, according to allocation method.  If no space is available on disk 1 and 2, then operation will fail.  In this case if you want to add more storage for "Movies/Action" folder, you would have to explicitly create "Movies/Action" on some other disk.

 

Similarly, if you create "Sports/soccer-2009-05-02.avi", the soccer-2009-05-02.avi file could only be created on disk3.

 

Along with split0 feature, added also a new allocation method called "Fill up", which will cause disk space to be allocated from the lowest numbered disk provided the amount of free space remaining is greater than "Min free space" (a new share config parameter which also applies to the other allocation methods).  Min free space units are in 1024-byte blocks.

 

rereading the release notes it seems what you do is supposed the be possible, not sure what is happening there

Link to comment

Yeah, I have my shares set up this way as well, and haven't had a problem. I do specify which disks the share resides on, though. Not sure if it's necessary, but just do it since I have other disks with other shares on them--and some that overlap.

 

I have a "Media" share.

 

A few drives have "Media/Movies", "Media/Music", and "Media/TV". Some of the drives overlap, some of them have 2 of the 3 of only 1. But I don't recall having to configure anything else, and it "just working."

 

Posting your full syslog from boot up isn't a bad idea either, as some of the experts here may see something.

Link to comment

FYI, I'm quite sure if you copy a file from somewhere else on disk3 to the Videos user share that it will stay on disk3, no matter which disk the share settings say the file should go. You posted "manual tests" which just made me think you are processing files in some way on the server before copying them to your Videos share. If you process the files on disk3 and then copy them to the share they will stay on disk3. Could this be the issue?

 

 

Peter

 

Link to comment

FYI, I'm quite sure if you copy a file from somewhere else on disk3 to the Videos user share that it will stay on disk3, no matter which disk the share settings say the file should go. You posted "manual tests" which just made me think you are processing files in some way on the server before copying them to your Videos share. If you process the files on disk3 and then copy them to the share they will stay on disk3. Could this be the issue?

 

 

Peter

 

 

I think this is the problem. I have my SABnzbd folder also on the unRAID user share. Files are downloaded here first, and then moved to Movies (by Couch Potato) or to TV (by Sick Beard). So this means that all my files will always go onto the one disk, where they are initially downloaded, regardless of any split level setting?

 

That seems like buggy behaviour to me. I did notice that for some TV shows, many older episodes that had been copied to the share (and their metadata, thumbnails etc) were on one disk, and then I ended up with a bunch of newly downloaded episodes on another disk (just the video files, metadata was still created on the first disk). I guess this is because the video file is downloaded and moved, and so remains on the same disk, but the metadata was created and the rules of the allocation method apply.

 

Do you know why it behaves this way? Is it by design? Is it likely to change? This means I will have to either download to a different machine/disk that is not part of the array before moving files to the array. The downloader machine is currently diskless though (xbmc live installed onto a USB stick, running SABnzbd, SickBeard, CouchPotato, downloading to the unRAID/SABnzbd user share/folder).

 

Link to comment

It's the way the file move operations behave on user-based filesystems. It prefers to keep a file on the same disk since it's significantly cheaper to change a few inode records than it is to move the entire data sectors.

 

It's not likely to change.

 

If you want to move something once it's on the user shares, your best bet is to be specific and use "/mnt/disk#/" instead of "/mnt/user/" (ie: use your explicit disk shares instead of the user share).

 

Another possibility is to download everything to your cache disk directory that is not part of your user shares (for instance "/mnt/cache/.downloads/") and then move onto the user share.

 

Link to comment

My understanding is that Samba causes it and it can't be changed, or to be more precise, can't be changed by Limetech without rewriting Samba which will never happen.

 

Your best bet would be to use a cache disk and do all file operations on it to create the final files. The mover script would then properly move the files to the array disks. You can still move to the share and the files would still stay on the cache disk since the share would now have a cache disk assigned to it.

 

Peter

 

Link to comment

My understanding is that Samba causes it and it can't be changed, or to be more precise, can't be changed by Limetech without rewriting Samba which will never happen.

 

Your best bet would be to use a cache disk and do all file operations on it to create the final files. The mover script would then properly move the files to the array disks. You can still move to the share and the files would still stay on the cache disk since the share would now have a cache disk assigned to it.

 

Peter

 

 

So I just enable the cache disk for the user share and that is it? As long as the mover script doesn't run after the download but before the move to the videos folder, it should choose the disk based on the allocation method?

Link to comment

Do any of those applications allow you to move the file after the download completes? That'll prevent any interference with the mover script--but as long as you download to a folder prefixed with a '.', it won't be moved anyways. But if the file is downloaded to /mnt/user/.downloads, then the application is able to move to /mnt/user/Videos share on complete, that'll solve everything for you (then the file will be temporarily on your cache drive in the same folder mapping, and not moved until the mover script kicks off at 3:40 AM, by default)

Link to comment

Well if the Videos share has "Use Cache Disk" set to "Yes", then he can transfer from /mnt/user/.downloads to /mnt/user/Videos, and it'll stay on the cache drive--just under Videos/, until the script runs.

 

Does the dot prefix prevent the mover script only at the root folder, or also for nested folders? And does the unmenu package that adds an underscore prefix to the script work the same?

 

If I have the cache disk enabled and leave my folders as they are, it should work unless the mover script runs after a download but before the apps rename and move the download? This would be a small race condition, but I assume that moving a file that via the use share which happens to be on the cache disk would remain on the cache disk until the mover script runs?

Link to comment

Well if the Videos share has "Use Cache Disk" set to "Yes", then he can transfer from /mnt/user/.downloads to /mnt/user/Videos, and it'll stay on the cache drive--just under Videos/, until the script runs.

 

Does the dot prefix prevent the mover script only at the root folder, or also for nested folders? And does the unmenu package that adds an underscore prefix to the script work the same?

Both work only at the root level.

 

If I have the cache disk enabled and leave my folders as they are, it should work unless the mover script runs after a download but before the apps rename and move the download? This would be a small race condition, but I assume that moving a file that via the use share which happens to be on the cache disk would remain on the cache disk until the mover script runs?

A new feature in 5.0beta8 will make this much easier as it will allow you to have directories on the cache drive that are not user-shares and not moved.

 

The "mover" script will not move a file currently open for reading or writing.

Link to comment

Also, you can run the mover script manually, by running:

 

/usr/local/sbin/mover

 

The cron job for this is located in: /etc/cron.d/root

 

You could also modify your "go" script to run this more frequently than 3:40 AM. Or, easier yet, have the go script copy a shell script (.sh) to /etc/cron.hourly which contains this single line in it:

 

/usr/local/sbin/mover 2>&1 | logger

 

Make sure the .sh script is executable by root! (chmod 700 cache_mover.sh) with root as the owner.

 

Link to comment

Also, you can run the mover script manually, by running:

 

/usr/local/sbin/mover

 

The cron job for this is located in: /etc/cron.d/root

 

You could also modify your "go" script to run this more frequently than 3:40 AM. Or, easier yet, have the go script copy a shell script (.sh) to /etc/cron.hourly which contains this single line in it:

 

/usr/local/sbin/mover 2>&1 | logger

 

Make sure the .sh script is executable by root! (chmod 700 cache_mover.sh) with root as the owner.

 

 

Wouldn't the problem with that would be, if I am not downloading directly to a directory in the root of the cache drive prefixed with a dot or underscore, but the files are downloaded to the cache drive just because I have cache drive enabled for the user share that they are downloaded to, that the mover script would move the incomplete (or completed but not yet renamed and moved) downloads to the array, and once there, the rename and move would not take into account the split level 0 and disk allocation method but would keep the files on the same disk?

 

I will just try enabling the cache disk for a while first, and leave the mover script to execute once a day at 3:40 AM. Hopefully no downloads will be occurring at that time. If I run into any trouble, I will try downloading directly to a folder on the cache drive that is not moved by the script, or perhaps 5.0b8 will be out by then.

 

Also, if this problem is due to samba, would it still occur if I mounted my shares with AFP or NFS instead? Or would it occur if Sick Beard were running directly on the unRAID server, and not using any network share when renaming and moving downloads?

 

Link to comment

A new feature in 5.0beta8 will make this much easier as it will allow you to have directories on the cache drive that are not user-shares and not moved.

 

The "mover" script will not move a file currently open for reading or writing.

 

Now that 5.0beta8 is out, do you know if it is safe to use the modified mover scripts from unMENU with it? Or must these be disabled?

 

Cheers.

 

Link to comment

A new feature in 5.0beta8 will make this much easier as it will allow you to have directories on the cache drive that are not user-shares and not moved.

 

The "mover" script will not move a file currently open for reading or writing.

 

Now that 5.0beta8 is out, do you know if it is safe to use the modified mover scripts from unMENU with it? Or must these be disabled?

 

Cheers.

 

Answer: I have absolutely no idea.  I've not even downloaded beta8, never mind looked at its mover script.

 

Joe L.

Link to comment

A new feature in 5.0beta8 will make this much easier as it will allow you to have directories on the cache drive that are not user-shares and not moved.

 

The "mover" script will not move a file currently open for reading or writing.

 

Now that 5.0beta8 is out, do you know if it is safe to use the modified mover scripts from unMENU with it? Or must these be disabled?

 

Cheers.

 

Answer: I have absolutely no idea.   I've not even downloaded beta8, never mind looked at its mover script.

 

Joe L.

OK, a quick look and in 5.0beta8  (and likely onward...)

 

See here: http://lime-technology.com/forum/index.php?topic=13866.msg131265#msg131265

Link to comment

I'm confused, maybe you could clear something up. Were you downloading to a temporary share of some sort and then moving the files over to the final destination share?

 

I'm just confused due to the wording you are using. I'm not sure you mean moving and renaming as in the program is moving the final file to the server (having never been on the server before) and then renaming it or if you mean something else.

 

Peter

Link to comment

I'm confused, maybe you could clear something up. Were you downloading to a temporary share of some sort and then moving the files over to the final destination share?

 

I'm just confused due to the wording you are using. I'm not sure you mean moving and renaming as in the program is moving the final file to the server (having never been on the server before) and then renaming it or if you mean something else.

 

Peter

 

SABnzbd downloads the files, verifies them with par2, unpacks them. Then SickBeard or CouchPotato looks up the meta-data for the download and renames/moves the files to their final destination. E.g. it goes from /mnt/user/unRAID/SABnzbd/Complete/Movies/some-folder/some-file.avi to /mnt/unRAID/Videos/Movies/The Movie Name (2011)/The Movie Name.avi.

 

Of course, due to this issue of files already on the array remaining on the disk they are currently on when performing a move operation, I'll soon be creating a "cache only" share for SABnzbd.

 

Link to comment

That makes more sense. And yes, you will need to have the downloads on the cache for it to work.

 

While you are at it, I'd go ahead and install the other programs on the server too.

 

Peter

 

 

I had them running on unRAID before. I created the new unMENU packages for all three programs (in

another thread), but found that it's just too much work trying to get these things running reliably on unRAID, especially when they can cause problems like preventing the array from stopping due to files being open. I have another machine that is on 24/7 that runs Ubuntu (XBMC Live install), and it is much easier to install and maintain these programs on that machine while avoiding these problems. Now I feel much better that unRAID is dedicated solely to safe storage of files, and no rogue processes are running that might interfere with that.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.