January 13, 201115 yr I'm convinced something is buggered with the split level algorithm; either that or my 20 years IT experience has been for nought. As per Tom's own admission, setting a split level of 0 tells unraid to unconditionally put a newly created file on the same disk of the parent - essentially letting you manage the location of files yourself. Using this as an example, this is where I'm convinced something is wrong /mnt/user/videos - split level 0, "fill" allocation, 10GB minimum free-space I have a folder for example full of episodes /mnt/disk2/videos/tvshows/Lost/Season 1/ (disk 2 has 30GB free) I copy a new file to: /mnt/user/videos/tvshows/Lost/Season 1 and unRAID ends up creating a new folder on disk5 and copies the file there (/mnt/disk5/videos/tvshows/Lost/Season 1). I've also tried setting the split level to 2, changing the allocation method.. This has happened on 4.6 and 5.0b2 servers that I'm running. Hints? PS: Didn't want to hijack the thread, but I think it's OT.
January 13, 201115 yr My head hurts I said it was a draft, I was just trying to prevent someone from re-inventing the wheel (no good deed goes unpunished). Look, what split-level does is very simple, just explaining it has proven not to be for some reason. Let's try it this way... Suppose you start with a blank array and you create a single user share called Movies, but so far there are no files or subdirectories in the share yet. In this state the Movies share will show up in windows under Network, but if you click on it, it will be empty. Let's also assume the split-level field for the Movies share is blank, all the disks are the same size, and the allocation method is "most-free". Now let's say you create a directory in the Movies share called Alien. Now, unRaid has to decide which disk it will create the Alien directory on. Let's say it picks disk2. (To verify this click on the disk2 disk share under Network on windows. You will see Movies, and if you click on Movies you will see Alien.) Next lets say you go back and click on that Aliens directory under the Movies user share, of course it will be empty. Now create a directory there called VIDEO_TS. Again, unRaid has to decide which disk to create this VIDEO_TS directory on. Let's say it picks disk3. (To verify this click on the disk3 disk share under Network on windows. You will see Movies, and if you click on Movies you will see Alien and if click on Alien you will see VIDEO_TS. When unRaid picked the disk to use, it noticed that the parent directories needed before it created VIDEO_TS did not exist on disk3, so it created them first, then created the VIDEO_TS directory). In this fashion, any time you create an object (directory or file), unRaid will pick a disk to put it on, and if necessary create all the parent directories. So... this is the normal action. But in the case of that VIDEO_TS directory, we don't want unRaid to pick just any disk to create it on, we want it to pick the same disk that the Alien directory is already on. This is where split-level comes in. What the code does is count the number of directories from the root of the share to the parent directory that we are creating an object in. If this count is equal to or higher than the split-level value, then instead of picking any old disk, unRaid will create the object on the same disk that it found the parent directory on. In the example we're looking at, to keep VIDEO_TS (and everything underneath it) on the same disk as the Alien directory, we set split-level to 1. Did I just add to the confusion? Edit: added explicit pre-conditions so you can actually try it on a live system. Edit2: fixed the added explicit pre-conditions
January 13, 201115 yr I'm convinced something is buggered with the split level algorithm; either that or my 20 years IT experience has been for nought. As per Tom's own admission, setting a split level of 0 tells unraid to unconditionally put a newly created file on the same disk of the parent - essentially letting you manage the location of files yourself. Using this as an example, this is where I'm convinced something is wrong /mnt/user/videos - split level 0, "fill" allocation, 10GB minimum free-space I have a folder for example full of episodes /mnt/disk2/videos/tvshows/Lost/Season 1/ (disk 2 has 30GB free) I copy a new file to: /mnt/user/videos/tvshows/Lost/Season 1 and unRAID ends up creating a new folder on disk5 and copies the file there (/mnt/disk5/videos/tvshows/Lost/Season 1). Impossible. Are you sure you set split-level to single numeric digit zero (0)?
January 13, 201115 yr Probably Aha.. I think I thought of something here. And found a bug (oh no!?) - NewFile.ext is created on /mnt/user/extracted/files/blah - it's therefore created on let's say disk5 (whatever the share "extracted" is set to). - The file is then MOVED to /mnt/user/videos/tvshows/Lost/Season 9 (which is on disk2).. Result: the file won't get moved to another disk once it's on the user-share, thus forcing the creation of the directory on the same disk where the file currently resides Bug or intended behaviour?
January 13, 201115 yr Ok what happens if you choose one method and then want to change. However you have the share settings at the time an object is created will take effect.
January 13, 201115 yr Probably Aha.. I think I thought of something here. And found a bug (oh no!?) - NewFile.ext is created on /mnt/user/extracted/files/blah - it's therefore created on let's say disk5 (whatever the share "extracted" is set to). - The file is then MOVED to /mnt/user/videos/tvshows/Lost/Season 9 (which is on disk2).. Result: the file won't get moved to another disk once it's on the user-share, thus forcing the creation of the directory on the same disk where the file currently resides Bug or intended behaviour? The intended behavior is not move any files around. So yes, let say you have: /mnt/user/extracted/files/blah/NewFile.ext (and this is actually /mnt/disk5/extracted/files/blah/NewFile.ext) and now you move it (via 'mv' command in command shell), unRaid will create /mnt/disk5/videos/tvshows/Lost/Season 9/ and then mv the file there on disk5. It will also leave any now-blank subdirs there as well. BUT, if you did the 'move' via Windows explorer (right-click, select 'move') it would have behaved the way you expected.
January 13, 201115 yr What then if a file is created on a disk /mnt/disk5/blah/blah2/newFile.ext (no user share actually defined). and then mv newFile.ext /mnt/user/videos/tvshows/Lost/Season 9 will it then get put on the user share on the right disk?.. I mean, I know I can do this via a cache drive, but I had eliminated my use of a cache drive and was hoping it wasn't necessary anymore.
January 13, 201115 yr What then if a file is created on a disk /mnt/disk5/blah/blah2/newFile.ext (no user share actually defined). and then mv newFile.ext /mnt/user/videos/tvshows/Lost/Season 9 will it then get put on the user share on the right disk?.. I mean, I know I can do this via a cache drive, but I had eliminated my use of a cache drive and was hoping it wasn't necessary anymore. By "right disk" you mean disk2? No, it won't move files, it will create the parent path on disk5 first, then 'mv' it there (it stays on disk5). If you are wanting to move to the correct disk, and you are using the shell, then you should use the explicit disk path of the target: mv /mnt/disk5/blah/blah2/newFile.ext /mnt/disk2/videos/tvshows/Lost/Season 9/
January 13, 201115 yr Unfortunately, everything is automated with various programs so it'd be almost impossible to implement that. I guess there are two alternatives: 1) Use a cache drive so that when files are moved from their temporary location (cache) to their permanent (/mnt/user/...), they're put in the right place 2) Officially requests a share feature that can be enabled/disabled (disabled by default) to "strict allocation" which will do whatever it takes to keep the files together, even if moved around the array.
January 13, 201115 yr 2) Officially requests a share feature that can be enabled/disabled (disabled by default) to "strict allocation" which will do whatever it takes to keep the files together, even if moved around the array. Here's the problem with this. From a system point of view, /mnt/user is a single file system. So when you type the 'mv' command, the file system actually sees a RENAME operation (because mv thinks it's moving a file within the same file system and it knows it doesn't actually have to move it, just link it into a different parent directory). Now I can't tell the difference between a RENAME intended to do a physical move vs. a RENAME intending to just actually rename the file.
January 13, 201115 yr Guess it would have to be "cp".. you cp first then remove.. (although remember, /mnt/disk2 and /mnt/disk5 are two different disks, so if it's moving from 1 disk to another, that shouldn't be a problem for the system).. Otherwise, if you want to do it from the user share: hash check cp file hash check rm file done. Now that I know this behaviour, it may make troubleshooting the split level issues I've been having much easier (and maybe others in this forum who use software such as sickbeard/sabnzbd). This will REALLY mess with the split level if they don't store all the downloaded things on a cache drive first. To get back on topic, I can see how Split Level can be a pain.. How about getting a developer to program a gui for split level? or provide something within the gui that better explains it (little popup, etc.)
January 13, 201115 yr Ok so it sounds like I have 2 options, well more I know, If I create Movies and do a split level of 0 I can create as many sub folders as I want. Will movies span across all disks and still use most free or high water? Or would I be better served just setting split level to 5 and telling the wife not to get crazy with the sub folders in her picture file. What are the draw backs to setting the split level at a higher number than you think you will need or just setting it to 0.
January 13, 201115 yr Ok so it sounds like I have 2 options, well more I know, If I create Movies and do a split level of 0 I can create as many sub folders as I want. Will movies span across all disks and still use most free or high water? Or would I be better served just setting split level to 5 and telling the wife not to get crazy with the sub folders in her picture file. What are the draw backs to setting the split level at a higher number than you think you will need or just setting it to 0. It would be best if you post what you are trying to accomplish. Personally, I have a Photos share that is restricted to just one disk, a Music share restricted to just one disk, and a Backup share restricted to just one disk. This way, whenever I need to access one of those shares, just one disk spins up. (Actually I have Photos and Music on the same 2TB disk.) For rips, I have a single Movies share, but underneath that I have "DVD" and "BD". I want to keep all the DVD on separate drives than the BD (why? my OCD I guess), so I use split-level 0 and manually create the Movies/DVD and Movies/BD directories on the disks that I want to contain only DVD and BD. I should add: our Photos exist in a folder on a PC that is used for reading off cameras, and the Photos share is used to backup this folder, and every once in a while, I zip up the photo directories and copy to an off-site backup. I don't do this for our Video probably should
January 13, 201115 yr Thanks. I want to create user shares for Movies, music, pictures, gro pro movies, home movies, Backup, etc. I would like the shares to never fill up based on the available disk space, I currently have 2x2TB data disk and a 500 gb cache disk. I want to be able to keep the files in my gro pro and photo folders together so that when I import them into Imovie and I photo they are all together. So Go Pro would look something like: Go Pro "user share" Mountiain Bike "folder" Santos/Vortex "Folder" Video 1.mp4 "file" Video 2.mp4 "file" etc. When I want to import the Santos/Vortex file into I movie I would like all the file together. My wife would use the picture folder for something like this: Picture " User Share" Sam "file" 1 Year old "file" Jan "file" Feb "File" I thought about using separate disks Disk 1 is a 2Tb seagate I wanted to put anything that is video related on it. Disk2 is a western digital ears 2tb and I was going to put things that are rarely used but need to be kept together on this drive, music, pictures and the back up share. Disk 1 will fill up first so when it comes time to add disk 3 to the array will I have to change anything other than go to the Movies share and include disk 3 to the share. Does any of this make sense. I know what I think I want to do in my head but I am not sure I am explaining it that well. Thank you so much for taking the time to help me out. I guess performance wise it really makes no difference if the movies I stream fom the movies folder will be on the 5900 rpm drive or the 5400 rpm drive.
January 13, 201115 yr Bags; I think you need to more clearly lay out your desired directory structure. It sounds like go pro should be spit level 1 but I'm not clear. You don't create any subdirectories in the pictures share? If you have something like a pictures share that will never fill a single drive it could be set to just include a single drive. That way, the split level really does not matter. I'm convinced something is buggered with the split level algorithm Say we do all from another PC accessing the network shares. Do we agree on the following; Moving a file around inside a share may not respect the split level. Copying a file inside a share will respect the split level. Moving a file from share to share will respect the split level. Say we do some work from the command line on the server. Do we agree on the following; Moving a file around inside a share may not respect the split level. Moving a file from share to share may not respect the split level. Copying a file will respect the split level. When a move does not respect the split level the file will stay on the same disk it originally resided on, even when the split level setting means it should go to another disk. Peter
January 13, 201115 yr I thought it might sound confusing. I would like to keep it simple. Ok I might as well start out with what i want to do with this unraid server, I am sure it will change in the future. Currently I have the unraid pre cleared ready to start transferring data. My hardware is listed below. My immediate plan is to set up the following user shares: Movies, Music, music videos, pictures, Tv shows, Back up, Books, Home videos, Go Pro Movies, i photo events, i photo projects, i movie events, i movie projects. And I would also like to set up a folder on the cache drive called "Eye Fi" which will be for photos taken from a wifi sd card. The eye fi folder will be excluded from the picture folder and then manually transferred into the picture folder after the new file has been created and named. There will be sub directories under the pictures share. My wife already has 6 years of photographs of my daughter saved. The way she saved them is 1 yr, 2 yr, 3 yr, etc. Each year has a sub folder for each month. For example 1 year is the root folder which contain 12 sub folders one for each month of the year. I realize anything that is video related could go under the movies share. I am keeping them separate to keep it flexible. I understand that both Plex and XBMC dont let you do too much configuring on setting your own genres. Home videos has 3 separate movie formats DV, DVD, HD or MP4. My future plans are the following: 1) Stream movies back up from dvd and blue ray, unfortunately they are in different formats. I believe I want to use Plex to manage my library, I am choosing plex because I have 2 i phones, 1 Ipad, and 1 touch I would like to stream movies to. I also have the apple tv 2. I understand as of today plex will not run on unraid and it will not run effectively on Apple tv2. I am not married to Plex or Apple Tv2. I have considered going the HTPC route. So I would like to set my movies folder up for the best success no matter which way I go. 2) Install handbrake to run in unraid and transcode or re encode the back up dvds. 3) Back up the mac mini and the mac book pro from another external HD which is already tied to time machine. I understand the new unraid system is hoping to support time machine, however I have not looked at it hard enough to see if time machine can back up to different places. If not I was thinking about using crash plan to go from the external HD that already has the info on it. Or use crash plan strait from the mini and the mac book. 4) maybe run transmission. If you guys can get me up and running from this post feel free to use it hopefully this also helps out other new users. I am hoping to set it and forget it until the time comes to add more drives. Once I run out of space I want to be able to buy another drive or 2 pre clear them, install them, power up and pick up where I left off so I dont have to do too much or any messing with shares other than adding new ones. If anyone one understands what I am trying to do and see a better way PLEASE let me know. Again thank any and everyone who has taken the time out to help me. I hope one day to help someone else out.
January 14, 201115 yr "Personally, I have a Photos share that is restricted to just one disk, a Music share restricted to just one disk, and a Backup share restricted to just one disk. This way, whenever I need to access one of those shares, just one disk spins up. (Actually I have Photos and Music on the same 2TB disk.) For rips, I have a single Movies share, but underneath that I have "DVD" and "BD". I want to keep all the DVD on separate drives than the BD (why? my OCD I guess), so I use split-level 0 and manually create the Movies/DVD and Movies/BD directories on the disks that I want to contain only DVD and BD". Limetech, When using the above configuration, when those drives are full and you add new drives to the array other than pre clearing the new drives does anything else have to be done? To achieve the desired above settings I would name the user share, split level 0, then use the include, exclude to determine which disk is used. I am guessing the allocation method would not apply since it would not span disks?
January 14, 201115 yr "Personally, I have a Photos share that is restricted to just one disk, a Music share restricted to just one disk, and a Backup share restricted to just one disk. This way, whenever I need to access one of those shares, just one disk spins up. (Actually I have Photos and Music on the same 2TB disk.) For rips, I have a single Movies share, but underneath that I have "DVD" and "BD". I want to keep all the DVD on separate drives than the BD (why? my OCD I guess), so I use split-level 0 and manually create the Movies/DVD and Movies/BD directories on the disks that I want to contain only DVD and BD". Limetech, When using the above configuration, when those drives are full and you add new drives to the array other than pre clearing the new drives does anything else have to be done? Say I add "disk8" to be used only for BD rips. After adding the disk, I use windows explorer and navigate to the disk8 share. Here I create a Movies folder. Then I click on Movies folder and create BD folder. Done. To achieve the desired above settings I would name the user share, split level 0, then use the include, exclude to determine which disk is used. I am guessing the allocation method would not apply since it would not span disks? If you are restricting a user share to just one disk, all you have to do is set "Included disk(s)" to whatever disk you want it to be restricted to, eg, disk1. The split-level and allocation method can be anything because they are meaningless when a share is restricted to a single disk.
January 14, 201115 yr um I was trying to avoid having to make new files everything I add a disk. If disk 1 has all video content and fills up can I add disk 3 go back to the video user share and change it to include disk 1 and disk 3 select most free. Will it then create the folders on disk three and start adding new files to disk 3 since the allocation method is most free? It looks like I might have to go back and figure out my directory structrue and figure out split level. I do Thank you for taking the time out to help me get the most out of my unraid system.
January 14, 201115 yr um I was trying to avoid having to make new files everything I add a disk. If disk 1 has all video content and fills up can I add disk 3 go back to the video user share and change it to include disk 1 and disk 3 select most free. Will it then create the folders on disk three and start adding new files to disk 3 since the allocation method is most free? It looks like I might have to go back and figure out my directory structrue and figure out split level. I do Thank you for taking the time out to help me get the most out of my unraid system. Yes to avoid having to create the parent directories you should use split-level. Split-level works best when the directories on the share are 'flat', for example, If the entire movie share were organized like this: Movies/<movie name directory>/<movie files> use split level 1. Or if like this: Movies/<Genre>/<movie name directory>/<movie files> use split-level 2. Where you get in trouble is if you mix the two: Movies/Alien/<alien files> Movies/Kids/BugsLife/<bugslife files> no easy way to use split-level here.
Archived
This topic is now archived and is closed to further replies.