new split level

[Vr2Io]

10 minutes ago, Squid said:

http://man7.org/linux/man-pages/man1/touch.1.html

Seems those for file, does this apply for directory also. Thanks

 

Just got your edit, later will try. Thanks

Edited April 29, 2018 by Benson

[Squid]

mkdir -p /blah/test
ls -ail /blah

8842752 drwxrwxrwx  3 root root  60 Apr 29 14:53 ./
      2 drwxr-xr-x 19 root root 420 Apr 29 14:53 ../
8842753 drwxrwxrwx  2 root root  40 Apr 29 14:53 test/

touch /blah/test -d @123
ls -ail /blah

8842752 drwxrwxrwx  3 root root  60 Apr 29 14:53 ./
      2 drwxr-xr-x 19 root root 420 Apr 29 14:53 ../
8842753 drwxrwxrwx  2 root root  40 Dec 31  1969 test/

 

[NewDisplayName]

before (what split level now does, if u dont set it, or set it wrong):

/some/share/films/film1

-> file1 (hdd1)

-> file2 (hdd2)

-> file3 (hdd3)

 

/some/share/films/idc1

-> file1 (hdd1)

-> file2 (hdd2)

-> file3 (hdd3)

 

/some/share/films/idc1/something/ishere

-> file1 (hdd3)

-> file2 (hdd2)

-> file3 (hdd1)

 

/some/share/series/serie/serie s01/film1

-> file1 (hdd1)

-> file2 (hdd2)

-> file3 (hdd2)

 

after (what my new "zero intelligence needed" smart split level could do:

/some/share/films/film1

-> file1 (hdd1)

-> file2 (hdd1)

-> file3 (hdd1)

 

/some/share/films/idc1

-> file1 (hdd3)

-> file2 (hdd3)

-> file3 (hdd3)

 

/some/share/films/idc1/something/ishere

-> file1 (hdd5)

-> file2 (hdd5)

-> file3 (hdd5)

 

/some/share/series/serie/serie s01/film1

-> file1 (hdd2)

-> file2 (hdd2)

-> file3 (hdd2)

 

Maybe this is more clear.

 

I dont see anything why this should not work. The only real question is, how much ppl would use it and is it worth implementing, for me, i think i would use it in nearly all shares. And i guess, im the standard "a little bit" expierinced guy.

 

I edited my first post to show the progress we did in this thread.

Edited April 29, 2018 by nuhll

[Vr2Io]

I understand that, but there have many other factor will cause not practical.

 

For example (some member already pointout), I have a large directory, inside was bluray ISO file, this need 2+ disk, that means must split across disks.

 

If split was perform during writing and no space left and trigger large amount file in_out. Would user waiting for such longtime.

 

I have a backup server, which sync in disk by disk, so I can easy import FIP (checksum) to the backup instead calc one more time (this is my case). Checksum suit also in same way.

Edited April 29, 2018 by Benson

[NewDisplayName]

I think you see it the wrong way.

 

Its the same as it is already with the split levels today at this moment.

 

Just other criterias.

 

And like i said, if the files in that directory wont fit into a single hdd, just fallback to standard random.

Its nothing new, its just advanced because you dont need to FIX set the split level for the whole share.. Thats the only difference.

 

My adanced idea to this, a "nice to have" feature would be splitting and moving AFTERWARDs.

It just changes and makes it more complicated and more resource heavy if you AFTERWARDS put (files in directorys) together. But that is just an nice to have feature which probably isnt worth making - i dont know. Or someone just writes and plugin for it.

 

 

Edited April 29, 2018 by nuhll

[Vr2Io]

I agree someone may need this, but drawback may be too much.

Edited April 29, 2018 by Benson

[lionelhutz]

That already kind of exists. Set split to manual and create the top level share directory on every disk you want the share. Create sub-directories in the share on each disk if there are other directories you want to be split across multiple disk.  Then, each new sub-directory will only be created one time and then anything put into that sub-directory or below that sub-directory will remain on a single disk. It's not quite the same because it doesn't allow for a sub-sub-directory to go to a different disk. 

 

Your idea really doesn't work require more work and reading each directory because once /some/share/films/idc1 is created on 2 disks then the mover has to go to each disk and see if the directory specifically contains a file but not directories before knowing where the file should go. The current setup more or less works by directories only, not caring about there being files in the directory.  So, it writes to the directory and the file goes to the disk dictated by the split level and allocation method without regard to reading the current disk contents. 

Edited April 30, 2018 by lionelhutz

[itimpi]

3 hours ago, lionelhutz said:

That already kind of exists. Set split to manual and create the top level share directory on every disk you want the share. Create sub-directories in the share on each disk if there are other directories you want to be split across multiple disk.  Then, each new sub-directory will only be created one time and then anything put into that sub-directory or below that sub-directory will remain on a single disk. It's not quite the same because it doesn't allow for a sub-sub-directory to go to a different disk. 

The manual method does allow for sub-directories to span multiple disks as long as you manually create the sub-directory on all the disks to be involved.   Once you have done that then the allocation method comes into play to select which disk to use to hold the contents of that sub-directory.

[NewDisplayName]

I dont see any negative impact.  But maybe a unraid dev can tell us more about it. @limetech

 

Split level needs to check anyway which directory it should go on which disk, so there is no difference, but we need a new rule while he do that (to be "smart"). Its that simple.

 

You can do everything manual, thats not the point, were talking about automation which could be more effective without any negative impacts. And also, if someone dont want to use it, he just dont pick smart split.

 

Noobs may choose it and wont have a negative impact, but the positive effects, ppl who know how split level work and have a share for each thing, can still set it manual like they want.

Edited April 30, 2018 by nuhll

[pwm]

I don't really believe in "Its that simple." and "without any negative impacts."

 

I believe unRAID either needs to spin up more disks or cache more file information.

[NewDisplayName]

Im no programmer, so i cant tell exactly, its possible, and if someone with exp with unraid programming could lighten us up, it would be cool.

 

i think the normal split level also has to check which hdd and which directory, so it should be cashed (or already read), it just needs to put instead of random (hdd) or specific to the split level selected(hdd)  put new files into the hdd the other files are. From information needed i dont see a difference in the split levels we actual have. It just needs to make  a different decicion bases on the facts he already know.

Edited April 30, 2018 by nuhll

[pwm]

The current implementation basically needs to keep track of free disk space, and all new root paths down to the configured split level. With your idea, I think unRAID needs to keep track of all new paths for any depth.

[bonienl]

I understand the wish to make split levels more easy to use and a "smart" setting certainly is a nice idea, but it is not a trivial thing to add.

 

Your assumption is that all folders under the share have the same layout and based on this layout a "smart" level can be calculated.

It becomes a lot more complicated when different layouts exists, e.g. user has a mix of mkv folders, DVD folders and blu-ray folders. They all have different folder structures/depths, and need to be treated differently.

 

In addition when a share spans multiple disks it may result in an undesirable spin up of disks to calculate the "smart" level.

 

The current implementation does not need to calculate the split level, as it is given in the setting of the share. Consequently it can make a direct decision based on the selected allocation method.

 

[limetech]

On 4/24/2018 at 1:19 PM, nuhll said:

-Put all files, if possible (size)(which are in the same directory) on same HDD-

 

I see what you want to do...  Would this be the rule?

 

1. When creating a Directory, use split-level, included/exclude to determine which device to create the directory in.

2. When creating a File, always try to create the file on the same device that contains the parent directory.

[NewDisplayName]

3 hours ago, limetech said:

 

I see what you want to do...  Would this be the rule?

 

1. When creating a Directory, use split-level, included/exclude to determine which device to create the directory in.

2. When creating a File, always try to create the file on the same device that contains the parent directory.

1. yes. All like normal. 

 

2. just to be sure, parent directory would mean the directory before or? I mean the current directory. 

 

So if you have /some/dir/i/dont/know

every directory could be on a seperate disk

 

Is this possible?

Edited April 30, 2018 by nuhll

[lionelhutz]

3 hours ago, limetech said:

 

I see what you want to do...  Would this be the rule?

 

1. When creating a Directory, use split-level, included/exclude to determine which device to create the directory in.

2. When creating a File, always try to create the file on the same device that contains the parent directory.

 

No, that is not what is being asked for.

 

#1 = when creating a new directory, only use the allocation method and include/exclude to determine which device gets the directory.

 

#2 = when creating a file where the directory exists, only put the file onto the device where files exist inside the directory. Only use the device where the directory contains files even in cases where the directory exists on more than one device.

 

I would also add about#2 that there MUST be a rule that gets followed if the file can't be created on the same device as the existing directory that contains files. Otherwise, what is supposed to be a directory of files on one disk could instead end up being spread out over 2 or more disks.

 

The rules limetech proposed are the same as using the manual method except for the top level share directory automatically being created on every disk in the share.

 

 

 

6 hours ago, nuhll said:

You can do everything manual, thats not the point, were talking about automation which could be more effective without any negative impacts. And also, if someone dont want to use it, he just dont pick smart split.

 

 

What I posted is NOT doing everything manual. You manually create the top level share directory on each disk ONLY. Once they exist, all other directories created inside the share AUTOMATICALLY get created on one disk and one disk only.

 

Edited April 30, 2018 by lionelhutz

[NewDisplayName]

I guess i could explain it better in german...

But the e.g. i gave, should explain it. I guess.

[NewDisplayName]

55 minutes ago, lionelhutz said:

What I posted is NOT doing everything manual. You manually create the top level share directory on each disk ONLY. Once they exist, all other directories created inside the share AUTOMATICALLY get created on one disk and one disk only.

 

 

I dont get it, sorry.

 

[lionelhutz]

Make a directory called "Share" on every disk you want to have "Share". Then, every sub-directory you create in "Share" will only be created on one disk.

[NewDisplayName]

And whats about the next folders after share?

[limetech]

10 minutes ago, lionelhutz said:

Make a directory called "Share" on every disk you want to have "Share". Then, every sub-directory you create in "Share" will only be created on one disk.

 

Unless you manually create a same-named path on another disk.  Then it uses allocation method/exclude/include to determine which disk to create the object on.

[NewDisplayName]

Just now, limetech said:

 

Unless you manually create a same-named path on another disk.  Then it uses allocation method/exclude/include to determine which disk to create the object on.

 

How u like my smart split idea?

[limetech]

21 hours ago, nuhll said:

How u like my smart split idea? 

 

Has it's place I guess.  Suppose you have this:

 

Movies/Aliens/video_ts/*.vob

 

All the *.vob files will get placed in same directory (video_ts), but nothing forces video_ts to exist in Aliens.  Would be possible (though unlikely) that

Movies/Aliens  (directory) exists on say disk1, and then

Movies/Aliens/video_ts (directory with files) get created on disk2.

Hence perhaps possible 2 disks spin up to watch Aliens.

 

So maybe you wouldn't use this 'smart split' with such structured contents, but then it's not so smart ?

[lionelhutz]

23 hours ago, nuhll said:

 

How u like my smart split idea?

 

 

Just don't create manual directories..... 

 

 

 

23 hours ago, nuhll said:

And whats about the next folders after share?

 

As I already explained.

 

23 hours ago, lionelhutz said:

Then, every sub-directory you create in "Share" will only be created on one disk.

 

[NewDisplayName]

11 hours ago, limetech said:

 

Has it's place I guess.  Suppose you have this:

  

Movies/Aliens/video_ts/*.vob

 

All the *.vob files will get placed in same directory (video_ts), but nothing forces video_ts to exist in Aliens.  Would be possible (though unlikely) that

Movies/Aliens  (directory) exists on say disk1, and then

Movies/Aliens/video_ts (directory with files) get created on disk2.

Hence perhaps possible 2 disks spin up to watch Aliens.

 

So maybe you wouldn't use this 'smart split' with such structured contents, but then it's not so smart ?

Yes in that case, its not THAT smart, but with wrong split level and or "random" mode it could be even more bad. Like maybe 3 or 4 disks spin up.

 

I would like to beta test it!