March 19, 200818 yr The idea came to me when I thought if changing my original concept of doing this (example): Disk 1: Audio Video ...etc. Disk 2: Audio Video ...etc. Disk 3: Audio Video ...etc. ...since this would probably involve having all disks spin up, consuming energy and creating delays. So my second option was this: Disk 1: Audio Disk 2: Video Disk 3: Audio ("overflow") Video ("overflow") ...etc. This would mean I would keep 3 disks spinning (inc. parity) not all. Remember this is just an example. Same thing if I had more than one folder type on each disk and keep one as extra space for all folders of all other disks (and more than 3 disks - in my case 7 disks originally). Then it came to me... Right now you make (or unRAID on your command makes) one folder for each user share you need in the disks you tell it to. Right? If those disks fill up, even if other disks have empty space, you cannot use it, until you manually add the folder name to that "emptier" disk. Right? Wouldn't be a good idea to add an "allow overflow" flag for user shares and an "allowed for overflow use" for physical disks? This would automatically take care of this manual work. An example: Disk 1: (marked for Video overflow) Audio Video (overflow of Video share in disk 2) Disk 2: (marked for Audio overflow) Video Audio (overflow of Audio share in disk 1) Disk 3: (marked for Audio and Video overflow) Audio (overflow of Audio share in disk 1) Video (overflow of Video share in disk 2) ...etc. The above example is a bit stupid (cross-overflowing the folders wouldn't make any sense, as instead you could use more of each disk for its "main type" of data), but is just to demo the idea. In most cases, each disk would have 2-3 main user shares allowed to overflow on the last disk of the array. This way each disk would have MOST of the data of each "type", while the last disk would have a bit of all the types all other disks have (and didn't fit in them). So all your videos are in disk 1, but what didn't fit is allowed (without you knowing) to use the last disk of the array (that is flagged as allowed for use in an overflow case), or maybe the two "last" disks. Same for your audio, all in disk 2 and what couldn't fit, in last disk. Then if you replace your disk 1 with a larger one, you could then manually copy back your extra data from the last disks (or any disks where you allowed to write overflow), to your "main for that type" disk. How is this different from what happens now? Now you manually setup all share folders on the disks you want and define some allocation method variable to arrange how data are distributed on those user-made-user-shares. This doesn't mark any disk as the "main" disk for the share, so keeping less disks spinning, is not directly controllable (without manual work). With the idea of overflow, one or more disks are the "main" disks for each user share (like before), but EXTRA disks are "allowed" to AUTO-CREATE the user share (ideally the last disk of the array would be kept mostly empty just for this) so that you don't get disk full notifications when in fact THERE IS disk space in some disks. It is a shift of the main concept, but VERY close in its implementation. Setting a new user share would be like that (note that order is important): Share: Videos Share disks: disk1, disk2 Overflow disks: disk5, disk6, disk4 Share: Audio Share disks: disk3, disk4 Overflow disks: disk5, disk6 (in this example, I expect high input of videos, so I am allowed to use disk4 that is normally for use by Audio -if it has space- AS A LAST RESORT, if disks1, 2, 5 AND 6 don't have enough space) (again in this example, let say for Videos, disk1 and 2 are used in the normal sense like today, with the allocation method used today, and ONLY AFTER they are filled up, would disk5, THEN disk6 and THEN disk4 be used) In simpler words, the idea of a "second level" of disks used for user shares is introduced. The idea could be extended in the future even to allow for "transparent" file moving (by unRAID itself), to keep user shares as "compacted" as possible (eliminating overflow) in case you replace with bigger disks or if you erase data from the "main" disks! This would mean significant work by Tom, but the "first level" of the idea mentioned above (the use of two flags for each user share) seems rather simple even for the next 4.Y version! What do you people think? Tom?
March 19, 200818 yr I had a simalar though. See this thread: http://lime-technology.com/forum/index.php?topic=1604.0 The idea was to have a third allocation method for user shares. Sequential. Which would fill up the first disk in the allowed list, then move to the next, etc.
March 19, 200818 yr I would think you could do this by using the include disks/exclude disks to define what disks are part of the user share. If the list is not sorted internally perhaps it uses them in a sequential fashion. I guess that is for Tom to answer. (or someone to experiment with).
March 19, 200818 yr Author Armbrust indeed your idea is close to mine, but I'd say mine is a bit more "elaborate" (it is both your idea and the idea that already works in unRAID, at the same time). Weebotech, include/exclude has no idea of "preference" (of disks over other disks), or "sequentiality" (is there such a word?), that is in fact introduced by my idea. I think it would make shares much cleaner AND less restricted at the same time (which with current implementation, one prevents the other)... Let's hope we have Tom's feedback on this.
March 19, 200818 yr Then what's the purpose of the highwater value? It was my impression that you can include disks you want as part of the share you can set a highwater value that means, when a disk approaches this value go to the next disk. This does not encompass the sequential (round robin) space allocation, however it does seem to handle the overflow disk allocation as long as the INCLUDE DISKS parameter is searched through sequentially and you have a good value for split level. Perhaps I have brain fog. I always thought something like this. DISK1 AUDIO = INCLUDE (sda, sdb, sdc) HIGHWATER=(some high water value) DISK2 VIDEO = INCLUDE (sdb, sda, sdc) HIGHWATER=(some high water value) So am I to assume that allocation to the user shares does not take into account 'the order' of the INCLUDED DISKS ? (I don't have that many files allocated on my system to test this). Additional reading, See high-water allocation algorithm: http://lime-technology.com/forum/index.php?topic=1520.msg10701#msg10701
March 19, 200818 yr Author maybe you are right yet, it is not clear if there is order in included disks the difference in "my" ("my" is a bit strong since I haven't IMPLEMENTED anything) system is that I make sure both that there is order and that there are two levels of "preference" on choosing where to write - something that would make more sense if ever the SECOND part was implemented too (compacting data to the "primary" share disks and automatically cleaning the overflows - in case some data from the primary share disks are manually erased or the whole disk was replaced with a bigger one) I am happy with the system currently, I am just proposing the "next step" (and the step after the next step)
March 23, 200818 yr Then what's the purpose of the highwater value? It was my impression that you can include disks you want as part of the share you can set a highwater value that means, when a disk approaches this value go to the next disk. I didn't think you could set the highwater mark - it was just taken as 1/2 the larget disk in the user share.
March 23, 200818 yr Then what's the purpose of the highwater value? It was my impression that you can include disks you want as part of the share you can set a highwater value that means, when a disk approaches this value go to the next disk. I didn't think you could set the highwater mark - it was just taken as 1/2 the larget disk in the user share. You are correct, my misunderstanding there. (It's been a while since I've been at that screen and I remember setting something, but that was split level). Perhaps there needs to be a user setting.
Archived
This topic is now archived and is closed to further replies.