One thing that bugs me about highwater is that once your NAS start getting full and your dives are down to a similar amount of space on each and assuming you have not excluded disks it causes a write to almost every drive. This means that you access a share and basically your whole NAS spins up. Once you add a new drive you have the chore of manually recovering from this situation by merging lots of folders on lots of disks onto the new drive.

 

I understand why this happens as its essentially what it is designed to do but I think there could/should be some break point added where the impact of this is limited.

 

Open to ideas.

Yes I feel the same.

 

I hope there is a feature/option like overflow when the drive is become full before it will go and write to the next share drive so it won't write on every single drive we share.

There needs to be some logic improvement that says something like if drive free space within xx% of each other then use the drive with the most free until xx% full.

 

Otherwise what you need to do is always have lots of free disk space.

 

The main issue i have is that recovering from this scenario means manually moving files between disks and to do that you need to manaully decide what disk should get the data copied to it. Its long slow boring and frustrating as unRAID could undo your work at any time.

I think a sequential highwater mark is a logical way to do things.

When a drive reaches it's highwater mark, let's say 90%, it should go to the next drive.

 

I'm not too fond of the current round robin mechanism.

 

I strongly agree with 2 posts above me.

 

Is there any way can this be done?

I strongly agree with 2 posts above me.

 

Is there any way can this be done?

 

Not if we continue this conversation in this forum.  I doubt if lounge material makes much impact on the product release schedule.

 

 

Bill

In reality I don't think it makes much difference where we talk since nothing have be added to the laundry list in 7 months.

In reality I don't think it makes much difference where we talk since nothing have be added to the laundry list in 7 months.

 

I don't understand your point, but my point is that the ideas are better expressed in the right forum.  The Lounge is fine for off-topic chats, but when they become something more (IMO) useful as this one, we should put them in the Features Request section.

 

 

Bill

Im sure Tom will move it to another forum if he feels it worthy. I cant see him ignoring something arbitrarily just because it in one forum or another.

 

As for my point it is quite clear: nothing is being added to the laundry list regardless of where it is posted or how much it is discussed. But we are getting completely off topic.

I was also thinking that the allocation algorithm should favour spun up disks as well. No point spinning up a disk just because it got 100MB more free than one that is already up and running.

All good suggestions, but they may have more visibility in the correct forum.

I plan to ask for the thread to be moved once Tom comes back.

 

My main focus here is to try and identify a way to clump layer 2 folders to single disks as much as possible.

 

So for instance with //TOWER/TV/TV - A/ the last folder deep should try as much as possible to clump onto one disk. This more naturally favours the multimedia slant of unRAID users. e.g in this example users are more likely to watch two items starting with A in a row than A then Z (notice I am saying likelihood not a certainty).

 

The scenario where by 10 disks spin up so that you can watch 5 episodes of a show needs remedied.

My main focus here is to try and identify a way to clump layer 2 folders to single disks as much as possible.

 

So for instance with //TOWER/TV/TV - A/ the last folder deep should try as much as possible to clump onto one disk. This more naturally favours the multimedia slant of unRAID users. e.g in this example users are more likely to watch two items starting with A in a row than A then Z (notice I am saying likelihood not a certainty).

 

I'm assuming with the above structure you have split level set to 2.  This would allow any object created in the TV-A directory to be allocated to any of the disks (according to allocation method).  This is what you want so that you don't run out of storage, but on the otherhand, it could cause 'episode files' in the TV-A directory to eventually get spread among the disks.

 

I think a solution to this problem is to set the split level to 1 (in conjunction with a small change in the software).  With split level set to 1, at the time you create the TV-A directory some disk will get chosen; and, then any 'episode' files will get created on the same disk as the TV-A directory exists on.

 

This is fine until that disk runs out of space.

 

So the change in the software is this: if space has run out on a disk, then regardless of split level, choose a new disk to create the object on.  To implement this, we would need to add a "Min free space" config value for each share.  If the actual amount of free space on a share is less than this 'Min free space' value, then we would treat it as having 'run out of space'.

 

Does this make sense?

 

It does indeed. I have delayed posting as I have a couple of thoughts that i haven't fully formed yet. Well actually i have had many and I have shot them down myself  asbeing overlly complicated so I want to think about it some more and try and post something specific.

It does actually make sense. Giving a user to have an option on how much minimum storage space before writing to the next disk share would be an optimal choice. Redifying highwater mark solution by using all storage space of the first drive before going to the next is a great solution.

 

So yes I would like to see implementation of this "Min FREE Space" or % of run out of space.

So the change in the software is this: if space has run out on a disk, then regardless of split level, choose a new disk to create the object on.  To implement this, we would need to add a "Min free space" config value for each share.  If the actual amount of free space on a share is less than this 'Min free space' value, then we would treat it as having 'run out of space'.

 

Does this make sense?

 

 

A little clarifying please.  Above you mention space on a disk, then you mention amount of free space on a share.

Do you mean free space on a disk within the share?

 

In any case, what I'm gathering is that when free space falls beneath a user defined level, no matter what, a new disk within the share is chosen for output.

 

What I think I'm missing is to use the disks in a sequential order rather then round robin order.

I.E. I want to limit all writes in a split level to the first disk, until it reaches the "min free level", then choose the next disk.

I guess it's how it choose the next disk which I'm trying to see what is people want.

 

I'm under the impression the current environment chooses the most free disk.

and I've seen requests for the environment to choose the next disk in a sequential order. As in overflow.

 

So Perhaps logic choices would be based on MIN FREE value.

1. Choose most free at split level.

2. Choose next sequential at split level.

 

In a perfect world the logic should probably be "Use the disk that already has the most data at this split level. When you reach a % remaining free on disk move on to the next disk that already has the most data at this split level. If there is no data on this split level then start with the disk with the most free space."

 

The problem is that this logic breaks the fundamental design requirement of not spinning up disks that it doesn't need to. If also is clunky needing space calculations done to choose a disk. As such it isnt workable.

 

The solutions presented so far DO solve the highwater limitation that causes lower split level data to be added to many disks in certain scenarios so that is a boost but perhaps if we keep talking we can enhance it even more.

I am guessing a lot of people are new to UnRAID and perhaps building a server from ground up. By that said everyone is using free version and some smaller hard drive to try it out, and when satisfied and confident how the basic unRAID works - that's before they decide to buy plus or pro version. Also not all of us can afford to buy up 6-16 new hard drive right away - which means we buy HDD when we almost run out of space or if we find good price of HDD.

 

UnRAID is a true incremental storage so it make more sense to choose sequential disk overflow (on disk share) so you exactly know which disk is almost full and which one is still empty or have most storage space.

 

I am not sure how everyone start their unraid but I started with just 500GB(parity), 200GB and 80GB then as I become more confident I start changing each drive to the next biggest I could afford started of course with parity drive, then the parity drive(500GB) would replace 200GB and 200GB would replace 80GB. So my point is - in my case the smallest drive would always be the last one. Future target would be the maximum 16X1TB; 1TB increments. Lesser disk will spin as Videos/Movies have separate disk share and Data, Photos and Music have their own disk or disk share anyway.

 

When disk space runs out (MIN FREE space less or equal to set value) overflow should be in sequential order of disk share.

 

 

What I think I'm missing is to use the disks in a sequential order rather then round robin order.

I.E. I want to limit all writes in a split level to the first disk, until it reaches the "min free level", then choose the next disk.

I guess it's how it choose the next disk which I'm trying to see what is people want.

 

I'm under the impression the current environment chooses the most free disk.

and I've seen requests for the environment to choose the next disk in a sequential order. As in overflow.

 

 

Agreed. It would be nice to fill disk 1 to 90%, then roll to disk 2 to 90%, then disk 3, etc.  However, there does need to be some logic to check the available amount of free space in relation to the amount of data being copied.  For example, I just copied a folder of size 77 GB+.  Because of my split level limitation, this folder should not span disks. 

 

 

What I think I'm missing is to use the disks in a sequential order rather then round robin order.

I.E. I want to limit all writes in a split level to the first disk, until it reaches the "min free level", then choose the next disk.

I guess it's how it choose the next disk which I'm trying to see what is people want.

 

I'm under the impression the current environment chooses the most free disk.

and I've seen requests for the environment to choose the next disk in a sequential order. As in overflow.

 

Agreed. It would be nice to fill disk 1 to 90%, then roll to disk 2 to 90%, then disk 3, etc.   However, there does need to be some logic to check the available amount of free space in relation to the amount of data being copied.  For example, I just copied a folder of size 77 GB+.   Because of my split level limitation, this folder should not span disks. 

 

Actually that is already available:

 

"Most-Free

In this method, the system will simply pick the disk which currently has the most free space. "

 

and is almost exactly the opposite of what we are trying to acheive here. Let me explain.

 

This method suits users who add more disks over time however it would also guarantee that if you kept adding data to a share say //tower/tv/a/ that after a long period has passed and lots of data was added browsing to this share would spin up lots of disks.

 

This is exactly what we are trying to avoid by coming up with a new algorithm or an enhancement to Highwater that clumps data of the same split level and folder to the same drive. The result of this is that if you went to //tower/tv/a/ the minimum number of drives would spin up. This is better for heat, speed and actual real world usage for some users folder layouts.

 

So what we are really looking for is a way to identify the best drive to add to based on the split level and folder having the most data there already,  balanced with how much space is available on that drive.

 

Toms idea should defnately be implemented as it is a marked improvement. I think we can keep going and tweak it more.

I've suggested here (especially the last 3 paragraphs) an alternative way to handle this.  I called it Ordered, but Sequential would be fine too, except it would be nice to have the option to specify the order.  In particular, if you wanted to use one of your drives as a general overflow drive, then you might specify it as the last Included disk in *each* of your Ordered shares.  This would allow you to setup your storage shares with this 'overdraft protection', and if you discovered your DVD's were starting to save to the overflow drive, you would know it is time to add another large drive to the DVD share.  You would insert it before the overflow drive, and manually move the overflow files to the new drive.

 

The ability to specify the Minimum Free Space for each share would be a nice enhancement.

So then there is.

 

1. MOST Free selection of share among drives in split level allocation. (current method)

 

2. A Request for MIN FREE value for defining when to select the next drive at a split level within a share.

 

3. ORDERED Selection of next drive in split level allocation. (using INCLUDED disks as sequential order).

 

I don't quite see the need for an overflow disk if you have ordered selection of the next drive.

 

 

 

 

 

Sounds good to me. I would suggest we cobble up a quick script that takes an input of a split level and path e.g. \TV\A\ and summarises the current allocation of space.

 

i.e.

 

showsplit \\tv\a\

disk1 13GB

disk2 1GB

disk3 0GB ...

...

diskxx xxGB

 

we could even have it recommend a disk order by simply sorting the amount on each disk.

 

This would also help users to recover from the current highwater problem.

 

 

I had another thought that could be a superb and simple optional improvement.

 

"Prefer to use spun up drives"

 

Thats it Very simple but the implications are profound. Disks will stay spun down much longer, delayes to spin up drives before a write will disappear in many cases.

 

Thoughts?

I had another thought that could be a superb and simple optional improvement.

 

"Prefer to use spun up drives"

 

Thoughts?

 

Just the be clear...

The space allocation on writes is...

 

Select the "most free space drive"..  that is currently spinning.

 

Or do we still want to have

 

Fill drives in sequential order of listing. (Yet prefer drives that are currently spinning).

 

I'm still a fan of sequential order up to high water limit.

I find myself manually doing this.

High water limit just isn't doing it for me.