Hello,

 

I have a parent folder which has 8 folders in it, and each of those has other folders and files underneath (named contents in this example).

I have two 2TB disks that I want to split the data across using high-water allocation method, and they have been selected as included disks in the share.

 

Parent

   Folder 1

      Folder 1 Contents

   Folder 2

      Folder 2 Contents

...

   Folder 8

      Folder 8 Contents

 

All folder 1 contents should remain under folder 1, folder 1 can reside on either disk.

All folder 2 contents should remain under folder 2, folder 2 can reside on either disk.

...

All folder 8 contents should remain under folder 8, folder 8 can reside on either disk.

 

(But in addition to the above) when copying the parent folder to the share, it splits the data equally between the two disks that are selected as included i.e folders 1-4 go to first disk, folders 5-8 go to second disk   OR   folders 5-8 go to first disk, folders 1-4 go to second disk.

 

 

Q1. When using high-water, does the first disk have to reach half full before it will write anything to the second disk?

Q2. Is it possible to get the 8 folders equally distributed across both disks if the total size of the parent directory is smaller than half the size of one of the disks?

Q3. How do I copy the data across to prevent folders all being created on one disk, and it copying everything to only one disk?

Q4. How should my split level setting be set to achieve the structure proposed?

 

Any help is much appreciated

Thanks

 

Edited November 16, 20241 yr by Dominoes0522

?lets a review with some FAQs..
https://docs.unraid.net/unraid-os/manual/shares/user-shares/

To achieve the desired structure and behavior in Unraid with the High-Water allocation method, here's how you can address each of your questions:

 

Q1. When using high-water, does the first disk have to reach half full before it will write anything to the second disk?

Yes. High-Water allocation works as follows:

1.It starts by filling the disk with the most free space until it is half full.

2.Once the first disk is half full, it moves to the next disk with the most free space and starts filling it until it's half full.

3.This process continues with progressively smaller thresholds (e.g., 25%, 12.5%, etc.).

So, the first disk must reach half its capacity before data is written to the second disk.

 

Q2. Is it possible to get the 8 folders equally distributed across both disks if the total size of the parent directory is smaller than half the size of one of the disks?

Not directly with High-Water alone, as it prioritizes filling disks based on the above rules. If the total size of the parent folder is smaller than half the size of one disk, High-Water will likely write all the data to the first disk.

To achieve an even distribution:

You can manually copy the folders to each disk using tools like Midnight Commander or the Unraid terminal, bypassing the allocation method.

Alternatively, set a custom split level and use the "Most Free" allocation method (explained in Q4).

 

Q3. How do I copy the data across to prevent folders all being created on one disk, and it copying everything to only one disk?

To prevent all data being written to one disk:

1.Set the split level correctly to control folder hierarchy (explained in Q4).

2.Ensure "High-Water" or "Most Free" is selected as the allocation method in your share settings.

3.Verify that both disks are included in the share's configuration.

4.If needed, manually move some folders to the second disk using the Unraid terminal or a file manager after the initial copy.

 

Q4. How should my split level setting be set to achieve the structure proposed?

The split level controls how folder structures are maintained across disks. In your case:

Split Level Setting: Set it to 2.

 

Why Split Level 2?

Level 1: "Parent" stays on the same disk.

Level 2: Allows "Folder 1", "Folder 2", etc., to reside on different disks, but keeps their contents together on the same disk.

This ensures that the contents of each folder (e.g., Folder 1 Contents) remain on the same disk as their parent folder (e.g., Folder 1), while allowing the main folders (Folder 1, Folder 2, etc.) to be split across disks.

##################

 

Steps to Implement:

Go to Shares in the Unraid web GUI.

Select your share or create a new one for the "Parent" folder.

Set the following:

Allocation Method: High-Water (or "Most Free" for even distribution if preferred).

Split Level: 2.

Include Disks: Select your two 2TB disks.

Exclude Disks: Leave blank (unless you want to exclude specific disks).

Copy the data to the share.

Additional Tips:

If High-Water doesn't distribute the folders as desired, you can override it by manually copying some folders to the second disk via the terminal or Midnight Commander (mc).

Ensure that your disks have sufficient free space before starting the copy.

 

Edited November 16, 20241 yr by bmartino1

Thanks so much for taking the time to provide such a detailed response - very helpful and much appreciated!

 

It leads me to a few more questions:

 

Re. Q1)

1 hour ago, bmartino1 said:

It starts by filling the disk with the most free space until it is half full.

Does this take into consideration the included disks selection? So if disk 3 & 4 are the only included disks in the share, it will only look at those disks to find the disk with the most free space?

 

Re. Q2)

1 hour ago, bmartino1 said:

To achieve an even distribution:

You can manually copy the folders to each disk using tools like Midnight Commander or the Unraid terminal, bypassing the allocation method.

Lets say I decided I wanted to always manually move and copy files to the disk(s) of my choice, would it be a better approach to create disk shares rather than user shares? So rather than adding files/folders to a user share which may end up on either disk depending on configuration, I could just add them directly to the disk I want them to be on? And would there be any drawbacks to this approach?

 

Thanks!

1 hour ago, Dominoes0522 said:

Thanks so much for taking the time to provide such a detailed response - very helpful and much appreciated!

 

It leads me to a few more questions:

 

Re. Q1)

Does this take into consideration the included disks selection? So if disk 3 & 4 are the only included disks in the share, it will only look at those disks to find the disk with the most free space?

 

Re. Q2)

Lets say I decided I wanted to always manually move and copy files to the disk(s) of my choice, would it be a better approach to create disk shares rather than user shares? So rather than adding files/folders to a user share which may end up on either disk depending on configuration, I could just add them directly to the disk I want them to be on? And would there be any drawbacks to this approach?

 

Thanks!

 

Re. Q1: Does High-Water Allocation Consider the Included Disks?

Yes, the High-Water Allocation method will only consider the disks specified in the "Included Disks" setting of the share. If disks 3 and 4 are the only included disks for that share, the High-Water logic will only apply to those two disks. It will:

1Start by filling the disk with the most free space (among the included disks) until it reaches half of its capacity.

2Then, switch to the next disk with the most free space and fill it up to the same threshold.

3This process repeats for progressively smaller thresholds until all included disks are full.

Key Point: The Excluded Disks setting will also restrict the disks used for the share, ensuring that High-Water ignores those disks entirely.

 

Re. Q2: Manually Managing Files with Disk Shares vs. User Shares

Disk Shares vs. User Shares

Disk Shares: These provide direct access to individual disks (disk1, disk2, etc.). When you manually move or copy files to a specific disk share, you have full control over which disk the data is stored on.

Pros:

Absolute control over file placement.

Avoids any ambiguity about which disk a file resides on.

No need to worry about split levels, allocation methods, or included/excluded disks for that share.

Cons:

If you inadvertently place related files across disks (e.g., parts of a movie folder), this can cause potential issues with file access if you later switch to User Shares.

Disk shares bypass the User Share allocation logic, which could lead to uneven disk usage if not managed carefully.

Harder to manage at scale for large shares with frequent changes.

 

 

User Shares: These are logical shares that span multiple disks based on allocation methods, split levels, and include/exclude settings.

Pros:

Simplifies file management by abstracting physical disk locations.

Dynamically distributes files across disks according to the configured rules.

Works well for large shares with automated or network-based data transfers.

Cons:

Less control over which disk the data is stored on unless you manually configure split levels and allocation.

Requires proper configuration to prevent unintended data distribution.

 

When to Use Disk SharesL

 

Use Disk Shares if you want total control over file placement, for example:

Archival data where you want specific folders/files on specific disks.

Scenarios where the data structure needs to align perfectly with your backup strategy.

 

When to Use User Shares

Use User Shares when:

You don't need granular control over file placement.

You want Unraid to handle file distribution automatically, especially for media libraries like Plex or file servers.

You plan to expand your array or frequently add/modify data.

 

Drawbacks of Disk Shares

Potential for Split-Level Confusion: If you manually place files in a disk share and then try to access them via a user share, you might encounter issues if split levels aren’t configured correctly. For example, files spread across disks may result in incomplete directories when accessed via a user share.

Less Flexibility: As your storage needs grow, manually managing file placement on specific disks can become tedious.

User Share Conflicts: If you're using both disk and user shares for the same data, it can cause problems due to Unraid's "User Share Copy Bug." This happens when you copy/move files between a disk share and its corresponding user share, leading to potential data corruption or duplication.

Best Practice

If you're committed to manually managing file placement and don't need the flexibility of User Shares, Disk Shares can work well.

However, a safer and more efficient approach might be to continue using User Shares with:

Split Level set to control folder hierarchy.

High-Water Allocation to balance disk usage.

Manual file movement only when necessary (e.g., via Midnight Commander or rsync).

Once again - thank you for your thorough explanation - this has cleared up a lot of my doubts.

 

I do however have another question or two (sorry)

 

When my drives were in my windows desktop everything was done manually so I'm used to that way of working.

 

I'm still unclear about what benefits of user shares I would be missing out on, other than that it distributes files across disks. You would still need to copy or move files or folders to the user share, so you're either copying or moving to a user share or a disk share - it's more that the user share controls file and folder placement on the array automatically based on settings, whereas disk share is just specifying which disk your data goes to (pretty much the same as if your saving files to a disk in windows). Does that sound correct?

 

You also mentioned about user shares being beneficial when you want to expand your array or frequently add/modify data. Could you explain why as I'm unsure on this.

 

Thanks!

 

Hope it doesn't confuse you ... advance linux stuff think windows shortcuts...

so lets define some linux terminology:
A symbolic link, or symlink, is a file in Linux that points to another file or folder on the system. Symlinks are similar to Windows shortcuts and can be useful for quickly accessing files or folders with long paths.


it has to do with how unraid uses what is called a "FUSE" and info on symbolic links.

the pathing for /mnt/user is the FUSE system and user share setup.

the WebUI shares tab will make fuse sym links to the data on the disks 

where as a disk share and data exist and is the file on the disk.

The fuse symbloci link is a temp file that is used. simlar to when you make a shortcut on the desktop in windows it llinks to the actul location.
this is what /mnt/user deos via user shares...

depeding on setting such as hi water/split... dictates where the file data is and how the "shortcut/sym linking works and where the actual data is located. This is beneficial as the data can be ditacted via how the sym link is made between the files and wher teh file is located.

It the question of "who touched what where and what touched what where". This is answering how we are touching what where and how we want it to contuse touching what where ....

docs, youtbe videos such as spaceinvaders... I have found that it is better to use Unriads Fuse sytem with /mnt/user....

Summary:

Disk shares offer more control, which might appeal to you if you like managing your storage manually, like on your old Windows system.

User shares automate the file placement process and are ideal if you want to simplify storage management, especially as you expand your array or deal with frequent data changes.

To generalize review:
User Shares vs. Disk Shares: Core Differences

1.Automation and Simplified Management:

With user shares, you don't have to think about which physical disk to use when saving or organizing your files. Instead, you just specify a share (e.g., "Movies" or "Backups"), and Unraid handles where the files are placed based on the share's settings.

Disk shares, by contrast, are more manual. You need to specify exactly which disk a file goes to, which is similar to saving directly to specific hard drives on a Windows system. While this offers control, it can be tedious when managing a lot of files or expanding your storage.

2.Dynamic Growth:

If you’re expanding your array or frequently adding/modifying data, user shares can be a big time-saver. Here’s why:

When you add a new disk to your array, Unraid can automatically include it as part of your user shares. This means any existing user shares can now use this new storage space without you needing to manually change anything.

For disk shares, when you add a new disk, you need to decide if and how you want to start using it, and you might need to manually adjust how you store files.

3.Distribution of Data:

User shares also allow you to control how data is distributed across disks through settings like "Split Level," "High-Water," and "Allocation Method." This can help prevent certain disks from filling up while others are mostly empty, which can help keep the array balanced and optimize storage use.

With disk shares, you have to decide this manually, which might lead to some disks filling up quickly, while others are underused.

Examples of User Share Benefits:

Expanding Your Array: Suppose you add a new disk to increase your storage. With user shares, this additional space automatically becomes available to all the shares configured to use it. You don’t need to manually manage where data should be saved—you just keep using your shares as usual.

Ease of Adding and Modifying Data: If you’re frequently adding or modifying files (e.g., downloading new media or making backups), you can just place everything into the user share without having to think about which disk has enough space or where each file should go.


review:

 

also may help
review:
 

 

All makes perfect sense and I'm starting to see why user shares are more flexible.

 

The main reason I'm hesitant about user shares at the moment is I'm new to Unraid and I find the split level and allocation method settings a bit confusing, especially because a lot of my folders are not organised with common hierarchy. Also the transfers of data I've done with user shares (using Krusader) have either put all the files on one disk or have given unexpected results. I expect that this is just due to lack of understanding & experience so will persevere and keep experimenting.

 

Thanks again for your time and very helpful responses. 

On 11/17/2024 at 7:04 PM, Dominoes0522 said:

Also the transfers of data I've done with user shares (using Krusader) have either put all the files on one disk or have given unexpected results.

Typically that's because file transfers often create and populate folders before starting actual file transfer.