Pool selectable for secondary storage (pool-to-pool mover)

[CanOfSocks]

With the changes coming up in 6.12 around how pools are displayed, it would be great if we could use pools like the main array.

 

For example, I have a separate pool for my torrents, where I can use cheap disks that don't need a parity without needing to exclude disks and works quite well. I would love for this pool to be able to use an SSD cache so I can move things off my pool quicker, then the mover can move them to normal bulk storage for seeding down the path.

 

The way pools and the array are presented on the webui make it seem like this may be able to work with a little modification, but I'm no developer. This should be disable by default if implemented, like exclusive access is right now, to prevent issues with newer users who don't need this feature.

It could also be an option to hold over until multiple arrays are available in unraid in the future.

 

 

[itimpi]

I believe that something like this is planned for the 6.13 release.

[Takiyon]

Hey everyone! So, I bought unRaid pro two weeks ago. Still in the process of migrating all my data from Open Media Vault which I have been using for years. I chose unRaid because of the ease of VM use as well as the community plugin support which is a huge selling point IMO. Anyway after trying the setup with a regular array. It was too slow for me. Nothing wrong with it but it is just not good for my main server use case at this time. So I set up my server with 3 Pools.

 

6 x 8TB RAIDZ1 Pool - Named "Alpha"

3 x 8TB RAIDZ1 Pool - Named "Bravo"

3 x 8TB BTRFS Pool - Named "Charlie"

2 x 2TB BTRFS SSD Pool - Name "Cache"

 

My issue right now is that it is my assumption that with unRaid using MergerFS "All" files regardless of where they are consolidated under /mnt/user. This is great and works fine since it would be a single path for any of my dockers to point to. Everything is working great right now except for the following, and I am not sure if this is intentional or maybe a feature that can be implemented.

 

Basically if I create a share called "My Files" and set the primary storage to pool "Alpha" I should be able to set the secondary storage to "Bravo". Just like if I set my primary to "Cache" and secondary to "Array". Right now shares created on one pool are strictly for that pool. It would be great if the Pools themselves were treated as single disks in this manner so that if "Alpha" fills up the mover process can automagically move stuff to "Bravo" or any other 2 pools that I set. Or have some kind of "Pool Balancing" to keep equal space on both or all pools. If this is not possible then I would request that, as a feature to be able to shut off MergerFS. This way /mnt/user is not used, and all of my paths are set explicitly avoiding confusion as to where the files are. This will also allow to spin down pools that are not being used as well.

 

Also it would be great if we did not need the "Array" portion of the system. Right now I am using one SD card just so the system can start the "Array" and not complain but it is never used. Not in this case anyway as my main server. I plan on buying another 2 or 3 unraid licenses for other use cases where this would make sense. But I do think as use cases expand for unRaid certain features may need to be turned off or be able to be accommodated. It opens up other uses and other markets where one would not otherwise be. Making more $$$ for LimeTech and their great software. 🙂

Edited July 25, 2023 by Takiyon
Added the last part to finish.

[itimpi]

2 hours ago, Takiyon said:

Basically if I create a share called "My Files" and set the primary storage to pool "Alpha" I should be able to set the secondary storage to "Bravo".

 

This is a feature that is meant to be coming in a future release although I have no idea of the timescales.  There were changes made in the 6.12.x releases to start getting ready for this.

 

2 hours ago, Takiyon said:

Also it would be great if we did not need the "Array" portion of the system. Right now I am using one SD card just so the system can start the "Array" and not complain but it is never used

This is likely to happen in the same timescale as the above feature.   In the 6.13 release we have been told that the aim is to make the current Unraid 'array' just another pool type and then you can have any mix of pool types (including multiple Unraid 'arrays' if you want.

 

However having said that until it actually happens we cannot be certain when it will happen or if Limetech will deem some other mix of features to be higher priority.

 

[Takiyon]

On 7/25/2023 at 5:19 PM, itimpi said:

 

This is a feature that is meant to be coming in a future release although I have no idea of the timescales.  There were changes made in the 6.12.x releases to start getting ready for this.

 

This is likely to happen in the same timescale as the above feature.   In the 6.13 release we have been told that the aim is to make the current Unraid 'array' just another pool type and then you can have any mix of pool types (including multiple Unraid 'arrays' if you want.

 

However having said that until it actually happens we cannot be certain when it will happen or if Limetech will deem some other mix of features to be higher priority.

 

Awesome! As I slowly migrate all my data over to my 2 ZFS pools. I had another good idea where datasets could be duplicated across pools. I guess this would work well with the idea of treating each pool as a single disk to be merged with the existing FuseFS system. So from the GUI, if I make a dataset called "movies". It should be automatically create that dataset on any pool that is part of the file system to keep FuseFS consistent. This way the mover can balance the storage pool space. I am sure as I use the system there will be other ideas but this is just one of them that is on the top of my head.

[flex420]

this is def needed for me since i want to use a striped zfs mirror with 4 drives as my main array and docker etc to be on the cache nvme's 

 

[semtex41]

I would also find this interesting. Like putting a nvme nfs raid0 as primary and a raidz2 pool as secondary. Finally, adding the md array as tertiary. 

 

I dont know what the multiple zfs pool overhead would be, but I like this idea. 

 

 

[Mustafa Aydin]

I would prefer this way too. Have a raidz1 with 6 drives. They can sleep hole time then and only awake if something is missing on the large cache drive. Maybe with different mover timeframes to adjust how offen the cache will be emptied.

 

[Diego Spinola]

Anyone knows if it's really coming ? this would be AWESOME for my new workstation... my power hungry zfs pool (10x6TBs SAS) could use some rest

[Antiq]

If so, for example, I can have a pool of ZFS raidZ of 3 HDDs, R/W at around 400MB/s, as Secondary Storage; then a Cache pool of ZFS raid0 of 2 NVME Drives at a 3000MB/s.

it would be possible for the HDD ZFS pool to use NVME Cache Pool for a even better performance than the original HDD ZFS Pool.

And in the other hand, this will benefit the ALL-SSD configures, with Slim-but-fast NVME drives as Cache, and Mass-but-slow SSDs at ZFS raidz for storage.

[ChatNoir]

It is already in the works.

Should be available for 6.13 or 6.14.

[JustOverride]

On 7/25/2023 at 6:19 PM, itimpi said:

 

This is a feature that is meant to be coming in a future release although I have no idea of the timescales.  There were changes made in the 6.12.x releases to start getting ready for this.

 

This is likely to happen in the same timescale as the above feature.   In the 6.13 release we have been told that the aim is to make the current Unraid 'array' just another pool type and then you can have any mix of pool types (including multiple Unraid 'arrays' if you want.

 

However having said that until it actually happens we cannot be certain when it will happen or if Limetech will deem some other mix of features to be higher priority.

 

 

🙃

I thought this was implemented but when I went to actually use it today I see that it isn't. I guess we wait now.

[mythic]

I have yet to try Unraid so im not entirely sure what im saying has bearing, building my server right now. But as i have read mover can be made to move all files and files of a certain age when a fill rate of the drive is reached. My whish would extend i bit more than just choosing Array A move to B if % fill. But also Move A to B and B to C if % fill until % fill is reached starting with last used files. If this was possible i could have Fast nvme raidz array A slower sata ssd raidz array B and even slower hdd raidz array C. And the files would drip down based on use, and not move back and forth like a read/write cashe, killing the ssds. So if like array is 90% filled then empty to 80% fill. And so forth. That would be perfect and i would not have to manually monitor what needs to be placed where.