Increase Data Drive Limit


Auggie

Recommended Posts

2 hours ago, Helmonder said:

ZFS is meh... At least for now.. It brings back several limitations that we currently do not have with unraid and that I really like not having... Like not deciding how back your pools are, no need to have same type/same size disks.. You also need an amount of RAM per TB in your server and that adds up quickly..

 

My info is from a few years back so stuff might be different..

From what I am aware there are still limitations with disk sizes and adding disks to a pool. unraid does have the nice feature of mixing disk sizes.

 

So if unraid added the feature for multiple arrays it would definitely be a pulling point for more people to use unraid.

Link to comment

You can right now setup an unraid vm on your unraid server, I've done it.  There are other threads that offer some good information on how to do this.  I spent quite a bit of time fussing around with it to get everything working correctly because I had issues other people probably wouldn't,  but at the very basic level I just passed a usb controller card and a sata controller card through to the vm.  I agree that Limetech making it possible to create 2 array's on the same server would be more elegant, but if you're looking for a way to expand past 30 drives on the same hardware right now, you could do this.

Edited by TSM
Link to comment
4 hours ago, TSM said:

You can right now setup an unraid vm on your unraid server, I've done it.  There are other threads that offer some good information on how to do this.  I spent quite a bit of time fussing around with it to get everything working correctly because I had issues other people probably wouldn't,  but at the very basic level I just passed a usb controller card and a sata controller card through to the vm.  I agree that Limetech making it possible to create 2 array's on the same server would be more elegant, but if you're looking for a way to expand past 30 drives on the same hardware right now, you could do this.

 

For me, I'm concerned about running an UnRAID-based media server in VM; it doesn't matter for my backup server.  All my videos are 1:1 imaged full Blu-Ray discs (which precludes running Plex or any other current media streamer) so anything that could impede performance and cause drop-outs, stuttering, or pauses in playback would be unacceptable.

Edited by Auggie
Link to comment
  • 3 weeks later...
On 9/13/2019 at 7:34 AM, Helmonder said:

ZFS is meh... At least for now.. It brings back several limitations that we currently do not have with unraid and that I really like not having... Like not deciding how back your pools are, no need to have same type/same size disks.. You also need an amount of RAM per TB in your server and that adds up quickly..

 

My info is from a few years back so stuff might be different..

Right, but I am just wondering if there would be a way to set up like a master and a slave unraid servers. It would integrate it all under a single SMB share, or something similar.

Link to comment
  • 5 months later...
On 10/8/2018 at 10:40 AM, bonienl said:

The limit is in the current size of super.dat, it can not hold more than 30 (=28+2) disk references.

Limetech would need to invent a solution which can handle a bigger super.dat file, while maintaining backward compatilibty.

And  - as you mentioned - the second parity disk which is the highest number in the array would need to change.

On top, several updates to the GUI need to be done to accommodate a larger array and a different parity2 assignment.

I'm just wondering if that's the limit now.  How is 45drives and Linus using more drives?   

Link to comment
8 hours ago, jonathanm said:

They utilize the cache pool or Unassigned Devices which doesn't use super.dat

but if they use unassigned devices - its like that drive isn't apart of the array or am I not following?  I swear I'm getting dumber as I get older. 

Edited by cbr600ds2
Link to comment
2 minutes ago, cbr600ds2 said:

but if they use unassigned devices - its like that drive isn't apart of the array or am I not following?  I swear I'm getting dumber as I get older. 

The parity array is just those disks in Main - Array Devices. That doesn't include any disks in Main - Cache Devices (cache pool) or Main - Unassigned Devices.

 

Only disks in the parity array have parity protection. Disks in the cache pool may have some redundancy independent of the parity array, depending on which btrfs raid configuration you have set up.

Link to comment

Disclosure

The upcoming version of Unraid supports multi cache pools, and allows the user to create as many cache pools as needed.

Each pool can consist of 1 up to 30 devices, and with a pro license, you are truly unlimited in number of devices to use.

 

Edited by bonienl
  • Thanks 2
Link to comment
10 minutes ago, bonienl said:

Disclosure

The upcoming version of Unraid supports multi cache pools, and allows the user to create as many cache pools as needed.

Each pool can consist of 1 up to 30 devices, and with a pro license, you are truly unlimited in number of devices to use.

 

When you say "cache pools," this doesn't mean the ability to have multiple arrays running simultaneously on a bare-metal server, no?

Link to comment
On 3/22/2020 at 10:18 PM, bonienl said:

Disclosure

The upcoming version of Unraid supports multi cache pools, and allows the user to create as many cache pools as needed.

Each pool can consist of 1 up to 30 devices, and with a pro license, you are truly unlimited in number of devices to use.

 

Can I configure a new cache pool to have an array protection identical to an array pool? Something like a dropdown selection "BTRFS RAID1, BTRFS RAID5, ..., XFS Unraid 1 parity, XFS Unraid 2 parity"?

 

When I saw that multi-pool picture in the 2020 news I hoped for multi-array pools. Never had multi cache-pools on my wish lists. Thought that new super(n).dat files with corresponding web pages and (n) Unraid processes running in parallel would be easy to achive 😉

 

Link to comment

Multi cache pools allows you to create multiple pools similar to what is used now for the cache pool with the same feature set (different raid configurations).

 

This feature is useful when you want to segregate devices for different applications.

E.g. a pool for docker applications, another pool for VM disks and yet another pool for downloads. All these pools can be configured independently.

 

More details will be given by Limetech when the next Unraid version becomes available (probably I already said too much 🙂)

 

It's a pretty nifty feature!

Edited by bonienl
Link to comment

With multiple cache pools, you can get faster SSD storage that is capable of having redundancy. There are lots of ways to use that.

 

Then just use the much larger, cheaper, and slower HDDs for archiving.

 

Just upsize HDDs instead of adding more. More disks requires more ports and other hardware, more license if you don't already have max, and each disk is just another point of failure.

 

I've never understood why some people have 20 or more 2TB disks in their array.

  • Like 1
Link to comment
10 minutes ago, trurl said:

With multiple cache pools, you can get faster SSD storage that is capable of having redundancy. There are lots of ways to use that.

 

Then just use the much larger, cheaper, and slower HDDs for archiving.

 

Just upsize HDDs instead of adding more. More disks requires more ports and other hardware, more license if you don't already have max, and each disk is just another point of failure.

 

I've never understood why some people have 20 or more 2TB disks in their array.

 

You are completely ignoring that there are other applications of unRAID that require the largest arrays possible, such as media servers.  Cache pools are of limited value in these setups.  You may not understand it, but that doesn't mean that there isn't a truly legitimate need for these types of arrays.

 

I welcome any new features to unRAID, including the increasing the number of cache pools available, as it expands unRAIDs capabilities for those in the mass market that could use them.

 

But the spirit of this particular thread is to encourage the expansion of the number of data drives unRAID is capable of incorporating into a protected array, which at the present, is limited due to how the super.dat file is formatted.

Link to comment
1 hour ago, Auggie said:

You are completely ignoring that there are other applications of unRAID that require the largest arrays possible, such as media servers.

Not ignoring anything of the sort.

 

Media server was the primary audience for Unraid from the beginning, and probably still is. I don't even have any VMs and most of my dockers are about media.

 

You can have a lot of media storage without having a lot of drives since drives are so big today. That was my point.

  • Thanks 1
Link to comment

I'm looking forward to the multiple cache pools. 

 

I'm currently running my VM's and Docker containers off unassigned devices because my cache drive exclusively handles the decompression on incoming downloads. Even with an NVME drive, I get media playback issues if I decompress incoming downloads while streaming, and that's with CPU pinning. The drive is the bottleneck. 

 

Having multiple cache pools will make life so much easier especially for docker apps that don't like running off any directory that isn't /mnt/user like musicbrainz.

 

 

As for larger arrays with better redundancy; you could run unRaid on ESXi or Proxmmox and set up ZFS pools to pass through as virtual drives to unRaid. That way you can haz all the storage in raidz (see what I did there) and still use the unRaid array in the way you're used to.

The only downside with something like ZFS is that it's best to start off with all your storage drives and not adding single drives over time. 

I also have no idea if you can pass something like a GPU from say Proxmox to unRiad and then from unRiad into a docker/VM, and whether there's a performance penalty for doing so.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.