What's the veredict on using ZFS on array disks? does it kill hd-idle / power efficiency?

[TheLinuxGuy]

There's this discussion on reddit suggesting that using ZFS on array disks may cause the disk to never go to sleep (random reads, seeks?), and that XFS or other filesystems are better at power efficiency. How true are those statements?

 

This post from July 24, 2023: https://unraid.net/blog/zfs-guide - there is a confusing mention

For NON-ZFS filesystems, a Pros:
* "The regular array offers excellent power efficiency as idle disks can be powered or spun down. So, if a movie is playing from disk 1, all the other disks, if not being used, can be spun down."

 

But then when it goes into "Hybrid array" which is basically an array disk zfs formatted it says "Idle disks can be powered down to conserve energy when unused."

This seems to be contradicting the redditor's post - so looking to gain some insights into people's observations when using hybrid/zfs array disks and if they do spin down as expected, like any btrfs or xfs array disk. thanks!

Edited August 9, 2023 by TheLinuxGuy

[MAM59]

there are seperate setups to discuss. The problem is that UNRAID uses an "Array" and ZFS also can build and use Arrays. They are totally different.

 

To make things worse you could use ZFS-Array(s) inside an UNRAID Array.

 

So it often happens that two people talking about "Arrays" have a different thing in mind and therefore the given statements are invalid for the other one.

 

They (UNRAID) are trying to move away from that problem by changing the phrase "Array" into "primay storage" (which seems to confuse even more people).

 

Note that there is a serious performance problem with "single-ZFS-Filesystem-within-UNRAID-array". Not everybody faces it, but when it hits you, you surely will fast move away from ZFS again...

(in general, ZFS and UNRAID make no sense together. Either I want flexibility in growth (Unraid), or I like to have data safety beeing forced to use always the same disk type (ZFS-RAID)

I've tried single ZFS within the array (BAD, VERY BAD!, some say it works well, my experience was really different) or in a "pool" (now called "secondary storage"). Both turned out to have an indiscussable write performance (<70MB/s, 220MB/s should be normal here).

ZFS contains a read cache (that UNRAID is badly missing), I've spend 96Gigs of RAM for it, the result was not really convincing too.

But, as I said, SOME seem to be happy with ZFS... feel free to try.

 

(the new names "primary" and "secondary" storage were formed after the "where new files go first" scheme. Before it was "cache" and "array")

[Frank1940]

4 hours ago, MAM59 said:

(the new names "primary" and "secondary" storage were formed after the "where new files go first" scheme. Before it was "cache" and "array")

 

I don't know if it is really that clear yet as this 'new' usage of terms is concerned.  I point out this section of the release notes in the Documentation:

 

     https://docs.unraid.net/unraid-os/release-notes/6.12.0#share-storage-conceptual-change

 

This are a couple of screen shots from my 6.12.3 Unraid server:

and

 

I will point out that both of my servers are basically NAS setups with a single cache drive as the Primary storage and an Unraid array as the secondary storage.  I seem to recall that they decided not to change the notation of 'Cache' and 'Array' for existing setups.   I can't recall what was done for those more complex setups having multiple drives in ZFS and BTRFS configurations.

 

It is my opinion that this mixing of terminology is going to be confusing---  at least in the short term and to the casual/new user of Unraid.  It might have been preferable to change the designation for the 'Unraid Array' to, say, 'UnArray'.   One thing that needs to be done (In my opinion) is for @limetech  (or one of the Unraid staff) to generate a table of terms that they are going to be using to describe/define all of the various configurations and add as a section in the Documentation.  I really don't care what things are called as long as I know exactly what the term defines.

Edited August 9, 2023 by Frank1940

[itimpi]

One thing that can be done now is start thinking of ‘cache’ as a capability of a pool rather than a pool type.    It is not helped by the fact that at the mome t the default name for a pool still seems to be ‘cache’.   Perhaps it should be changed to ‘pool’ or somethibg like that.    
 

I do agree that a better name for the Unraid array that is more type oriented (such as the UnArray’ suggestion, or perhaps simply ‘hybrid’) would be a good idea.   It will become much more important when we can have multiple arrays of the type of the current main array, and everythibg starts getting treated as a pool.

[Frank1940]

41 minutes ago, itimpi said:

I do agree that a better name for the Unraid array that is more type oriented (such as the UnArray’ suggestion, or perhaps simply ‘hybrid’) would be a good idea.   It will become much more important when we can have multiple arrays of the type of the current main array, and everythibg starts getting treated as a pool.

 

One thing that could be done, is for the Unraid developers to take some time to think about how they see Unraid developing in the next few years.  Then set down and lay out a chart of how they visualize things being organized when this long range outlook has been completed.  At that point, they should be able to tell us how things will be organized now and in the future.  (Perhaps, everything will be a pool.  Perhaps, they see being able to setup a pool within a pool.  Perhaps, a pool might be assignable as 'Primary' or 'Secondary'.  Perhaps, some types pools might only be be assigned as either 'Primary' or 'Secondary'.)   There should consideration to making this overview flexible enough so that some future storage scheme could be integrated into it without requiring a complete revision of how things are organized in the bigger picture.  They might also look at how VM's will be allowed to access storage including how isolated this access should be.  (I can see the case of some folks wanting a VM to only have access to virtual storage so that the VM is completely isolated from the underlying hardware.  Other might want a VM to have seamless access to any of the hardware storage pools and arrays.)

 

I know this seems like a lot of work without a single benefit in the next few releases.  But Unraid is now becoming a very complex set of integrated systems and sub-systems.  A short pause to figure how everything is going to fit together into any integrated whole without major issues would have massive benefits down the road. 

[JorgeB]

1 hour ago, itimpi said:

I do agree that a better name for the Unraid array that is more type oriented (such as the UnArray’ suggestion, or perhaps simply ‘hybrid’) would be a good idea.

Tom mentioned using unRAID when you are referring to the array, especially after it can be set just as another pool type, Unraid refers to the OS.

[itimpi]

2 hours ago, JorgeB said:

Tom mentioned using unRAID when you are referring to the array, especially after it can be set just as another pool type, Unraid refers to the OS.

Petsonally just having capitalization differences is likely to be too confusing i would think.

[Frank1940]

50 minutes ago, JorgeB said:

Tom mentioned using unRAID when you are referring to the array,

 

I have read this post several times now.  My impression after doing this is that it sounds like legalese written by a lawyer attempting to flimflam the reader.  (Sorry, Tom)   Some questions and comments that I have are summarized below:

 

1--- What are we to call a collection of disks with one (or two) parity disks(s) and up to twenty-eight data disks?  

• unRAID? 
• unRAID array?  
• unRAID Array?  
• array? 
• Array?   

2--- Can the above collection of disks be assigned as either Primary or Secondary storage?

 

3--- Pools appear to be any other collections of storage devices.  In fact, I believe that you can actually setup RAID-like configurations for BTRFS and ZFS if desired.  Does this allow them to be configured as secondary storage with a primary storage?  

 

4--- Per Tom:

Quote

 

<<< We have a single "unRAID" array(*) and multiple user-defined "cache pools", or simply "pools".   >>>

 

<<<* note: we use the term "unRAID" to refer to the specific data organization of an array of devices (like RAID-1, RAID-5, etc).  We use "Unraid" to refer to the OS itself.>>>

 

 

Does this use of unRAID mean 'one (or two) parity disks(s) and up to twenty-eight data disks'  or does it mean any collection of devices formatted using a RAID configuration?

 

5--- There should be a consistent nomenclature of terms.  Is it going to be "pools" or "cache pools"?    If "array" is going to mean 'one (or two) parity disks(s) and up to twenty-eight data disks' then don't use it as term to refer a collection of disks that are formatted ZFS or BTRFS RAID-type configuration!  That is why real thought must be given to deciding what to name things.  (I, personally, would like to see 'array' become a more generic term (having a meaning of collection) rather rather than LimeTech's unique parity scheme.  Let's just call it--- unRAID!  Then if someone refers to unRAID array, it would have the same interpretation as  RAID-1 array)

 

PS---  I did take a bit of contrarian viewpoint as I wrote this because I wanted to present what a Neophyte would be thinking as he tried figure things out.  At the present time, the Documentation is really out of sync with the current software versions on this point.  And I think that the current release of the software, the GUI headings need to be brought more into line with the direction things are going. 

[JorgeB]

29 minutes ago, Frank1940 said:

Does this use of unRAID mean 'one (or two) parity disks(s) and up to twenty-eight data disks'  or does it mean any collection of devices formatted using a RAID configuration?

As I understood it unRAID will mean a pool using the MD driver, what we now call the array, it doesn't mater how many parity or data disks you have, it will be just another pool, you can have for example one unRAID pool, one ZFS pool, and one btrfs pool, etc, all can be primary or secondary storage.

[Frank1940]

1 hour ago, JorgeB said:

As I understood it unRAID will mean a pool using the MD driver, what we now call the array, it doesn't mater how many parity or data disks you have, it will be just another pool, you can have for example one unRAID pool, one ZFS pool, and one btrfs pool, etc, all can be primary or secondary storage.

Isn't there also a cache pool which consists of a single disk?  (I would think it could not be a secondary storage...) 

[sjrahn]

18 hours ago, TheLinuxGuy said:

There's this discussion on reddit suggesting that using ZFS on array disks may cause the disk to never go to sleep (random reads, seeks?), and that XFS or other filesystems are better at power efficiency. How true are those statements?

 

That may have been my comment. The TL;DR I came away with was that it was temporary files being written to these disks and that it was not coming from any plugin (maybe). I didn't want to deal with it so I reformatted them, but yes in the few days that I them in the array they never spun down. It was very tiny writes happening every 10-15 seconds on each of the ZFS disks in the array.

 

Here is some further discussions of the same issues I experienced:

 

[TheLinuxGuy]

1 hour ago, sjrahn said:

 

That may have been my comment. The TL;DR I came away with was that it was temporary files being written to these disks and that it was not coming from any plugin (maybe). I didn't want to deal with it so I reformatted them, but yes in the few days that I them in the array they never spun down. It was very tiny writes happening every 10-15 seconds on each of the ZFS disks in the array.

 

Yeah its mentioned in a few places. Most recently in this post from 24 hours ago - someone else called out ZFS as not letting their disks go to sleep: 

 

I guess my question for the unraid devs (and others with more experience on it)... is this normal and expected? Because the blog post said that disks can be spun down with "Hybrid" ZFS array disks in unraid.

[JorgeB]

12 hours ago, Frank1940 said:

Isn't there also a cache pool which consists of a single disk?  (I would think it could not be a secondary storage...) 

That's up to you, I expect any pool can be set as primary or secondary.

 

9 hours ago, TheLinuxGuy said:

I guess my question for the unraid devs (and others with more experience on it)... is this normal and expected? Because the blog post said that disks can be spun down with "Hybrid" ZFS array disks in unraid.

No, it's not normal, I've have multiple servers using ZFS on the array (and pools) and they all spin down, I remember reading about some spin down issues for users using the ZFS Master plugin.

[TheLinuxGuy]

9 hours ago, JorgeB said:

That's up to you, I expect any pool can be set as primary or secondary.

 

No, it's not normal, I've have multiple servers using ZFS on the array (and pools) and they all spin down, I remember reading about some spin down issues for users using the ZFS Master plugin.

 

ZFS Master plugin. I wonder if this is the unsung root cause for all those reported issues. Hope someone report back to confirm if this may be it.

[sjrahn]

4 hours ago, TheLinuxGuy said:

 

ZFS Master plugin. I wonder if this is the unsung root cause for all those reported issues. Hope someone report back to confirm if this may be it.

 

In one of those threads I linked someone tried without plugins and it still happened. I never bothered to verify so I can't say either way for sure but it would honestly make the most sense if it were the ZFS Master or some other plugin.

[bernieleogmx]

Had the same problems. The solution for me was found in this post:

"...turning the ZFS Master "Refresh interval" to "no Refresh", my ZFS-formatted disks stop spinning up all the time."