[PLUGIN] ZFS Master

[apandey]

On 5/11/2023 at 10:46 AM, Iker said:

The lua script is a very short ZFS channel program executed in read only mode for safety and performance reasons.

I can confirm that zfs_get_pool_data.lua is what spins up the pool for me. I'll have to probably spend a bit more time to deconstruct it and find the exact step that is the culprit, and see if any caching configuration can help. listing datasets and snapshots with zfs command does not spin up the pool for me

[Iker]

11 hours ago, apandey said:

I can confirm that zfs_get_pool_data.lua is what spins up the pool for me. I'll have to probably spend a bit more time to deconstruct it and find the exact step that is the culprit, and see if any caching configuration can help. listing datasets and snapshots with zfs command does not spin up the pool for me

 

ZFS commands don't spin up the pool because they don't include the plethora of properties that the script does. IMO, the most likely properties that could cause disks to spin up are "use" and "available", here is the properties list for snaps and datasets:

 

• snap properties = 'used','referenced','defer_destroy','userrefs','creation'
• dataset properties = 'used','available','referenced','encryption', 'keystatus', 'mountpoint','compression','compressratio','usedbysnapshots','quota','recordsize','atime','xattr','primarycache','readonly','casesensitivity','sync','creation', 'origin'

[isvein]

hello
What does the Primary Cache-->All/Metadata/None mean?

[samsausages]

On 4/26/2023 at 9:35 AM, Iker said:

Sounds good!, I will take a deeper look in the coming days, as this is a very unexpected behavior, and I haven't been able to reproduce it in a unRaid VM.

So I have run into a few other people with the same spin-up issue and it goes away removing the ZFS Master Plug-in.
I confirmed that one of those people also uses an LSI 9300-16i, like me.  That is what I'm starting to see as the culprit.

example where others are solving it by removing the ZFS-Master plugin:

 

[samsausages]

18 hours ago, isvein said:

hello
What does the Primary Cache-->All/Metadata/None mean?

It refers to the cache used for storing frequently accessed data and metadata in memory.
All stores Metadata & data in the cache
Metadata only stores metadata in the cache (used for listing directories/file lookups)
None stores none in the memory

[isvein]

32 minutes ago, samsausages said:

It refers to the cache used for storing frequently accessed data and metadata in memory.
All stores Metadata & data in the cache
Metadata only stores metadata in the cache (used for listing directories/file lookups)
None stores none in the memory

aha so it controlls the zfs-arc-ram-cache 
Thank you!

What does the sync-->Always/Disabled refer to?

Edited May 16, 2023 by isvein

[samsausages]

6 minutes ago, isvein said:

aha so it controlls the zfs-arc-ram-cache 
Thank you!

What does the sync-->Always/Disabled refer to?

it's for synchronous writes.
Always = every write operation will wait until the data is committed to stable storage before acknowledging the write as complete.  More latency but better data consistency.

Disabled = ZFS acknowledges the write operation as complete as soon as the data is written to the ZFS Intent Log (ZIL) Faster but vulnerable to power loss / system failure.
 

[Iker]

One question regarding this issue. Are you experiencing the same behavior even with the main tab closed? I have stated several times before, for other "issues," that if you don't have the unRaid GUI open in the main tab, ZFS Master doesn't perform any actions or execute any commands; from the info that you have presented, I agree with the conclusion that loading the main tab is going to wake up your disks, but, there are no other background processes in the plugin besides the ones running when the main tab is open. So if your spin-up issue persists even with the unRaid GUI closed, something else is happening.

Edited May 16, 2023 by Iker

[samsausages]

1 hour ago, Iker said:

 I agree with the conclusion that loading the main tab is going to wake up your disks

I haven't tested that, I just know every time I look they are spun up.  Haven't tried to correlate it with going to the "main" page.
I can't test for that right now, I don't have any ZFS formatted drives that aren't utilized & set to spin down, in the array right now.  Might have to make a pool with some spare disks next time I stop the array.

[Nogami]

Any chance would could get an option (or default) show datasets to be enabled?

[yarx]

Hi! Maybe everyone will be interested if the author describes in a few words the parameters that can be set when create dataset, like cache, metadata and others. I have few nvme and interesting what will be best settings for windows vm. Thanks!

[Iker]

Hi sorry for the late response to everyone. To your questions:

 

On 6/8/2023 at 11:57 PM, Nogami said:

Any chance would could get an option (or default) show datasets to be enabled?

 

Sure thing, I will push a new update once Unraid 6.12 is released; the update will include multiple new options, including the one you mentioned.

 

On 6/12/2023 at 3:35 AM, yarx said:

Hi! Maybe everyone will be interested if the author describes in a few words the parameters that can be set when create dataset, like cache, metadata and others. I have few nvme and interesting what will be best settings for windows vm. Thanks!

 

This is outside the plugin scope; however, you can search for multiple ways to improve your ZFS workloads.

 

Additionally, here are some other functionalities that I'm planning to include in the next version:

• Support for multiple charsets for datasets names
• Lazy load of snapshots counts and other info for datasets (This should improve the plugin loading times)
• Support for "zfs send."
• Some dialogs simplification.

Edited June 13, 2023 by Iker

[TheThingIs]

The plugin doesn't show datasets which have spaces in their name, is this correct? I thought ZFS was ok with spaces?

[Stokkes]

Hey, love this plugin, but wondering if there is going to be an update pushed in the future (is it still supported)? Watched the video on Youtube about ZFS datasets and installed this.

 

My issue I think is that it queries (on the Main tab) all the snapshots/datasets I have (which are A LOT) because I use snapoid/syncoid to auto-snap every 15 minutes and copy the snaps to another zfs pool, so there are thousands of snapshots, this really really slows down the Main tab, taking anywhere from 30-60 seconds to load.

 

Hoping we can get an option to have it not scan datasets or snapshots to make this snappier - ideally if we can just go into a different part of the webGui to view snapshots, etc.

 

Thanks!

[Iker]

21 hours ago, TheThingIs said:

The plugin doesn't show datasets which have spaces in their name, is this correct? I thought ZFS was ok with spaces?

 

Kind of. You can not create them from the user interface (Next update I will fix that), but you can list them and operate over them.

 

 

58 minutes ago, Stokkes said:

wondering if there is going to be an update pushed in the future (is it still supported)?...this really really slows down the Main tab...

 

Please read a couple comments back in this thread.

 

[Stokkes]

1 hour ago, Iker said:

Please read a couple comments back in this thread.

 

Omg so sorry! Thank you that's awesome. Looking forward to the release!

[TheThingIs]

Quote

Kind of. You can not create them from the user interface (Next update I will fix that), but you can list them and operate over them.

Interesting, mine isn't showing any datasets which have a space in their name.

[Iker]

4 hours ago, TheThingIs said:

Interesting, mine isn't showing any datasets which have a space in their name.

Please share what Unraid and Plugin version are you using, and if possible the ZFS list command output from the top directory that contains those datasets not being detected and the dir tree; if it's not possible, at least some mock up names and tree that can help me to reproduce the situation.

[deliverer]

This is probably a dumb question, but is there a way to set up automatic ZFS snapshots based on time?  Say I want to take a snapshot every 4 hours, is this possible?  Is there also a way to maintain how many snapshots are kept? 

 

Lastly, what does "Recursively create snapshots of all descendent datasets" mean when I manually perform a snapshot?  I don't quite understand if I should select this or not?  Based on my google research, it's more if you want to do the entire pool?  Since the UI only allows the dataset to be selected for a snapshot, does it matter if I select this option or not?

[boomam]

Probably a silly question, as ZFS is ZFS, but is there any negative impact to the usage of this plugin or ZFS features, considering they are not exposed in Unraid itself nativly?

For example, if i create sub-datasets and/or snapshots, then Unraid 6.13 adds support for managing that in the GUI, i assume it'll just work?

[boomam]

15 minutes ago, deliverer said:

This is probably a dumb question, but is there a way to set up automatic ZFS snapshots based on time?  Say I want to take a snapshot every 4 hours, is this possible?  Is there also a way to maintain how many snapshots are kept? 

 

Lastly, what does "Recursively create snapshots of all descendent datasets" mean when I manually perform a snapshot?  I don't quite understand if I should select this or not?  Based on my google research, it's more if you want to do the entire pool?  Since the UI only allows the dataset to be selected for a snapshot, does it matter if I select this option or not?

If you know the command you want to run to achieve that as a 'one off', you can just use User Scripts to schedule it with Cron.

[Iker]

16 minutes ago, deliverer said:

This is probably a dumb question, but is there a way to set up automatic ZFS snapshots based on time?  Say I want to take a snapshot every 4 hours, is this possible?  Is there also a way to maintain how many snapshots are kept? 

 

Yes, it is, take a look at ZnapZend, Sanoid and other programs that can help you with that, is one of the most important and handy features.

 

17 minutes ago, deliverer said:

Lastly, what does "Recursively create snapshots of all descendent datasets" mean when I manually perform a snapshot?  I don't quite understand if I should select this or not?  Based on my google research, it's more if you want to do the entire pool?  Since the UI only allows the dataset to be selected for a snapshot, does it matter if I select this option or not?

 

Let's say you have the following datasets structure:

 

 

Note that only powlarr, radarr and sonarr, are Datasets; my_folder is a regular folder, and my_file, well is just a file. Now imagine you want to create a snapshot that apply to every single dataset in appdata and regular folders and files; you check the option "Recursively create snapshots...", if you don't, the snapshot is only going to apply to the regular folder and files, but not the datasets inside appdata. this is handy if you want for example, backup the entire appdata dir every 6 hours.

 

21 minutes ago, boomam said:

Probably a silly question, as ZFS is ZFS, but is there any negative impact to the usage of this plugin or ZFS features, considering they are not exposed in Unraid itself nativly?

For example, if i create sub-datasets and/or snapshots, then Unraid 6.13 adds support for managing that in the GUI, i assume it'll just work?

 

No negative impact, I'm using standard ZFS stuff here, so everything should be compatible with Unraid in the future.

 

21 minutes ago, boomam said:

If you know the command you want to run to achieve that as a 'one off', you can just use User Scripts to schedule it with Cron.

 

I strongly suggest you to get away of custom scripts or "manual scheduled" snapshots, the lifecyle of those things can be a complete nightmare, and the more you have, the more complicated things start to be, use well stablished solutions that create and delete snapshots in a scheduled; Sanoid and ZnapZend are my goto choices. Personally I use ZnapZend (In a container, not the CA plugin) and only create snapshots for particular tasks, like replicating a dataset to another pool or server.

 

PD: I understand this is very different from what you are used to, but ZFS is a very powerfull system with many functionalities (Clones are a great example), once you get the hand of it, you can really use it at his full, This article should help you to jump start with the concepts: https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/

 

[boomam]

9 minutes ago, Iker said:

No negative impact, I'm using standard ZFS stuff here, so everything should be compatible with Unraid in the future.

Thanks for clarifying.

 

As a question, the quota's on the datasets shows a default of 'B', so I assume bytes, but the maximum I can put in is 10000 B?

Is this a known thing?

[Iker]

Quota shows B because is the smaller unit, but you can specify it in K, M, G, T at dataset creation or when you edit a dataset. (Probably worth to include the unit as a list in the Dataset creation dialog for clarity)

[Stokkes]

11 hours ago, deliverer said:

This is probably a dumb question, but is there a way to set up automatic ZFS snapshots based on time?  Say I want to take a snapshot every 4 hours, is this possible?  Is there also a way to maintain how many snapshots are kept? 

 

Lastly, what does "Recursively create snapshots of all descendent datasets" mean when I manually perform a snapshot?  I don't quite understand if I should select this or not?  Based on my google research, it's more if you want to do the entire pool?  Since the UI only allows the dataset to be selected for a snapshot, does it matter if I select this option or not?

 

There is ZnapZend and Sanoid, I suggest using Sanoid, it's not too difficult to configure and you can run it via User Scripts.. I have it snapshot my apdata and other volumes ever 15 minutes, keep those for 3 days, snapshot every hour, keep those for a week, snapshot every day, keep those for 30 days, you get the idea. 

 

It will also auto-clean up snapshots..

 

There is another tool that comes with it, Syncoid that you can then sync the snapshots to another ZFS pool, which I also do. That zfs pool can be another pool on your unraid server or even better, another host altogether!