[PLUGIN] ZFS Master

[gyto6]

Hi @Iker,

 

One idea after another, would it be possible to manage automatic scrub through your plugin and user Unraid mails settings to send automatic report once a scrub begins and finishes?

[jlficken]

I created a Pool and Dataset but for some reason via SMB I get "Permission Denied" when trying to create a file/directory.

 

I used 775 as the permissions when creating the dataset and here is the SMB Extra config that I added for SMB:

[ZFSTest]
      path = /mnt/disks/ZFSTest/Test
      comment =
      browseable = yes
      writeable = no
      read list = 
      write list = jlficken
      valid users = jlficken
      case sensitive = auto
      preserve case = yes
      short preserve case = yes
      vfs objects =
 

I can view it I just can't modify it in any way.

 

I'm a complete newbie at ZFS so anything you need me to grab just ask and I'll get it for you.

 

This is what I see for permissions on that directory:

root@FSNAS1TU:~# ls -l /mnt/disks/ZFSTest
total 1
drwxr-xr-x 2 root root 2 Jul 28 08:40 Test/

 

I can create files/directories through the terminal in the directory just not through SMB.

Edited July 28, 2022 by jlficken

[Iker]

The SMB configuration is beyond the scope of this plugin; however, this are my "templates" for the SMB Shares; the permissions are 777 and the owner of the folders is "nobody":

 

Private Folder

[example_private_share]
	path = /ssdnvme/example_private
	comment = example share
	browsable = yes
	guest ok = no
	writeable = yes
	write list = iker
	read only = no	
	create mask = 0775
	directory mask = 0775
	vfs objects = shadow_copy2
	shadow: snapdir = .zfs/snapshot
	shadow: sort = desc
	shadow: format = %Y-%m-%d-%H%M%S
	shadow: localtime =  yes
	shadow: snapdirseverywhere = yes

 

Public Folder

[example_public_share]
	path = /ssdnvme/example_public
	comment = UnRaid related info
	browsable = yes
	guest ok = yes
	writeable = yes
    write list = iker
	read only = yes
	create mask = 0775
	directory mask = 0775

 

Private Hidden Folder

[example_private_hidden]
	path = /ssdnvme/example_private_hidden
	comment = My Private info
	browsable = no
	guest ok = yes
	writeable = no
	write list = iker
	read only = yes
	create mask = 0775
	directory mask = 0775
	vfs objects = shadow_copy2
	shadow: snapdir = .zfs/snapshot
	shadow: sort = desc
	shadow: format = %Y-%m-%d-%H%M%S
	shadow: localtime =  yes

 

Hope it helps!

[calvados]

Fantastic work @Iker!  In a future version would it be possible to automate/schedule scrubs?  Alternatively, is there another way to automate ZFS scrubs?

 

Thanks @Iker,

Cal.

[gyto6]

1 hour ago, calvados said:

Fantastic work @Iker!  In a future version would it be possible to automate/schedule scrubs?  Alternatively, is there another way to automate ZFS scrubs?

 

Thanks @Iker,

Cal.

 

Add User Script plugin and create a script with the relative command :

zpool scrub tank

 

Twice a month for public drives and once a month for professional drives.

[Iker]

Hi Folks, I just posted an update for the Plugin; this improves load times by 50% in most cases. Please check the exclusion pattern change. Unfortunately, the backend change requires to use of a Lua Pattern instead of a regex

 

2022.08.02

• Warning - Please Update your exclusion pattern!
• Add - Browse Button for Datasets
• Add - Support for listing volumes!!
• Add - Lua script backend for loading dataset informatin http://lua-users.org/wiki/PatternsTutorial )
• Change - UI columns re-organized to the unraid way (sort of)

Edited August 3, 2022 by Iker

[HumanTechDesign]

5 hours ago, Iker said:

Hi Folks, I just posted an update for the Plugin; this improves load times by 50% in most cases. Please check the exclusion pattern change. Unfortunately, the backend change requires to use of a Lua Pattern instead of a regex

 

2022.08.02

• Warning - Please Update your exclusion pattern!
• Add - Browse Button for Datasets
• Add - Support for listing volumes!!
• Add - Lua script backend for loading dataset information
• Change - Exclusion pattern for datasets (Please check http://lua-users.org/wiki/PatternsTutorial)
• Change - UI columns re-organized to the unraid way (sort of)

Hi, first of all thank you for this great plugin! Really useful to manage snapshots etc.

 

Concerning this new change in pattern construction: I'm not that strong in pattern matching and therefore have problems to adapt this to my setup. Could you maybe provide some basic examples for really generic use cases (e.g. excluding all datasets below a certain folder etc.)?

In my case, I have my docker system share on a dedicated SSD ZFS pool. Due to the docker-ZFS implementation, it generates a dataset for every layer of all the images (see e.g. this explanation). With the previous regex, I just exluded all sub datasets below the "system" share. I now tried just the basic versions (SSDPool is my mountpoint)

/SSDPool/system/
/SSDPool/system/%w+
/SSDPool/system/(%w+)

but none of them matched. Is there a way to match the whole content of the system subdirectory?

[Iker]

7 hours ago, HumanTechDesign said:

Hi, first of all thank you for this great plugin! Really useful to manage snapshots etc.

 

Concerning this new change in pattern construction: I'm not that strong in pattern matching and therefore have problems to adapt this to my setup. Could you maybe provide some basic examples for really generic use cases (e.g. excluding all datasets below a certain folder etc.)?

In my case, I have my docker system share on a dedicated SSD ZFS pool. Due to the docker-ZFS implementation, it generates a dataset for every layer of all the images (see e.g. this explanation). With the previous regex, I just exluded all sub datasets below the "system" share. I now tried just the basic versions (SSDPool is my mountpoint)

/SSDPool/system/
/SSDPool/system/%w+
/SSDPool/system/(%w+)

but none of them matched. Is there a way to match the whole content of the system subdirectory?

 

This Lua Patterns are very simple, this should do the trick:

 

/system/.*

 

Let met know if it helps.

Edited August 3, 2022 by Iker

[muddro]

11 hours ago, Iker said:

Hi Folks, I just posted an update for the Plugin; this improves load times by 50% in most cases. Please check the exclusion pattern change. Unfortunately, the backend change requires to use of a Lua Pattern instead of a regex

 

2022.08.02

• Warning - Please Update your exclusion pattern!
• Add - Browse Button for Datasets
• Add - Support for listing volumes!!
• Add - Lua script backend for loading dataset informatin http://lua-users.org/wiki/PatternsTutorial )
• Change - UI columns re-organized to the unraid way (sort of)

Loads super fast...fast than the unassigned devices plugin. Awesome work!

[RedTechie]

@Iker

 

This looks like a great update! Though I am having problems with the new exclusion list. My "Main" page is loading VERY slowly now. I'm assuming its becasue my old exclusion pattern no longer works. But I can't get the new one to work either based on your reply above.

 

This is what my ZFS Datasets look like:

 

I tried the following exclusion pattern with no luck: 

Quote

zfs-nvme/system/docker/.*

 

I also tried the following with no luck:

Quote

zfs-nvme/system/docker/*

 

Am I missing something here? Thanks!

[RedTechie]

3 minutes ago, RedTechie said:

@Iker

 

This looks like a great update! Though I am having problems with the new exclusion list. My "Main" page is loading VERY slowly now. I'm assuming its becasue my old exclusion pattern no longer works. But I can't get the new one to work either based on your reply above.

 

This is what my ZFS Datasets look like:

 

I tried the following exclusion pattern with no luck: 

 

I also tried the following with no luck:

 

Am I missing something here? Thanks!

 

@Iker I figured it out, I can't include the pool name. This worked:

 

Quote

/system/docker/.*

 

[HumanTechDesign]

5 hours ago, Iker said:

 

This Lua Patterns are very simple, this should do the trick:

 

/system/.*

 

Let met know if it helps.

Awesome, yes, this one worked. I was apparently overcomplicating things!

 

Also just a heads up: I was gonna report the issue that datasets with spaces in them (e.g. when there is a VM called Windows 10) weren't picked up correctly. This now apparently also seems to be solved, as it now correctly shows actually all datasets. So thank you very much once more!

 

EDIT: Now that I'm here anyways, could I make two wishes (really not in a position to ask an open source developer, but I was thinking, maybe you could think about them)

1.) Could you think about a feature to delete multiple snapshots at once? Maybe just by having multiple checkboxes or by being able to select multiple rows in the Snapshot Admin?

2.) Maybe add another confirmation dialog when deleting a snaphot? Right now you really need to make sure that you clicked the button on the right row

Edited August 3, 2022 by HumanTechDesign

[Iker]

Great! the new method to obtain datasets, volumes, snapshots, and their respective properties; is as good as it can be to prevent parsing errors and, overall, much quicker than the previous one. Now that I have finished with that and it is working (As far as I know) very good, I will focus on new functionality and improvements over the UI; the two you describe are on their way.

[gyto6]

54 minutes ago, Iker said:

Great! the new method to obtain datasets, volumes, snapshots, and their respective properties; is as good as it can be to prevent parsing errors and, overall, much quicker than the previous one. Now that I have finished with that and it is working (As far as I know) very good, I will focus on new functionality and improvements over the UI; the two you describe are on their way.

 

If I can help you concerning the full dataset snapshots deletion:

 

zfs destroy tank/docker@%

 

The % in ZFS is like the most common * character to include any value. So any snapshot from the docker dataset shall be removed.

[Gdtech]

Are you suppose to be able to browse files on ZFS datasets within Unraid GUI, when i click on browse button on any dataset it comes up with invalid path

I can browse the files on windows workstation through SMB, is this something to do with Dynamix File Manager or ZFS Master or Unassigned Devices.

 

Any help would be appreciated 

thanks

[BasWeg]

On 4/10/2022 at 7:16 PM, Iker said:

2022.08.02

 

• Warning - Please Update your exclusion pattern!
• Add - Browse Button for Datasets
• Add - Support for listing volumes!!
• Add - Lua script backend for loading dataset information (50% faster loading times)
• Change - Exclusion pattern for datasets (Please check http://lua-users.org/wiki/PatternsTutorial)
• Change - UI columns re-organized to the unraid way (sort of)

Hi,
I've just updated the plugin and corrected the exclusion pattern.
Nevertheless the Main Page does show following warning:

"Warning: session_start(): Cannot start session when headers already sent in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(522) : eval()'d code on line 3"

 

Any idea? (I've cleared the browser cache of course)
UNRAID Version 6.9.2

 

Edited August 5, 2022 by BasWeg

[Iker]

4 hours ago, Gdtech said:

Are you suppose to be able to browse files on ZFS datasets within Unraid GUI, when i click on browse button on any dataset it comes up with invalid path

I can browse the files on windows workstation through SMB, is this something to do with Dynamix File Manager or ZFS Master or Unassigned Devices.

 

Any help would be appreciated 

thanks

 

It really depends on the unRaid version; in the new RC Series (6.11) it's possible to explore any path from the builtin browser, that's why I revive this functionality; as far as I know it's not possible in the stable 6.10.

Edited August 5, 2022 by Iker

[Iker]

2 hours ago, BasWeg said:

Hi,
I've just updated the plugin and corrected the exclusion pattern.
Nevertheless the Main Page does show following warning:

"Warning: session_start(): Cannot start session when headers already sent in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(522) : eval()'d code on line 3"

 

To be Honest; I have not tested the plugin installation and functionality in the version that you referr, is going to take me a couple days to test the plugin and debug the error.

[calvados]

On 8/1/2022 at 12:20 AM, gyto6 said:

 

Add User Script plugin and create a script with the relative command :

zpool scrub tank

 

Twice a month for public drives and once a month for professional drives.

Thank-you @gyto6!  Very much appreciated.

Cal.

 

Edited August 6, 2022 by calvados

[dianasta]

Hi  @Iker, 

 

I've noticed the same issue like other users, UNRAID Version 6.9.2.

 

Thanks for the excellent plugin, any assistance is much appreciated.

 

Warning: session_start(): Cannot start session when headers already sent in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(518) : eval()'d code on line 3

 

 

Edited August 11, 2022 by dianasta

[Iker]

19 hours ago, dianasta said:

Hi  @Iker, 

 

I've noticed the same issue like other users, UNRAID Version 6.9.2.

 

Thanks for the excellent plugin, any assistance is much appreciated.

 

Warning: session_start(): Cannot start session when headers already sent in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(518) : eval()'d code on line 3

 

 

 

Hi, I have reproduced and isolated the issue, but fixing it is a little bit harder; in the coming days, I'll post an update with this issue fixed and a couple of new UI improvements.

[Iker]

Hi Guys, I just posted a new update for the Plugin; this introduces support for Datasets Encryption, including creation and lock and unlock datasets. Also there is a major change in the UI, now things are displayed like a folder structure. Also this version fix the error for you guys that are using the unraid 6.9.2 version (@dianasta @BasWeg).

 

For the next update i'll focus on the snapshots management thing (Dialogs, multiple selection, etc.).

 

2022.08.21

• Change - UI into "folder" structure
• Add - Support for ZFS Encryption
• Add - Unlock and Lock actions for encrypted datasets
• Fix - Error on unRaid 6.9.2 associated with session management

Edited August 21, 2022 by Iker

[nerdsmcgee]

Suggestion: The ability to have a dataset minimized on Dashboard load. I'm using a pair of SSDs for direct Docker access via folder, but this results in a ton of non-concerning entries on the dashboard page with no ability to filter out (unless I'm doing it wrong!).

[Iker]

Datasets can be minimized, and the configuration is stored on a cookie that lasts a year; the same happens for pools; for your case, if you want to handle things a little bit better, you can create a child dataset (dockerfiles for example) and filter or collapse that dataset.

 

[HumanTechDesign]

On 8/21/2022 at 11:24 PM, Iker said:

Hi Guys, I just posted a new update for the Plugin; this introduces support for Datasets Encryption, including creation and lock and unlock datasets.

Great new updates, working flawlessly! Thank you once more!

 

When it comes to encryption, I might have another feature request (although I'm not sure if that is something for your plugin or on the ZFS for Unraid plugin side): I have two encrypted datasets (encrypted using CLI long before your plugin was around). In order to ensure an actual at rest protection for these datasets, they need to be unmounted AND the key unloaded (see here for example). Unfortunately, Unraid does not do this by default when shutting down (based on this comment but also general first-hand experience). Therefore I have a User Script to run at array stop (containing the commands

zfs unmount tank/dataset
zfs unload-key -r -a

and the respective opposite commands for loading the key and mounting on start of the array).

 

However, I sometimes had the problem, that the User Script did not properly execute completely (sometimes because the device was still busy) before the server shut down, resulting in a kind of unclean shutdown (auto-starting a scrub on the next boot). My workaround so far was stopping the array manually (and waiting for correct locking of the datasets) before shutting down.

 

I was wondering if there is a way for your plugin to communicate with Unraid to encrypt at shutdown after unmounting with a higher priority than the User Scripts plugin.