[PLUGIN] ZFS Master

[Iker]

Hi guys, a new update is live with the following changelog:

 

2022.11.05

• Fix - Error on pools with snapshots but without datasets
• Fix - Dialogs not sizing properly
• Add - Snapshot Batch Deletion

Be aware that for the Snapshots Admin dialog, the delete and other operations are now reported through the status message; however, the dialog doesn't refresh the info, so you have to close it, hit the refesh button and open it again.

 

I will work on one more feature (Clone snapshots and promote them) before the year ends; after that; I will focus on rewriting a portion of the backend, so every functionality becomes a microservice; that should open the door for  a good API (Automation is coming...).

 

Best

Edited November 5, 2022 by Iker

[Iker]

Last update of the year (Probably):

 

2022.11.12

• Fix - Error on dialogs and input controls
• Add - Clone capabilities for snapshots
• Add - Promote capabilities for datasets

 

General Advice: Be careful when using clones and promoting datasets; things can become messy quick

[daver898]

Hi,

I installed the update and could not see any difference. Does it only work for unraid 6.11.n

[Iker]

9 hours ago, daver898 said:

Hi,

I installed the update and could not see any difference. Does it only work for unraid 6.11.n

 

Hi daver, the new functionalities are for the most part associated with snapshots; for examploe, batch deletion and cloning, are exclusively present on the snapshot admin dialog.

[1WayJonny]

Hi,

 

Using this plug in and its working well but concerned with what it does to my pools. I am new to zraid and when looking at the history I see some concerning items.

 

This plug-in is running exporting of data twice per hour as a duplicate.

 

Why is running zpool import -a and importing datasets command. I have never asked the plug to import anything.

 

My zpools with the duplicate data exports have a ton of history and commands that I am not sure is necessary. 

 

Is there any way to erase this zpool history or has this plug-in runs its course? Not sure why its running twice, all the time.

 

2022-12-17.10:18:39 zfs program -jn -m 20971520 XXX /usr/local/emhttp/plugins/zfs.master/scripts/zfs_get_pool_data.lua XXX
2022-12-17.10:18:39 zfs program -jn -m 20971520 XXX /usr/local/emhttp/plugins/zfs.master/scripts/zfs_get_pool_data.lua XXX

 

2022-12-17.11:05:54 zpool import -a

2022-12-17.11:13:01 zpool import -a

 

Why is this importing twice in 10 min? I have never run the import command at all.

[Iker]

Hi @1WayJonny, plugin author here; I'm not so sure if you are in the correct forum, but if case you are, it seems that you get a couple of things wrong:

1. The plugin doesn't implement any commands related to importing pools; the source code is available, and you can check in the repo "zfs import" command doesn't exist.
2. There are no actions performed automatically or in the background by the plugin besides refreshing the info every 30 seconds (or the time you configure) when you are in the main tab.
3. The zfs program you correctly identify is the refreshing action; it only occurs when interacting with the main tab.
4. There is something else messing with your pools; I'm not sure why, but I suggest you uninstall all the zfs related plugins and then check the history commands.

Best,

[1WayJonny]

On 12/23/2022 at 7:45 PM, Iker said:

Hi @1WayJonny, plugin author here; I'm not so sure if you are in the correct forum, but if case you are, it seems that you get a couple of things wrong:

1. The plugin doesn't implement any commands related to importing pools; the source code is available, and you can check in the repo "zfs import" command doesn't exist.
2. There are no actions performed automatically or in the background by the plugin besides refreshing the info every 30 seconds (or the time you configure) when you are in the main tab.
3. The zfs program you correctly identify is the refreshing action; it only occurs when interacting with the main tab.
4. There is something else messing with your pools; I'm not sure why, but I suggest you uninstall all the zfs related plugins and then check the history commands.

Best,

 

@Iker

 

Thanks for the great detailed reply. It will help me figure out what is calling the additional commands.

 

I only have ZFS, ZFS-companion and ZFS-Master and not much else installed besides Plex. I see you find it weird that something has been calling for import more then needed. I have never manually imported anything either is really weird.

 

As for #3 if I am reading correct that is from the ZFS-Master plugin but only when I click the main page? Its a new setup so I have been clicking more often.

 

Summary: No back ground calls are made and only refreshed every 30 second when clicking on the main page and sitting active.

Edited December 29, 2022 by 1WayJonny

[1WayJonny]

Just an update to my question.

 

The command is actually coming from the ZFS plugin itself and it ran after a reboot. Once ZFS becomes part of UnRaid that may go away but that is what is issuing the command.

 

The reason I seen it more was because of the restarts I was doing. When first with the test rig I did not have this plug-in installed and was not rebooting at all. I started rebooting more and found this plug at the same time.

 

Thanks for the reply and plug-in work! 😃

Edited January 4, 2023 by 1WayJonny

[HumanTechDesign]

Thank you @Iker once more for this plugin!

 

I have a small issue to report, when it comes to the loading times on the Main page: I recently attached a zfs-formatted external USB HDD as a backuppool. Because of the nature of this being a backup device, quite a lot more snapshots are being kept there than in my main pool(s) (~12.5k on the HDD vs ~1.2k on the largest main pool). Since I added this backuppool, loading times of the main page have gone up dramatically (up to around a minute or even two). Not surprisingly, the Snapshot Admin UI also takes ages to load. The `system` dataset (with all of the docker stuff) is already excluded via pattern and I noticed this change after hooking up the backup HDD and sending my snapshots over.

I was therefore wondering, if there is any way for you to optimize (e.g. maybe "lazy load") the snapshots when loading UI? Comparable to the improvements you made with the general main UI.

Thanks already for your hard work!

[Iker]

This lazy load functionality has been on my mind for quite a while now, and it will help a lot with load times in general; that's why I'm focusing on rewriting a portion of the backend, so every functionality becomes a microservice, but given the unRaid UI architecture, it's a little more complicated than I anticipated, I understand your issue but please hang in there for a little while.

[gyto6]

1 minute ago, Iker said:

This lazy load functionality has been on my mind for quite a while now, and it will help a lot with load times in general; that's why I'm focusing on rewriting a portion of the backend, so every functionality becomes a microservice, but given the unRaid UI architecture, it's a little more complicated than I anticipated, I understand your issue but please hang in there for a little while.

 

Your concern is kind. Thanks however

[Ter]

I am not sure if this is just happening to me since I have switched over from TrueNAS Scale, but the encrypted dataset I have does not want to show up when I unlock it with this plugin. It might be because the encrypted dataset has child datasets that contain actual data ("tank/usr" is encrypted and inside that I have child datasets for users, the child dataset "folders" get mounted but nothing is visible inside). To make it show up I also have to run `zfs mount -a` after I do the unlock in UI.

 

Yup I can confirm the actual "bug". "tank/usr" gets mounted but the child datasets do not. If I do `zfs mount tank/usr/child` the files become visible for that dataset.

Edited February 3, 2023 by Ter

[Iker]

Hi @Ter thanks for your feedback, let me take a closer look into this and get back to you. The problem with "-a" option, is that it mounts all the datasets, not just the childs, that can be a little bit problematic for other users.

Edited February 3, 2023 by Iker

[Ter]

On 2/3/2023 at 3:20 PM, Iker said:

Hi @Ter thanks for your feedback, let me take a closer look into this and get back to you. The problem with "-a" option, is that it mounts all the datasets, not just the childs, that can be a little bit problematic for other users.

Thank you. I have edited my comment, so I am not sure if you managed to see it. This can be done without mounting with "-a". Instead, child datasets of the encrypted "parent" datasets have to be mounted too. Otherwise the "parent" gets mounted and the directories for child datasets are visible, but they are empty as they are not mounted.

Edited February 10, 2023 by Ter

[Veriwind]

Is editing the nfs/samba files still the best way to 'export' these datasets from a zfs pool? I was looking around here and not seeing a great option other than that. Any advice/plugins?

[Iker]

On 2/10/2023 at 8:18 AM, Ter said:

Thank you. I have edited my comment, so I am not sure if you managed to see it...

Yes, sorry for the late response, I'll be pushing an update next week, with the nested mount command.

 

20 hours ago, Veriwind said:

Is editing the nfs/samba files still the best way to 'export' these datasets from a zfs pool?...

 

For now at least that's the way to go, in the coming 6.12 version ZFS will be a first class citizen, that means that you can import pools and create shares using unRaid's GUI.

[Veriwind]

On 2/22/2023 at 8:16 AM, Iker said:

Yes, sorry for the late response, I'll be pushing an update next week, with the nested mount command.

 

For now at least that's the way to go, in the coming 6.12 version ZFS will be a first class citizen, that means that you can import pools and create shares using unRaid's GUI.

Well, its been an absolute pleasure to use your plugin while we wait for this. I've got it set up nicely. Thank you so much!

[Iker]

Hi guys, a new update is live with the following changelog:

 

2023.02.28

• Fix - PHP 8 Upgrades
• Fix - Export pool command
• Fix - Error on parsing dataset origin property

 

I`m still working on the lazy load thing ( @HumanTechDesign ), a page for editing dataset properties, and make everything compatible with Unraid 6.12. The microservices idea is a big no, so probably I'm going to use websockets (nchan functionality) to improve load times, the Snapshots are going to be loaded on the background. 

@Ter About the recursive mount, ZFS doesn't have an option for that use case, so I have to code myself, still evaluating if is worth impementing or not.

Best,

[gyto6]

On 2/28/2023 at 7:41 PM, Iker said:

I`m still working on the lazy load thing ( @HumanTechDesign ), a page for editing dataset properties, and make everything compatible with Unraid 6.12. The microservices idea is a big no, so probably I'm going to use websockets (nchan functionality) to improve load times, the Snapshots are going to be loaded on the background. 

 

 

Looking forward with an certain enthusiasm. Thanks for your work. 💖

[Bizquick]

Any Idea how I can make the snap shots visable in windows shadow copies?

I've tried several people's ideas on this and none of them are complete enough to make this work. This simple this is making me want to try migrating to TrueNas Scale. I've been waiting so long for Unraid to add ZFS and now they are close. I've never been able to make this one part work.

 

[Iker]

It's relatively easy, you can find examples of shares and vsf objects on this post:

 

However, my advice is to read and understand the samba options and naming convention necccesary for this to work, this is very well explained on the levelone forums post (https://forum.level1techs.com/t/zfs-on-unraid-lets-do-it-bonus-shadowcopy-setup-guide-project/148764).

 

Bests,

[Bizquick]

I really tied to follow all those and still no luck.
if its really just the naming convention more or less then almost all the setting people have posted should be working. I dont see others changing the naming from the defaults unless they use someone else's idea.

Just seams kind of silly I have to do alot of trial and error that TrueNas has figured out with perfection pretty easily. And I really dislike the guys over at XI-systems. But I'm starting to not feel very good about this either.

[Bizquick]

On 3/27/2023 at 7:21 AM, Iker said:

It's relatively easy, you can find examples of shares and vsf objects on this post:

 

However, my advice is to read and understand the samba options and naming convention necccesary for this to work, this is very well explained on the levelone forums post (https://forum.level1techs.com/t/zfs-on-unraid-lets-do-it-bonus-shadowcopy-setup-guide-project/148764).

 

Bests,

Thanks for all that. you know out of everything not sure where. But in the end I found the only reason it was not working.
was because I had to turn of the sharing at the share in unraid. The SMB extra settings I kept trying in were not applying. I spent so long doing many other things before trying that. They should say if you use the extra setting to do that in the Manual. It would have made so much more sense. Or better yet add allowing vfs object in the global smb.conf  But I'm going to guess Limetech has way more things to fix before going back to SMB config stuff. They are most likely happy with how it works now and don't have time to look at global config issues.

 

 

[Iker]

Hi folks, a new update is live with the following changelog:

 

2023.04.03

• Add - Rename datasets UI
• Add - Edit datasets UI
• Add - unRaid 6.12 compatiblity
• Add - Lazy load for snapshots admin UI
• Fix - Improve PHP 8 Compatibility

 

Snapshots lazy load functionality is slowly moving forward; in the next version, it will probably be integrated on the main page.

Best,

[AgentXXL]

I have a relatively simple single vdev pool that I created using the old plugin, but it won't import properly under 6.12. ZFS Master and unRAID see the correct used and free space, but no folders (should only be one called TV) are seen under the dataset mountpoint. The bug report is here with diagnostics and some screenshots.

 

Any idea how I can fix this? This is my 2nd attempt to update to 6.12, with the same failure both times. I'm holding off on the rollback to 6.11.5 for the time being, but will have to rollback later today if I can't find a solution. Any assistance appreciated!