[PLUGIN] ZFS Master

[Iker]

1 hour ago, d3m3zs said:

But also would be great to automatically remove tmp folders after rsync command done.

 

Sorry, I'm very reluctant to delete data without user verification; I prefer you folks verify that everything is fine and the data that you need is already in the dataset, and then and only then, you manually delete the source folder; this prevents accidentally losing data in case of unexpected errors. As I said before, I'm working on changing the dialog for conversion to a progress bar integrated on the main UI.

[d3m3zs]

13 minutes ago, Iker said:

I'm working on changing the dialog for conversion to a progress bar integrated on the main UI.

Thank you! That is exactly what we need to this feature. All other - agree and understood. 

[Renegade605]

Today, I've noticed that the sorting algorithm for this plugin is lexicographic.

 

 

Switching to a more intelligent sort method in a future release would be desirable.

 

 

EDIT: Also, when creating a new dataset, the options for "Mount" are Yes/No. But, "Yes" is actually more like inherit, since it will not be mounted if the parent dataset is not mounted. The options should probably be Inherit/Yes/No, with "Yes" enabling (and requiring) the text box for a mountpoint and the other options leaving that text box disabled.

Edited February 22 by Renegade605

[d3m3zs]

Found one more issue - invalid character '[' in name

Need to remove such symbol first

find . -depth -type d -name '*\[*\]*' -execdir bash -c 'mv -- "$1" "${1//[\[\]]/}"' bash {} \;

 

Edited February 22 by d3m3zs

[Renegade605]

Fellow ZFS enthusiasts, I made this to notify me if (when lol) I accidentally create a folder somewhere that should have been a dataset instead. Please use and enjoy if you like.

https://gist.github.com/Renegade605/8eba0af1e7fa1b16cb74af0e79f3be98

 

Also note, I discovered recently that the mover (with CA Mover Tuning installed, not sure if it's part of the default mover) as part of it's operation destroys empty datasets when it's finished. If this isn't desirable to you for the obvious reasons, my current workaround is to snapshot datasets when they're empty and the destroy operation will fail.

There are a few other quirks with that plugin on ZFS pools, see here for more info:

 

[Iker]

New update with the following changelog:

 

2024.03.03

• Fix - Dataset name validation before convertion
• Fix - rsync command for folders with whitespaces or special characters

• Fix - Errors on empty exclusion patterns

• Add - Compatibility with 6.13 beta

 

Edited March 3 by Iker

[MowMdown]

On 2/28/2024 at 2:28 PM, Renegade605 said:

Also note, I discovered recently that the mover (with CA Mover Tuning installed, not sure if it's part of the default mover) as part of it's operation destroys empty datasets when it's finished. If this isn't desirable to you for the obvious reasons, my current workaround is to snapshot datasets when they're empty and the destroy operation will fail.

 

 

Well that's to be expected. You've set your share to be moved off of your pool onto the array. No idea why you'd want to snapshot an empty dataset that's never going to have data in it?

 

Edit:

 

@Iker

 

I have a QOL suggest for the snapshot admin pop-up window. Instead of the pop-up window, can the snapshots just expand below the selected dataset? I have issues with the pop-up window on small screens where the width is very narrow requiring horizontal scrolling even when using landscape orientation.

 

 

Edited March 8 by MowMdown

[Iker]

@MowMdown I have been thinking about reworking the entire snapshot admin dialog, but it is still in the draft phase. However, what you suggest maybe even more problematic than the current approach. Many people using ZFS have tens, if not hundreds, of snapshots in a single dataset, and displaying that inline along with the dataset will be a nightmare for you as a user and for me as the developer.

 

Right now, I'm more inclined to directly present snapshots grouped by date (no more than ten buckets) on the contextual menu and implement another contextual menu for the dataset with all the related actions. But if you folks have any other ideas, I'm more than open to reading them.

[d3m3zs]

Can anyone tell me why when I have ZFS drive in Unassigned Devices - this drive will never spin down.

[Iker]

Some questions that could help to pinpoint the issue source:

• Are the other pool disks sleeping?
• "Last refresh" date changes?
• Do you have other processes/plugins accessing the folders/files stored on that pool?

 

[Renegade605]

On 3/8/2024 at 10:24 AM, MowMdown said:

Well that's to be expected. You've set your share to be moved off of your pool onto the array. No idea why you'd want to snapshot an empty dataset that's never going to have data in it?

 

O.o ... Datasets have settings... Just because I want the data moved to the array doesn't mean I want my settings erased every time that happens?

 

I'd want to snapshot the empty dataset to prevent the dataset from being destroyed. I thought that reasoning was clear. It's empty in the snapshot; it won't be empty forever.

[Renegade605]

On 3/8/2024 at 1:03 PM, Iker said:

Right now, I'm more inclined to directly present snapshots grouped by date (no more than ten buckets) on the contextual menu and implement another contextual menu for the dataset with all the related actions. But if you folks have any other ideas, I'm more than open to reading them.

 

Preferences probably depend a lot on how people are using their snapshots. Manual only, you might want to see them all. I have automatic snapshots and as you guessed, many of my datasets are carrying over 100 snapshots at any given time. I definitely don't want (or need) to see all of those all the time or even in the context menu.

 

Context menu isn't a bad idea, maybe for just the last one? Last two? Should cover being able to see one you've taken manually without cluttering.

 

 

In fact, for settings I preferred a pop-up over the new context menu. Fine for changing one thing, but if you want to change a bunch of settings at once it's a pain, so now I just do that from the terminal.

 

 

I've also noticed that a lot of settings aren't there or not all the options are available. I assume you tried to pick the most commonly used to avoid clutter which makes total sense. But, for example, Quota is there but not Reservation. I pretty much never use one without the other. Recordsize only provides a few options, but any power of 2 from 512 to 1M is valid.

[d3m3zs]

17 hours ago, Iker said:

Some questions that could help to pinpoint the issue source:

• Are the other pool disks sleeping?
• "Last refresh" date changes?
• Do you have other processes/plugins accessing the folders/files stored on that pool?

 

1. Yes, in pool I have one ZFS drive and it is sleeping.

2. No.

3. No. And disk spin up only when I open Unraid Main tab. I can manually spin down disk on Main tab and in few seconds it will spin up again.

[Iker]

8 hours ago, Renegade605 said:

Preferences probably depend a lot on how people are using their snapshots...

 

Fair enough, as I said, this is still a work in progress, and your feedback is very much welcome.

 

@d3m3zs As the other disks are sleeping and the last refresh date doesn't change, I suggest you post the problem on the unassigned devices plugin thread.

[Renegade605]

@Iker Just an additional note, not sure if you're aware, but reporting of zvol properties is a bit wonky:

[frodr]

Creating dataset thru the plugin -> appear as user folder in the Array.

 

Is this as it should be?

 

 

[Sobrino]

Hey everyone,

 

I have an encrypted dataset named "media" on one of my ZFS pools called "data." Perhaps the issue I'm experiencing comes from a misunderstanding, but I would expect the "folder" to be inaccessible when the dataset is locked, which doesn't seem to be the case.

 

Instead, it appears that there are separate files visible when the dataset is in a locked versus unlocked state.

 

When the dataset is unlocked:

- SSH into the console and navigate to /mnt/data/media.

- Create a file named test.txt and write something into it.

- Lock the dataset.
 

When the dataset is locked:

- Running 'ls' on /mnt/data/media shows no files.

- Create a file named test-2.txt and write something else into it.

Now, when I unlock and lock the dataset, I find either test.txt or test-2.txt present.

This is also an issue because the dataset is mounted into Docker containers, which will write files to the "locked dataset" that are not available after I unlock the dataset (and vice versa).
 

Any hints on what I might be doing wrong?

[frodr]

21 hours ago, frodr said:

Creating dataset thru the plugin -> appear as user folder in the Array.

 

Is this as it should be?

 

 

 

Nice if anybody can confirm/deny this? 

 

Cheers,

[Iker]

On 3/16/2024 at 2:22 PM, frodr said:

Creating dataset thru the plugin -> appear as user folder in the Array.

 

When you create a dataset at the root level, Unraid detects it as an user share.

 

On 3/16/2024 at 4:08 PM, Sobrino said:

Any hints on what I might be doing wrong?

 

It's working as intended. It's very simple: When you unlock a dataset, you're also mounting it, so you can access the content in the mountpoint (/mnt/data/media). When you lock the dataset, the mountpoint still exists as a folder, so you can write anything to it because it's a normal folder within the system.

[anibis2]

I have no option to create a snapshot when clicking on the "Actions" menu on a particular pool. I have a second ZFS pool that I use for VM's, it has the Snapshot option. My cache pool does not have this option, it simply has "Move/Rename". Any ideas?

 

 

*EDIT: I figured it out, I had shares and not datasets. Not sure why that happened in the first place, but it's fixed. 

Edited March 22 by anibis2
Resolved

[TallMan206]

I'm currently converting a folder to a dataset. It's a 8TB drive, folder has 6,5TB of data in it. While converting, my disk is full, what happens now? I cannot convert this folder because clearly you need double the space to convert.

 

Please help.

[Iker]

The convert to dataset functionality is just a convenient way of copying the data without having to execute the commands, but you can always do it manually. Create a dataset and the respective child ones, and start syncing each folder to the corresponding dataset from the source using rsync.

[Iker]

New update with the following changelog:

 

2024.03.31

• Change - Convert to dataset reporting on dataset row
• Add - Unraid notifications for Convert to Dataset
• Fix - Compatibility with 6.13 beta
• Fix - for folder listing corner cases
• Fix - for zvols mounting point

[couzin2000]

My ZFS Master Plugin is stuck in "pending" update. There isn't any way I can click the icon to open a contextual menu and stop the app, or anything. As well, CA doesn't allow me to "force" the update, nor can I completely remove the plugin using GUI. 

I wan to know what can be done to either force the update or remove it and reinstall it. I havent really used the plugin yet, so I don't have anything setup except for running SOME drives as ZFS. Removing or forcing an update will not make me lose anything,  I think. Any command line controls I should be aware of?

[dopeytree]

Can anyone help me understand whats going on?

 

I have a 2x 966GB Disk ZFS Pool for CCTV.

 

With no snapshots frigate records  3days of footage on rotation and keeps all events for 14days. This means if an incident isn't detected it can still be manually found in recordings for 72hours. While all detected events are kept for 14days.

 

So with no snapshots it uses around 400-500GB of storage (as 3k quality) but if I turn snapshots on the size on disk seems to increase to 800GB but within frigate it stays the same. 

 

I thought snapshots didn't use much space?

 

Is this a problem with my ZFS configuration?

 

Thanks

 

Edited April 4 by dopeytree