Hello everyone,

 

I deleted and have  up my CACHE drive and an external backup drive to ZFS. 

 

When setting up a RAID in Unraid settings, I also selected compression and encryption for this ZFS pools.

 

 

I assume that this has now been done for all data. (not much on it yet)

 

Now you can also create datasets with your ZFS Master, which also have the advantage of a snapshot. and others...

 

But what I don't quite understand yet:

Compression and encryption can now be selected here again.

Is the data still encrypted with passphrase selected in UNRAID drive settings, if I don't select anything there? (do not touch the "no") 

 

If I select encryption for a dataset and enter the same PassPhrase as on unraid, it will subsequently have a lock symbol in the ZFS master.... like on "medien".

Then of course I'm sure that it was encrypted,.... but twice ? Is backup and downloads also encrypted ?

 

Disadvantage: I then have to decrypt this data set separately with a password if I restart UNRAID... (even it is the same password as in Unraid encryption settings)

 

 

 

I hope one of you understands this better than I do.

Greetings

Mike

 

 

Hi @mmm77, most of your assumptions are correct, Unraid can create an encrypted pool; this process encrypts the entire disk using LUKS so your data is always encrypted. On a note, currently, Unraid doesn't support ZFS native encryption for pools.

 

Answering your question about data being encrypted twice, Yes, ZFS is not aware of the LUKS encryption, and in any case it allows you to have different keys/passphrases for different datasets, even if they are descendants of an encrypted one; moreover, ZFS allows you to have different settings for different datasets, that includes, compressions, mount points, etc; it's a very flexible system.

 

I recommend you to read the following article https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/.

Thank you for your help @Iker.  Tried to understand , but I will also look for a German explanation to understand better.

What I now did:

Move Data of my cache ZFS drive to array.

Formated cache again as ZFS without compression and encryption set in unraid settings...

 

I understand how I can enable compression and encryption in every dataset, but I do not understand how I can enable this for the zpool itself, so the datasets can inherit the same passphrase and I do only have to unlock once.

Okay, as child of the first Dataset this should work....

But Unraid creates the shares for first and second storage only in the home directory?  First storage is this ZFS drive and second storage is my normal "xfs encrypted" array.

 

 

Thanks !

 

Mike

 

Tried to solve myself: 

What I now did: create 1 Dataset with settings I need. 

Created child Datasets in ZFS Master.

To get this dataset in root I found the solution to change the mountpoint of this dataset with ZFS Master from /cache/pool/medien to /cache/medien

This way all should work now.

Only "problem" is that this new mountpoints /cache/medien will be shown in ZFS Master as a normal folder too.

 

 

 

 

EDIT:  It seems that it is not a correct child of the Dataset, when I create it in ZFS Master as described above... It does not unlock together..  This does only work when I create the child dataset in terminal with  "zfs create cache/pool/medien"

After restart the new mountpoint I set to cache/medien does also not work... The folder is empty.

I have to set another mountpoint eg. cache/medien2 and set it back to cache/medien.  Then it works...

 

 

Edited October 28, 20241 yr by mmm77

@Ikersorry, don´t want to bother you.... I am really a noob, and try to help myself with google.

 

Again the issues for me:  

1) Child Dataset only unlock automatically when I create them on terminal with eg. "zfs create ssd/daten/isos", not in ZFS Master as new "child" dataset ssd/daten/isos

 

2) I can mountpoint of child datasets with ZFS Master to root of the drive when needed (eg. isos / domains) , but it is not automatically mounted, when I unlock the parent dataset in ZFS Master. The folder is empty... the parent is normally mounted... I have to use on terminal command "zfs mount -a"  afterwards (thanks google) and it works then.

 

3) Child Dataset mounted to root of drive are also shown as normal folder in ZFS Master. 

 

So maybe something to improve for you or it is wanted as it is...

Only need to know... 

Edited October 28, 20241 yr by mmm77

Don't worry; it's not a problem, but I'm not so sure what type of functionality you are asking for. Here is the thing: Unfortunately, Unraid doesn't support the native ZFS encryption scheme (yet); however, the LUKS encryptions work just fine; if you need extra privacy, it's possible, as you have suggested, to create additional datasets with different keys, but you have to unlock them manually.

 

About the mount point listed as a folder, unfortunately, it is very hard to work in these little cases, as you can mount a dataset anywhere in the system, but I see what I can do.

Thanks for your answer... Now I understand it clearly... I think you are right... I will use LUKS encryption with the same password for all... 

I will disable compression in unraid to avoid double compression and enable it for each dataset. (so I will choose better and slower compression for some seldom used and changed data)

Thanks again 

 

 

On 9/5/2024 at 8:41 PM, Entxawp said:

If anyone ever want to clone a snapshot from one pool to another pool here is how you can do so on Unraid 6.12.x with ZFS Master.

 

zfs send pool1/dataset@snapshot_name | zfs receive pool2/dataset

 

It will look something like this in practice:

zfs send bxtyr/appdata@2024-09-02-190650 | zfs receive ark/test

 

If the dataset "test" already existed you'll have to add the -F

zfs send bxtyr/appdata@2024-09-02-190650 | zfs receive -F ark/test

 

 

You'll have to mount the dataset if it's not available
 

root@Example:~# cd /mnt/ark/test
root@Example:~# bash: cd: /mnt/ark/test: No such file or directory
root@Example:~# zfs mount ark/test
root@Example:~# cd /mnt/ark/test
root@GExample:/mnt/ark/test# 

 

 

What could be wrong with my output. I'm trying to clone a snapshot from one zfs pool to another.

root@Tank:~# zfs send cache/appdata@2024-11-01-111541 | zfs receive -F cache_backup/snapshots_appdata
cannot receive new filesystem stream: pool must be upgraded to receive this stream.
warning: cannot send 'cache/appdata@2024-11-01-111541': signal received
root@Tank:~# 

 

Edited November 3, 20241 yr by yogy

The receiving pool must be upgraded to support the operation as the error states. Reference https://openzfs.github.io/openzfs-docs/man/master/8/zpool-upgrade.8.html

That worked, but cannot do it via GUI (Admin Snapshots) 😕

I am thinking about testing out the latest Unraid 7.0 beta 4 and was wondering if i should wait?  Is there anything i need to watch out for since i am using your ZFS Master Plugin. 

 

Thank you.

Not really, the plugin is compatible with the latest beta (At least in my test lab). If you find any problems or issues, please let me know, as I'm currently working on a new release.

On 7/13/2024 at 1:13 AM, Iker said:

I tried the 6.13 beta that got changed to 7.0 beta; there were some changes in styles that are not yet definitive, I'm waiting for another RC version or beta to update all the styles as other changes may be introduced.

Love your plugin, thank you.

 

Just wanted to see if you put anymore though into updating the styling now that 7 is at beta 4? Trivial for sure, but a small continual annoyance. Assuming it maybe fell off the radar.

Edited November 9, 20241 yr by cybrnook

I'm already working on fixing the styling for unraid 7 and a couple of improvements.

Here is a new update with the following changelog:

 

2024.11.10

• Fix - Style for Unraid 7
• Add - Properties extraction for ZFS Volumes
• Fix - for folder listing corner cases

 

For the next version, I'll try to integrate the plugin with ZnapZend, initially only to check backup plans for the datasets and, if everything goes well, also to create backup plans from the GUI.

All of my snapshots disappeared with the new version. They come back if I disable Lazy Load and manually refresh.

 

Thanks for the report. There is already a new version with a fix for the bug.

Can we add the function of periodic snapshot and deletion of historical snapshot?

I run user script mostly to make snapshots...



I haven't seen any issues with zfs snapshots...

 

2 hours ago, 疯子哥哥 said:

Can we add the function of periodic snapshot and deletion of historical snapshot?

That's the idea with the ZnapZend integration.

Question/Scenario....

 

If we have app data on a ZFS pool as a dataset, with the inner folders also as individual datasets, if we need to move the files from the pool to the array to do some drive maintenance or relocate things, does the plugin provide the capability to move the datasets as blocks rather than individual files? Would the array drives also need to be formatted as ZFS for that to work?

 

I just went through shifting stuff around and moving 5,000,000 smallish files ended up taking ages. Moving as a block would speed this up massively.

Edited November 14, 20241 yr by aglyons

The plugin doesn't provide that functionality... yet; however, if both filesystems are ZFS you can use zfs send | zfs recv commands. Those work at the block level, so they are very efficient and have linear performance no matter how many small files you have. If the destination filesystem is not a ZFS filesystem, you have to copy the files; there is no other way.

12 hours ago, Iker said:

The plugin doesn't provide that functionality... yet; however, if both filesystems are ZFS you can use zfs send | zfs recv commands. Those work at the block level, so they are very efficient and have linear performance no matter how many small files you have. If the destination filesystem is not a ZFS filesystem, you have to copy the files; there is no other way.

I'm just getting my ducks in a row on a plan. If I convert each disk in the array to ZFS, then I should be able to use those commands to move datasets from the assigned cache pool back to the array. Does this function in any way f'up the FUSE system and corrupt the parity? Do the parity drive(s) need to be in ZFS as well? Currently, the parity drives show no FS at all.

 

My ultimate goal is to reduce the time it takes to move data around in the event of maintenance. I went through this recently, and our entire media system and all my home lab services were down for about 36 hours, just waiting for files to be moved. I'm not really looking for 100% redundancy, but I think that downtime could be cut by 75% just by moving data as block storage rather than at a file level. Hopefully, ZFS can be used in this situation.

1 hour ago, aglyons said:

If I convert each disk in the array to ZFS

Be prepared for serious performance issues if you do that. The way parity works means that the background stuff ZFS does to try to increase performance causes slowdowns instead of increases. For now, I'd keep array drives XFS, and only use ZFS in pools.

Parity does not care what file system is used on array drives as it works at the raw sector level so has no understanding of the file systems being used.

 

it is worth noting that at the moment using ZFS on array disks is known to not perform well.