ZFS plugin for unRAID


steini84

Recommended Posts

1 hour ago, BVD said:

 

Whole bunch to go through there, too much to type right now, but a few things to consider:

* Your test is for 256k IO using the random read/write algorithm, with sync enabled.

* The default zfs dataset has a 128k block size (half the test block size), so two write actions for each 1 from fio. With sync, you're having to physically finish and validate the write to disk before continuing, not an ideal workload for HDDs anyway.

* On top of that, weve got a 64 IO depth (which is essentially "how long can my queue be") is essentially halved by the default dataset blocksize; sort of "cancelling it out", down to 32 in effect

 

The most important part though is this - in order to properly test your storage, the test needs to be representative of the workload. I pretty strongly doubt you'll primarily be doing synchronous random r/w 256k IO across some ~20TB of space, but in the event you do have at least some workload like that, youll just ensure that one dataset on the pool is optimally configure to handle it in order to ensure your results are "the best this hardware can provide".

 

Also, would be happy to set aside some time with you still of course! As an FYI (just given the time of your response here), I'm in GMT-5, assuming we're basically opposite hours of eachother, but im certain we could make some time thatd work for us both. You just lemme know if/when you'd like to do so.

 

I'm actually working on some zfs performance documentation geared towards unraid on github currently (going over different containers with recommendations on both how to configure their datasets as well as test+tune, general "databases on zfs" stuff, tunable options from the unraid/hypervisors side and when/how to use them, and so on), and the above post has been enough to kick me in the rear and get back to it. It's been an off and on thing for (months? Hell, idk), but I'll try to share it out as soon as at least *some* portion of it is fully "done". Maybe itll help someone else down the line 👍

 

Thank you for your reply! Time zones are just a social construct, I'm sure we would manage somehow if needed.

 

I also had a feeling the test might be weird, but the idea for me at least was: He has this and that performance with that settings, so I should be around there as well.

 

Wendell has bigger disks (but same amount) and who knows about the rest of the system, but he claims with the same test command ~160MB/s while I was around 20. Of course with the same ZFS settings.

 

Not sure how to test the performance better and how to compare it.

Link to comment
1 hour ago, stuoningur said:

 

Thank you for your reply! Time zones are just a social construct, I'm sure we would manage somehow if needed.

 

I also had a feeling the test might be weird, but the idea for me at least was: He has this and that performance with that settings, so I should be around there as well.

 

Wendell has bigger disks (but same amount) and who knows about the rest of the system, but he claims with the same test command ~160MB/s while I was around 20. Of course with the same ZFS settings.

 

Not sure how to test the performance better and how to compare it.

 

Honestly too much to type - I'll hang loose for a bit while you're scoping this out further and just wait to hear back on whether youd like a second set of eyes on it with ya. Best of luck, and enjoy the journey! 

Link to comment

@stuoningur Finally had some time to sit down and type - 

As I was doing some quick napkin math though thinking about your situation today, some points on the test setup/config:

  • 4 disk raidz1, so 3 'disks worth of IOPs'
  • rough rule of thumb is 100 IOPs per HDD in a raidz config (varies a lot more than that of course
  • the default block size of 128k means 'each IOP is 128k'
  • 128KB * 3 disks * 100 IOPs, equals ~38MB
  • your test was for 256k block size, halving the IO - which results in our ~20MB/s

Outside of the above:

  • Confirm your zpool has ashift set to 12 zfs dataset. Nearly all newer non-enterprise drives (ironwolf being the SMB/NAS market instead) are 4k sectors (w/ 512b emulation). Huge potential overhead depending on the implementation, and really no downsides to this, so it's win/win. Lots of good background information on this out there for further reading if interested
  • Check your zfs dataset's configuration to ensure it's a one-to-one match for what you're comparing against - Wendell did his tests without case sensitivity, no compression, etc
  • Validate your disk health via SMART, ensuring no UDMA/CRC/reallocated sectors/etc are being encountered (which could easily contribute to hugely reduced performance
  • Ensure the system is completely idle otherwise at the time of the test
  • And finally, validate your hardware against the comparison point - Wendell's system had a 32GB l2arc, so the point about ensuring the file tested is bigger than the l2arc miiiiiiight've been one of those 'do as I say, not as I forgot to do' kind of things (he's a wicked busy dude, small misses happen to us all! However I don't think that's the case here, as ~45-60MB/s per drive for a 4 disk z1 is actually pretty average / not exactly unheard of performance levels)

Assuming the config 100% matches (or at least 'within reason'), the rest is unfortunately just going to be going through those steps mentioned earlier, ruling out one by one until the culprit's determined.

Link to comment

Cross-posting here, as I finally made some progress on documenting some ZFS on UnRAID performance related stuff

 

I'm using my vacation week to work on some of this as a passion project - hope it's found to be helpful by some, and just note, it'll continue to grow/evolve as time allows 👍

  • Like 3
Link to comment
On 8/1/2022 at 3:43 AM, gyto6 said:

What do you mean saying "that's happened in a few places already" ? That other functionalities has been annonced but not released, or at least in another upgrade?

Hey, just eluding the various places where Limetech have already eluded ZFS is coming - .e.g the video interview I saw quite a while back now, the poll for next feature which included (a winning) zfs as highest vote etc...

  • Like 1
Link to comment
  • 3 weeks later...

Great stuff! I am running into one issue, i followed spaceinvaders video for native ZFS and at the end he showed how to make a sym link. Which works... kinda. My total drivespace in windows shows up as only remaining disk space on trhe array drive that is used for apps and such, which is about 455 Gb left. When im trying to move my data back, im ruinning into an issue that im moving more stuff then 455 gig so windows just says no. The same issue is visible under my computer, where the share shows as only 455 out 480 free. 

 

I did some test runs and everyhting works fast as expected, data is being moved to the ZFS drives, and the free space on array 1 drive  remains the same. 

 

My question is, how do i get around this limitation and get windows to see that actual zfs space used and free and allow me to copy all my stuff back instead of the drive where sim link is originating from, sort of speak. 

 

UPDATE: Im an idiot, mounted the top level zpool instead on zpool/folder in wiondows so it still passed data along, but calculated wrong thing. all shows up as its supposed to now. Willl leave this up in case someone else is half asleep doing this :)

Edited by TheEugeneKam
Realised im stupid :)
Link to comment

Command line.

 

The useful commands I've noted for the disk based (L2) cache are:

 

Add Cache Drive: zpool add data cache /dev/nvme0n1

 

Info to see how good it's functioning (also good for the in memory cache)

arc_summary

arcstat

 

How to Remove Cache Drive

zpool remove data cache /dev/disk/by-id/ata-xxx

(Not sure if /dev/disk/by-id was what actually removed it in the end)

The second time this command removed it zpool remove DISKNAME /dev/nvme0n1

 

It is sort of diminishing returns but fun to try out because you can actually remove it.  I tried it out for a while but the hit rate was quite low.  Could be good if you had VM's stored on slow disks or something though.  One nice thing is it's persistent now, so it remembers what it cached after a reboot.

  • Upvote 2
Link to comment
17 hours ago, Marshalleq said:

Command line.

 

The useful commands I've noted for the disk based (L2) cache are:

 

Add Cache Drive: zpool add data cache /dev/nvme0n1

 

Info to see how good it's functioning (also good for the in memory cache)

arc_summary

arcstat

 

How to Remove Cache Drive

zpool remove data cache /dev/disk/by-id/ata-xxx

(Not sure if /dev/disk/by-id was what actually removed it in the end)

The second time this command removed it zpool remove DISKNAME /dev/nvme0n1

 

It is sort of diminishing returns but fun to try out because you can actually remove it.  I tried it out for a while but the hit rate was quite low.  Could be good if you had VM's stored on slow disks or something though.  One nice thing is it's persistent now, so it remembers what it cached after a reboot.

You are a God amongst men, thank you! I have an NVME drive that i was using as a boot drive for TrueNas that has no wear so i figured it wont hurt given how easy it is remove or add. 

 

My last issue is tring to write to the pool. I did the sym link as per space invader. Under shares, after creating the sym link i made sure to make that share private, with read/write access to the user i created for this. I can mount the drive but i cant write to it. Gives me the permission error. I am stumped. I tried to do run tools/new permission but that didnt do anything so I am stumped. 

 

Tried to do it with just public access and same issues :( 

Screenshot 2022-08-25 150221.png

 

 

 

Update: 

I tried to change the onwer of the DigitalAssets folder inside digitalassets/zpool to user trying to access it or nobody. Neither seems to stick. It reverts back to root. 

image.thumb.png.d6e0651a44f6d4a1b0046f9f65b30ba7.png

 

I think i got it to work by editing permissions as follows below:

image.png.ec868403201f9fa98936efbe08761b3a.png

 

Is there a security issue if other is set to read and write? Thats the only way i got access to the drive.

Edited by TheEugeneKam
Update
Link to comment

No problem.  I haven't actually ever used the GUI options and my setup may be very slightly different.  For example when I create an array, I set the mount point of the array simply into /mnt - it seems like you've put yours into /mnt/user/zpool.  I am not sure if that's in a guide or what, but it doesn't seem like a sensible way of doing it as you may have permissions problems being that it's a user folder.  Also, clearly you're also talking about SMB sharing.  I found that the unraid smb sharing doesn't work with ZFS really, but luckily zfs has it's own SMB built in - so you can edit the smbextra file in /etc somewhere - sorry can't look it up exactly at the moment - I think it's like /etc/samba/smb.conf/smbextra.conf or something like that.  Don't worry the smb format has examples and is super easy.  Also for me, any zfs shares in unraid under /mnt I just set them to nobody.user and that sorts them out.  having them as root definitely doesn't work.  Just always bear in mind with linux, folders must always be set to 777, and the files can be whatever you need.  Hope that sort of points you in the right direction.

  • Like 1
Link to comment

Something else to note - since you're on linux AND sharing out over SMB, you'll want to ensure you've aclmode set to passthrough, aclinherit to passthrough-x, and likely want to set xattr to 'sa' if not already done (make sure it's at least not set to 'off').

  • Like 1
Link to comment

@Marshalleq

 

So figured out how to get to /etc/samba

have these files there 

lmhosts  

smb-names.conf  

smb-shares.conf  

smb.conf

 

smb-shares.conf has this in it: 

 

zpool]
        path = /mnt/user/zpool
        comment = zpool
        browseable = yes
        # Private
        writeable = no
        read list =
        write list = eugenekam
        valid users =  eugenekam
        vfs objects = catia fruit streams_xattr
        case sensitive = auto
        preserve case = yes
        short preserve case = yes

 

wonder if just changing this to writable = yes 

would do the trick so i dont have to allow others write permission to this folder

 

and smb.conf:
       

 # configurable identification
        include = /etc/samba/smb-names.conf

        # log stuff only to syslog
        logging = syslog@0

        # we don't do printers
        show add printer wizard = No
        disable spoolss = Yes
        load printers = No
        printing = bsd
        printcap name = /dev/null

        # disable aio by default
        aio read size = 0
        aio write size = 0

        # misc.
        invalid users = root
        unix extensions = No
        wide links = Yes
        use sendfile = Yes

        # ease upgrades from Samba 3.6
        acl allow execute always = Yes
        # permit NTLMv1 authentication
        ntlm auth = Yes

        # hook for user-defined samba config
        include = /boot/config/smb-extra.conf

        # auto-configured shares
        include = /etc/samba/smb-shares.conf

 

The symlink path i got from space invader's native ZFS video 

Essentially just changed names form his battlestar to my digitalassets 

 

ln -s /digitalassets/assets/ /mnt/user/zpool/DigitalAssets

 

Once files transfer over im going to try to use this 

 

ln -s /digitalassets/assets/ /mnt/zpool/DigitalAssets

to create a symlink and see if that does anything else. 

ANNNNND nope 

root@DigitalAssets:~# ln -s /digitalassets/assets/ /mnt/zpool/DigitalAssets
ln: failed to create symbolic link '/mnt/zpool/DigitalAssets': No such file or directory

looks like it does need to mount mnt/user/xxxxxxx to work 

ALso if i have both SMB enable, the unraid and ZFS as you mentioned above, would that cause issues. AKA do i need to disable SMB in unraid under settings/network services tab/SMB?

 

@BVD 

My main machine is windows and my other machine is a Mac. 

I dont even know where to find the settings these settings looked around in SMB as well as netweok setting under settings tab and cant seem to see anything. 

Edited by TheEugeneKam
clearing things up
Link to comment

Welp gents, its been a ride. I dont think im ready for unraid until it has official ZFS support. To much hacking and slashing. Community here sure is great, and super helpful, just at the moment cant really feel safe with client work that i cant recreate so back to truenas scale i go :( 

 

Thanks all of you again for all the help.

Link to comment
2 minutes ago, TheEugeneKam said:

Welp gents, its been a ride. I dont think im ready for unraid until it has official ZFS support. To much hacking and slashing. Community here sure is great, and super helpful, just at the moment cant really feel safe with client work that i cant recreate so back to truenas scale i go :( 

 

Thanks all of you again for all the help.

 

 

All good, my friend! Ive often said that pure command line zfs isnt for everyone, and it's not even about "skill" or "ability"; for anyone new to it, its primarily about the time investment required.

 

I've been working on some guides that I hope will make some parts of it make some sense to folks (primarily geared towards performance tuning, but with other stuff occasionally thrown in) - I've added iSCS, NFS, and SMB sharing on zfs to the list if stuff to document, though I've already got a hell of a backlog of stuff to get through so it might be a bit...

 

I primarily use iSCSI and NFS on ZFS, but I've got my wifes mac and my work laptops time machine going there, so it should just be a matter of copying out the config parameters, then expanding in what the options are (and what they mean).

 

I wouldnt give up completely on unraid though if you've got a large media collection, at least if you're down to virtualize unraid - run it in a vm on truenas, pass through a controller, and save all that sweet sweet money by keeping inactive disks from sucking down unnecessary power 😁

  • Like 1
Link to comment
  • 2 weeks later...

Downloaded 6.11.0.r3 update before my trip (from 6.11.0.r2), but didn't reboot...in HI now, and a storm back home knocked out power long enough for server to intiate a shutdown.  It updated, but won't boot unless in safe mode.

 

Things I've tried - renames some/most plugins + reboot normally.  Removed all settings for regular boot except normal (and immou and pcie stuff) + reboot.  Seems it's a plugin, and the only one I haven't renamed is this one. 

 

Question, can I boot into safe mode and reinstall unRAID6-ZFS.plg to check my pools?  I've done ample testing prior, feel confident data is good - just trying to troubleshoot while away to get at least zfs plugin up and running again. 

 

Good thing I setup wireguard on firewall and have access to IPMI...but damn good lesson I need some basic HA on critical apps like bitwarden...phone rebooted, and I can't access my phone's local DB.  Luckily, my wife's iPhone either is different than android or hasn't rebooted - but what a PITA. Haha.

 

I have diagnostics and am going through them, but they were run after I discovered the issue...

 

 

Edited by OneMeanRabbit
Link to comment

It sounds to me like either the server didn't wait for the updated zfs module to get built, or it wasn't able to be built.  The quite solution to that might be to downgrade the server firmware to the previous version and go from there.  That shouldn't be too hard.

 

Failing that, ping either steini84 (or possibly others know) where there is a downloadable matchable version along with where to put it to force the process.

 

Sorry I don't know the process, it may be indicated somewhere in this thread though.

 

Marshalleq

Edited by Marshalleq
Link to comment

Also try removing and re-adding the zfs plugin.  Could also try stable and do it again.  But it does work on at least r3 because I'm running that.  (Sorry not sure what you're aware of here so I'm just going to say it - make sure you wait until the zfs plugin updates it's module before rebooting after an unraid version change).  And if you can boot into normal mode (not safe mode), perhaps you need to drop to the command line and reimport the pools.

Link to comment
7 hours ago, OneMeanRabbit said:

I tired rolling back to 6.11.0.r3 and reboot, no bueno.

Do you have a active Internet connection on boot?

 

I think I don‘t understand completely what you‘ve done so far…

 

What I would do in your case is that you boot into safe mode, turn off autostart, reboot again and boot as usual, remove the ZFS Plugin, reboot again, install the ZFS Plugin and after that successfully finishes you should be able to start the Array (I would recommend that you before starting the Array that you open up a Unraid terminal and see if the command „zpool“ is available).

 

@Marshalleq the modules/package should be built after about an hour after a new Unraid version was released if everything went well.

  • Like 1
  • Upvote 2
Link to comment
19 minutes ago, Digital Shamans said:

About to try the plugin, however since I pressed "Install" button, the app plugin is "Pending" from hours.

At what stage does it stop?

 

Maybe GitHub has again some issues.

Do you have a AdBlocker somewhere on your network?

Link to comment
  • 2 weeks later...

Hello,

 

I've been running the ZFS plugin on Unraid 6.10.3 since few months now and I'm mostly very happy with it.

 

However, I have encountered a very annoying issue that I can't seem to be able to resolve.

I run hourly snapshots on all my datasets in the ZFS pool, and I can see and access them just fine on the server by browsing to the hidden .zfs/snapshot directory, which contains a directory for each snapshot created on that dataset.

I have also set snapdir visibility to visible in ZFS. 

 

To be able to access the dataset on the ZFS pool from the client (MacOS) I use a Samba share that is configured like this:

 

[global]
vfs objects = catia fruit streams_xattr
fruit:aapl = yes
fruit:model = MacSamba
fruit:metadata = stream
fruit:resource = xattr
fruit:encoding = native
fruit:nfs_aces = no
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes
ea support = yes
map archive = no
map hidden = no
map system = no
store dos attributes = no

[ShareTest]
path = /zfs/test
comment = ShareTest
browseable = yes
writeable = yes
valid users = andrea
write list = andrea
guest ok = no
create mask = 0775
directory mask = 0775
vfs objects = catia fruit streams_xattr
veto files = /.DS_Store/
delete veto files = yes

 

When I connect to the Samba share from the MacOS client (but the same happens from Windows) like this:

smb://[IP address of server]/ShareTest

 

I can cd to the hidden /ShareTest/.zfs directory. 

Inside there I can list and cd into the snapshot directory.

However, if I do ls -a inside the snapshot directory I don't see any of the snapshots folders. 

 

andrea@192 snapshot % pwd
/Volumes/ShareTest/.zfs/snapshot
andrea@192 snapshot % ls -a
.	.. 

 

However, if I try to cd into any of the snapshot folders by starting to type the name of the directory and pressing tab to autocomplete, the auto-completion shows all the missing folders and I can cd into any of them.

 

andrea@192 snapshot % cd 2022-0
2022-08-21_18-00/  2022-08-24_08-00/  2022-08-25_20-00/  2022-08-28_15-00/  2022-09-03_22-00/  2022-09-09_23-00/  2022-09-11_10-00/
2022-08-23_22-00/  2022-08-24_23-00/  2022-08-25_21-00/  2022-09-03_12-00/  2022-09-03_23-00/  2022-09-10_17-00/ 

 

As far as I know, when something like this happens is often because I don't have read permissions on the snapshots directories. This isn't the case this time, though.

 

What's even more odd is that if I do the following:

- Try to (unsuccessfully) list the snapshots folders for a given share

- Restart the Samba service with this command: smbcontrol all reload-config

- Try again to reconnect to the share and list the snapshot folders

 

It almost always shows all the snapshots this time!!

This only applies to that given share (and not the others) and only until I reboot the server or the client.

For example, if I connect to a different share and I try to list the snapshots, I get an empty list again and I have to restart the Samba service once more.

 

Does anybody know what's causing this issue?

Link to comment
On 8/2/2022 at 3:18 AM, BVD said:

@stuoningur Finally had some time to sit down and type - 

As I was doing some quick napkin math though thinking about your situation today, some points on the test setup/config:

  • 4 disk raidz1, so 3 'disks worth of IOPs'
  • rough rule of thumb is 100 IOPs per HDD in a raidz config (varies a lot more than that of course
  • the default block size of 128k means 'each IOP is 128k'
  • 128KB * 3 disks * 100 IOPs, equals ~38MB
  • your test was for 256k block size, halving the IO - which results in our ~20MB/s

Outside of the above:

  • Confirm your zpool has ashift set to 12 zfs dataset. Nearly all newer non-enterprise drives (ironwolf being the SMB/NAS market instead) are 4k sectors (w/ 512b emulation). Huge potential overhead depending on the implementation, and really no downsides to this, so it's win/win. Lots of good background information on this out there for further reading if interested
  • Check your zfs dataset's configuration to ensure it's a one-to-one match for what you're comparing against - Wendell did his tests without case sensitivity, no compression, etc
  • Validate your disk health via SMART, ensuring no UDMA/CRC/reallocated sectors/etc are being encountered (which could easily contribute to hugely reduced performance
  • Ensure the system is completely idle otherwise at the time of the test
  • And finally, validate your hardware against the comparison point - Wendell's system had a 32GB l2arc, so the point about ensuring the file tested is bigger than the l2arc miiiiiiight've been one of those 'do as I say, not as I forgot to do' kind of things (he's a wicked busy dude, small misses happen to us all! However I don't think that's the case here, as ~45-60MB/s per drive for a 4 disk z1 is actually pretty average / not exactly unheard of performance levels)

Assuming the config 100% matches (or at least 'within reason'), the rest is unfortunately just going to be going through those steps mentioned earlier, ruling out one by one until the culprit's determined.

Thank you for taking time to reply! I know it has been a few days... I stopped caring about ZFS for a while but gave it today another shot with different configs, trying to incorporate your tips and also did some more research. I also stopped caring about that benchmark, I just used it as it was more or less comparable to my setup. Having said that, I of course "benchmarked" it, but just by samba file transferring and running VMs on it and stuff. Samba maxxed out at about 90MB/s (or about 750Mbit/s) which is still a bit slower than hoped for. Realistically it's fast enough, but still noticeably slower than a normal unraid+cache setup.

 

Also getting samba to work, I tried the symlink approach, has been quite annoying as often I would get errors about permissions.

 

Maybe I try truenas after my vacation which should be more guided in setup and see if maybe I still did something wrong somewhere.

Link to comment

One other thing immediately comes to mind - you're mounting you're zpool directly in /mnt, right?

 

If not, do that and re-test - putting zfs inside the virtual directory unraid uses to merge the disparate filesystems of multiple disks introduces a massive number of unaccounted for variables, and even unraid itself doesnt directly mount physical filesystems directly on top of virtual ones.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.