ZFS plugin for unRAID


steini84

823 posts in this topic Last Reply

Recommended Posts

That doesn't seem to make a lot of sense does it.  I assume you know about zpool upgrade and that shouldn't be impacting this.  Assuming you're using the plugin, perhaps just uninstall and reinstall the zfs plugin and reboot.  Also make sure this is the latest unraid version as recent versions were problematic.  Also, if your docker is in a folder rather than in an image I would suggest trying it as an image.  Docker folders on ZFS seem to be hit and miss depending on your luck.  Can't think of anything else to try right now.

Link to post
  • Replies 822
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

What is this? This plugin is a build of ZFS on Linux for unRAID 6   Installation of the plugin To install you copy the URL below into the install plugin page in your unRAID 6 web g

Built zfs-2.0.0-rc7 for unRAID-6.8.3 & 6.9.0-beta35   Great to see that unRAID is finally adding native ZFS so this might be one of the last builds from me   And yes, i´m alre

For anyone trying to set up zfs-auto-snapshot to use Previous Versions on Windows clients:   I placed the zfs-auto-snapshot.sh file from https://github.com/zfsonlinux/zfs-auto-snapshot/blob/

Posted Images

2 hours ago, Marshalleq said:

That doesn't seem to make a lot of sense does it.  I assume you know about zpool upgrade and that shouldn't be impacting this.  Assuming you're using the plugin, perhaps just uninstall and reinstall the zfs plugin and reboot.  Also make sure this is the latest unraid version as recent versions were problematic.  Also, if your docker is in a folder rather than in an image I would suggest trying it as an image.  Docker folders on ZFS seem to be hit and miss depending on your luck.  Can't think of anything else to try right now.

ZFS was not responsible for the problem.  I have a small cache drive and some of the files for docker and VM's still comes from there for startup.  This drive didn't show up on boot.  Powering down, and making sure this drive came up, resulted in VM and docker behaving normally.  I need to get all appdata files moved to ZFS and off this drive as I am not using it for anything else.

Link to post

Just to confirm, is there "ANY" difference between running ZFS compiled into the kernel via ich777 kernel builder vs using the ZFS plugin ?

reason behind question:
As i need the gnif vendor reset for my AMD gpu, which is only available via the kernel build option , i moved zfs to the kernel build option some time ago at the same time for convenience.
After going through a few version upgrades , where during the process i loose zfs (as step one is to install a clean unraid build and only then compile a new kernel with the docker) and all my dockers/vm etc are on zfs which is then unavailable, i have to go through complex and error prone temporary hoops to do this comfortable.
So in my setup i think of moving back to the ZFS plugin and leave the kernel build only for the vendor reset patch part so i can keep zfs working during the entire upgrade process when i move up (or down) unraid versions.

Link to post

Kernel build is more flexible and you can compile things in as you know. Plug-in is less flexible In that regard and simpler when kernels change. Yeah each has its pros and cons.

 

But yes as far as zfs goes I think there is no difference other than occasional version mismatches which are usually inconsequential. 

Link to post

update:

guess i stick with kernel build as i can not get the plugin to load on a compiled 6.9.2 kernel (clean with only the gnif vendor reset as extra module) vs the stock 6.9.2 kernel where the plugin works fine. Maybe some slight version diff or something but during the modprob load of the zfs module the kernel crashes. If i include zfs during build zfs works fine (of course plugin removed then). 

Nevermind. Will stick with a slighly more complex workflow then. No huge issue as updates are not that often.

Link to post
11 hours ago, Marshalleq said:

Ah yeah, that annoying app data folder that sometimes insists on creating itself in a location not set in the defaults!

I've attempted to move the docker image to ZFS along with appdata.  VM's are working.  Docker refuses to start.  Do I need to adjust the BTRFS image type?

 

Correction, VM's are not working once the old cache drive is disconnected. 

 

image.thumb.png.50bdd6a4cabedd82c21b2881236ad23f.png

Edited by tr0910
Link to post
  • 3 weeks later...
Posted (edited)

Not claiming to have it setup properly (I think docker still puts something on the unraid array due to btrfs vDisk setting) - but here's my setup.  Both dockers & docker.img are on a zfs dataset & running fine with snapshots.  

 

Our setups look very similar, so the only thing I remember having to fix when I initially moved it was to delete the docker.img & let it rebuild itself - got the fix here.

 

image.png.3b8415686ac484a7e1cea6e81f1ce50b.png

Edited by OneMeanRabbit
Link to post
Snapshots for VMs:
Better to perform in running mode (online) or stopped mode (offline)?

Just set up automatic snapshots using something like Sanoid on the running vms. https://forums.unraid.net/topic/94549-sanoidsyncoid-zfs-snapshots-and-replication/

Of course it’s always better to turn off the VM if you are doing something like a migration.


Sent from my iPhone using Tapatalk
Link to post
Not claiming to have it setup properly (I think docker still puts something on the unraid array due to btrfs vDisk setting) - but here's my setup.  Both dockers & docker.img are on a zfs dataset & running fine with snapshots.  
 
Our setups look very similar, so the only thing I remember having to fix when I initially moved it was to delete the docker.img & let it rebuild itself - got the fix here.
 
image.png.3b8415686ac484a7e1cea6e81f1ce50b.png

The problem is with using a newer version of zfs than 2.0.0. I tried saving the img file on ZFS (2.0.4) and both btrfs and xfs locked up the system. Something to do with the loopback mount. Too bad I could not use the folder mapping and I always got an error that Docker service could not be started.

I moved the docker.img to my cache drive and I don’t care too much since it’s disposable. I think I will make latest zfs the default for the next upgrade to unRaid and add a disclaimer that you cannot save the docker.img on ZFS if the problem will still be present.


Sent from my iPhone using Tapatalk
Link to post

So I did something either very stupid or very genius, feel free to comment:

- Installed the unraid ZFS plugin
- got familiar with the command line operations and ZFS in general
- Tuned using fio for large file storage, went for ashift 12 and recordsize 1M
- Created a raidz pool with 4 pieces of rust and a bunch of datasets (mounted at root, not /mnt/user)
- Configured docker for directory on ZFS, reinstalled my docker apps, tweaked some paths and everything up and running ok
- Limited arc size to 2 Gb of memory (sorry need my memory for docker/vm)
- Happily rsyncing terrabytes of data back

so far so good, but my fingers are getting blue, time to get some GUI, lets do something crazy ....

- installed a freebsd VM with truenas latest version, 10G virtio image and 4 Gb of Ram
- After It was up running shutdown and modified the XML to pass through my ZFS block devices (not the whole controller)
- Started the truenas VM, pretended a nose bleed and just imported the ZPOOL which is also imported and in use on unraid

- During import I got one warning: 
image.thumb.png.9cfb685dc23b6e8c5377f6f692fb46c1.png


To my utter surprise it imported and I got the nice truenas gui available for managing my snapshots.

Seems to work so far, any objections anybody?

Link to post
9 minutes ago, praaphorst said:

So I did something either very stupid or very genius, feel free to comment:

- Installed the unraid ZFS plugin
- got familiar with the command line operations and ZFS in general
- Tuned using fio for large file storage, went for ashift 12 and recordsize 1M
- Created a raidz pool with 4 pieces of rust and a bunch of datasets (mounted at root, not /mnt/user)
- Configured docker for directory on ZFS, reinstalled my docker apps, tweaked some paths and everything up and running ok
- Limited arc size to 2 Gb of memory (sorry need my memory for docker/vm)
- Happily rsyncing terrabytes of data back

so far so good, but my fingers are getting blue, time to get some GUI, lets do something crazy ....

- installed a freebsd VM with truenas latest version, 10G virtio image and 4 Gb of Ram
- After It was up running shutdown and modified the XML to pass through my ZFS block devices (not the whole controller)
- Started the truenas VM, pretended a nose bleed and just imported the ZPOOL which is also imported and in use on unraid

- During import I got one warning: 
image.thumb.png.9cfb685dc23b6e8c5377f6f692fb46c1.png


To my utter surprise it imported and I got the nice truenas gui available for managing my snapshots.

Seems to work so far, any objections anybody?

please share ur xml !

Link to post
2 minutes ago, Dtrain said:

please share ur xml !

Sure but remember this is highly experimental, you might wreck your pool for all I know, I've got everything backup-ed, so you are warned here.

For each block device used in your zpool add it to your VM using the following XML fragment under the <devices> section:

  <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/sdb'/>
      <target dev='hdd' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>

can also use /dev/by-id just be sure ;-)

Link to post
On 5/23/2021 at 7:43 PM, steini84 said:

Of course it’s always better to turn off the VM if you are doing something like a migration.

Ok, it's better but is there any knowledge or experience on whether it's possible?

Maybe someone else with answers besides steini84?

Link to post
2 minutes ago, JoergHH said:

Ok, it's better but is there any knowledge or experience on whether it's possible?

It's possible, but the VM will be in a crash consistent state, i.e., same thing as pulling the plug, so it can be done but there's a always a risk, I for example make daily snapshots of my VMs online (with btrfs but the principle is the same) but also do snapshots with the VMs shutdown at least once a week so I have more options.

Link to post
Posted (edited)

I forked a script called borgsnap a while ago to add some needed features for Unraid and my use-case.  This allows you to create automated incremental-forever backups using ZFS snapshots to a local and/or remote borgbackup repository.  I've posted a guide here.

 

This includes pre/post snapshot scripts so you can automate shutting down VMs briefly while the snapshot is taken.

Edited by jortan
  • Like 1
  • Thanks 1
Link to post
On 4/29/2021 at 10:36 AM, glennv said:

update:

guess i stick with kernel build as i can not get the plugin to load on a compiled 6.9.2 kernel (clean with only the gnif vendor reset as extra module) vs the stock 6.9.2 kernel where the plugin works fine. Maybe some slight version diff or something but during the modprob load of the zfs module the kernel crashes. If i include zfs during build zfs works fine (of course plugin removed then). 

Nevermind. Will stick with a slighly more complex workflow then. No huge issue as updates are not that often.

 

Just found out today from Ich777 that the gnif/vendor-reset patch , when compiled in the kernel , prevents plugin kernel modules such as ZFS to load (can as in my case , crash the kernel). This may change soon depending ongoing dev/testing efforts. 

So for now a good to know gotcha !!

If you need ZFS together with the gnif patch, you need to compile them both at the same time in the kernel for now.

Link to post
  • 2 weeks later...

Thank you for this Plugin.  I have been using it for a few months now and it's great!  It gave me the option to have fast storage that is durable on Unraid.  The Unraid Array is great, but too slow for big 10G Ethernet transfers.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.