Jump to content
steini84

ZFS plugin for unRAID

533 posts in this topic Last Reply

Recommended Posts

Posted (edited)

Is there a version of the ZFS plugin that works with 6.9.0 beta22? 

 

Installing from Community App gets the one that works with 6.8.3

 

Unsupported kernel detected!
ZFS not installed! - Please follow this post https://forums.unraid.net/topic/41333-zfs-plugin-for-unraid/ and reinstall the plugin when this version of unRAID is supported

 

Edited by tr0910

Share this post


Link to post
Is there a version of the ZFS plugin that works with 6.9.0 beta22? 
 
Installing from Community App gets the one that works with 6.8.3

Linux 5.7 is not yet supported

https://github.com/openzfs/zfs/releases/tag/zfs-0.8.4

I have not tried to build it because of that, but I’ll start a build now and see what happens.

But a great quote from the release notes from the 9.3 beta
“We are also considering zfs support.”


Sent from my iPhone using Tapatalk

Share this post


Link to post
1 hour ago, tr0910 said:

Is there a version of the ZFS plugin that works with 6.9.0 beta22? 

 

Installing from Community App gets the one that works with 6.8.3

 


Unsupported kernel detected!
ZFS not installed! - Please follow this post https://forums.unraid.net/topic/41333-zfs-plugin-for-unraid/ and reinstall the plugin when this version of unRAID is supported

 

Built ZFS from the latest master for 6.9.0-beta22. You can re-install the plugin to try it out.

I would not trust it for anything important, but then again you should probably not be running the beta version of Unraid on your production server :)

Share this post


Link to post
21 hours ago, steini84 said:

Built ZFS from the latest master for 6.9.0-beta22. You can re-install the plugin to try it out.

I would not trust it for anything important, but then again you should probably not be running the beta version of Unraid on your production server :)

I'm playing with it on a test server on 6.9.0 b22.  It installed and created a simple test pool so far.

Share this post


Link to post

I'm currently using this plugin on 8.0-1 .  I'm not really concerned with 'trusting it' given ZFS itself is not in beta.

 

@steini84am I reading right that your ZFS version is 8.0-1 when the latest stable is 0.8.4?  And interestingly your plugin says it's on 8.2.  This doesn't seem right.

 

# zfs --version
zfs-0.8.0-1
zfs-kmod-0.8.0-1

 

cat /sys/module/zfs/version
0.8.0-1

Share this post


Link to post
4 minutes ago, Marshalleq said:

I'm currently using this plugin on 8.0-1 .  I'm not really concerned with 'trusting it' given ZFS itself is not in beta.

 

@steini84am I reading right that your ZFS version is 8.0-1 when the latest stable is 0.8.4?  And interestingly your plugin says it's on 8.2.  This doesn't seem right.

 

# zfs --version
zfs-0.8.0-1
zfs-kmod-0.8.0-1

 

cat /sys/module/zfs/version
0.8.0-1

Ok let me try to make it clear :)

 

I just updated the plugin to 1.0 since new builds are not dependant on an update to the plugin. On boot the plugin checks for available builds and installs the latest one available for Unraid 

 

The latest build for Unraid 6.8.3 is OpenZFS 0.8.4 and I usually only build from the releases (see them @ https://zfsonlinux.org/)

 

Since Unraid 6.9 beta 22 is running on Kernel 5.7, which is not supported in OpenZFS 0.8.4, I made a build from the latest master (commit 2e6af52). I hoped that some changes were already in the master branch that added 5.7 support since it is the most up to date code.

 

The confusing part about the master branch of OpenZFS is that you (apparently) don't get a new version tag unless there is a release. So every build from the master branch since 0.8.0 has been marked as 0.8.0. See here: https://github.com/openzfs/zfs/blob/master/META 

I could change it of course, but I don't see a point in that since it could make things even more confusing and this is just a test build for a beta version of Unraid.

 

Hope that makes sense

Share this post


Link to post

OK, so OpenZFS from here 0.8.4 is not supported on kernel 5.7, so you're using master branch.

 

Master branch is newer? but has older version number.

 

I have to say, that was unexpected, but thanks for clarifying! :)

Share this post


Link to post
OK, so OpenZFS from here 0.8.4 is not supported on kernel 5.7, so you're using master branch.
 
Master branch is newer? but has older version number.
 
I have to say, that was unexpected, but thanks for clarifying!

Yes excactly. The master branch is still using version number 0.8.0 and only the releases get a updated version number.


Sent from my iPhone using Tapatalk

Share this post


Link to post
On 6/17/2020 at 2:48 PM, Marshalleq said:

Persistent l2arc!  I hadn't noticed that!  It will be a great feature - I'm running the non-persistent one on an NVME drive and it does make a huge difference for VM's and such - actually combined with a decent L1Arc makes VM's and dockers on mechanical HDD's very usable again.  ZFS sure is magic.

Just learning ZFS, so pardon the dumb questions about l2arc.

 

Can you describe your device setup for this?  And how you are using it?  Presently I have my VMs on a single SATA SSD cache drive using btrfs.  I don't have NVMe slots on this Intel 2600cp2 MB so I won't be able to go full speed, but I have 64gb ECC RAM installed.  How should I configure zfs for best VM performance assuming that I move my VMs from btrfs cache drive to some kind of zfs pool. Will ZFS provide a burst of speed to my aging hardware?

Share this post


Link to post

Hey - yeah it's quite a lot to learn isn't it!

 

I don't see l2arc benefitting your VM's if it's already entirely on an SSD and you don't have anything faster other than RAM.  However, if you've got a lot of spare RAM, then the good news is you can increase your L1ARC (which is actually more efficient than L2 anyway) and use that.

 

Essentially the arc, is a clever ZFS only read cache that performs a lot better than standard read caches.  What this means is for things that are on disk and meet the eligibility criteria, they will be read from ram (or l2arc if you have it) instead of from disk.  So in that sense, changing your VM's to ZFS on an SSD may get you a slightly greater performance, once they're up and running.

 

I like the persistent l2arc idea, because this should also then apply to newly started VM's as well.

 

For me I had an INTEL 128GB NVME drive and set up the l2arc to use that.

 

To get stats on how well the arc is working you can use arc_stats or arc_summary in the console and check the cache hit ratio.  Typically I find my l1arc to be at 100% and my l2arc to be at around 70% which is very decent.

 

My setup is:

1TB INTEL Enterprise SSD x1 with ZFS (mostly docker and unimportant VM's)

2x8TB Seagate Enterprise Capacity HDD's in a ZFS Raid 1 pool (Various Data, VMS and important dockers)

1x128GB NVME drive (L2arc).

 

If you come across something called a ZIL - ignore it, in my experience it's very unique cases and mostly it will actually slow your system down unless you know what you're doing.

 

I also have znapzend for backups for the things on the SSD since I only have one of them.  Znapzend is another plugin that utilises ZFS replication which is more performant and clever than even rsync.

 

Hope that helps a little! :)

Share this post


Link to post

I'm not really disappointed in the performance with btrfs.  I may just reformat and run this drive as zfs and let that be it.

Share this post


Link to post

Of course, BTRFS / XFS / ZFS doesn't really mattter too much regarding performance if compared on the same hardware, but if putting your VM's on slow HDD's ZFS helps with that in the form of cache.  My decision to switch to ZFS was due to three failures surrounding BTRFS, which have never happened with the same hardware on XFS or ZFS, and ZFS has the advantage of recovering from corrupted data as well as lost drives.  Running VM's on HDD for me is just because my SSD is not mirrored and the HDD's have more space - so it works well as a solution for me.  The L2ARC is amazing for that.

Share this post


Link to post

ZFS lockup Array stop and restart

I have a 6.8.3 production server that has ZFS installed recently.  I took a single disk and set it up with ZFS for testing as follows.

root@Tower:/# zpool create -m /mnt/ZFS1 ZFS1 sde

The server behaved as expected and the ZFS1 pool was active.  I then took the unRaid array offline and performed some disk maintenance.  I didn't do anything to unmount ZFS1.  By mistake, the ZFS disk was also pulled from the server and reinserted into the hotbay.  Bringing the server back online resulted in ZFS being locked up.  The ZFS and ZPOOL commands would execute, but a ZFS list command would result in a hung terminal window.  ZFS import also locked the terminal window.  The unRaid server was working fine, but to recover ZFS, the only thing I could do was reboot the server.  The ZFS pool came back online after reboot.

 

I expect that I needed to perform some ZFS commands on array stop and restart?  What should I have done?  Is there any way to recover a locked up ZFS without reboot?

Share this post


Link to post
15 hours ago, tr0910 said:

ZFS lockup Array stop and restart

I have a 6.8.3 production server that has ZFS installed recently.  I took a single disk and set it up with ZFS for testing as follows.


root@Tower:/# zpool create -m /mnt/ZFS1 ZFS1 sde

The server behaved as expected and the ZFS1 pool was active.  I then took the unRaid array offline and performed some disk maintenance.  I didn't do anything to unmount ZFS1.  By mistake, the ZFS disk was also pulled from the server and reinserted into the hotbay.  Bringing the server back online resulted in ZFS being locked up.  The ZFS and ZPOOL commands would execute, but a ZFS list command would result in a hung terminal window.  ZFS import also locked the terminal window.  The unRaid server was working fine, but to recover ZFS, the only thing I could do was reboot the server.  The ZFS pool came back online after reboot.

 

I expect that I needed to perform some ZFS commands on array stop and restart?  What should I have done?  Is there any way to recover a locked up ZFS without reboot?

I would like to hear if someone knows the answer to this. This has happened to me using a USB disk (power failure, disconnected etc.). On Linux (unRAID and Debian) the server works fine, but I get incredibly high load and nothing I try works to reset ZFS to a useful state. I could not even reboot normally and had to force a reboot. In FreeBSD this is similar, but since the load is calculated differently there the signs were not as obvious, but pretty much the same problem.

 

 

Share this post


Link to post
8 hours ago, steini84 said:

I would like to hear if someone knows the answer to this. This has happened to me using a USB disk (power failure, disconnected etc.).

So the conclusion is that ZFS is extremely sensitive to disconnections and does not recover on reconnection.  Have you ever seen data corruption as a result of these issues??

Share this post


Link to post
So the conclusion is that ZFS is extremely sensitive to disconnections and does not recover on reconnection.  Have you ever seen data corruption as a result of these issues??

No never


Sent from my iPhone using Tapatalk

Share this post


Link to post
On 6/26/2020 at 10:26 AM, steini84 said:

I would like to hear if someone knows the answer to this. This has happened to me using a USB disk (power failure, disconnected etc.). On Linux (unRAID and Debian) the server works fine, but I get incredibly high load and nothing I try works to reset ZFS to a useful state. I could not even reboot normally and had to force a reboot. In FreeBSD this is similar, but since the load is calculated differently there the signs were not as obvious, but pretty much the same problem.

 

 

Hey @steini84,

I ran into a similar problem, I don't know if you seen it already but I created the container to build a custom Kernel/Images for Unraid with nVidia/DVB/ZFS builtin without any plugin needed.

This is the default behaviour of ZFS since it goes to a 'wait' state read this: Click

Also this thread from L1 should make it a little bit clearer: Click (a little old but still up-to-date)

 

Hope this helps, this is all that I've found from my research.

 

EDIT: I personally don't use ZFS on Unraid because it's not my usecase... but as said above I ran into the same problem after accidentally disconnecting a drive from a running test ZFS pool.

Share this post


Link to post
Hey @steini84,
I ran into a similar problem, I don't know if you seen it already but I created the container to build a custom Kernel/Images for Unraid with nVidia/DVB/ZFS builtin without any plugin needed.
This is the default behaviour of ZFS since it goes to a 'wait' state read this: Click
Also this thread from L1 should make it a little bit clearer: Click (a little old but still up-to-date)
 
Hope this helps, this is all that I've found from my research.
 
EDIT: I personally don't use ZFS on Unraid because it's not my usecase... but as said above I ran into the same problem after accidentally disconnecting a drive from a running test ZFS pool.

Perfect thanks for this information, also great job with the container!


Sent from my iPhone using Tapatalk

Share this post


Link to post
14 hours ago, ich777 said:

Hey @steini84,

I ran into a similar problem

This is the default behaviour of ZFS since it goes to a 'wait' state read this: Click

Also this thread from L1 should make it a little bit clearer: Click (a little old but still up-to-date)

Hey, thanks for the report.  ZFS is very cool, but @steini84maybe we should update post1 with "Where not to use ZFS"

 

It would appear that using ZFS to mount USB devices is not a good use case, (or should only be done in cases where  you are aware that ZFS is not plug and play with USB).  For normal disk maintenance, unRaid leads us to believe that we can safely do this with the array stopped.  ZFS is a totally different animal. 

 

Would best practice be to put a  "zpool export" command into Squid's awesome user scripts plugin and set it to happen  on array stop, and "zpool import" on array start?  On first startup of array, you should not need this as ZFS will automatically do the zpool import.  It would seem that user scripts supports all this and could make ZFS behave like the unRaid array.  Would this make sense?

 

Share this post


Link to post
4 hours ago, tr0910 said:

Hey, thanks for the report.  ZFS is very cool, but @steini84maybe we should update post1 with "Where not to use ZFS"

 

It would appear that using ZFS to mount USB devices is not a good use case, (or should only be done in cases where  you are aware that ZFS is not plug and play with USB).  For normal disk maintenance, unRaid leads us to believe that we can safely do this with the array stopped.  ZFS is a totally different animal. 

 

Would best practice be to put a  "zpool export" command into Squid's awesome user scripts plugin and set it to happen  on array stop, and "zpool import" on array start?  On first startup of array, you should not need this as ZFS will automatically do the zpool import.  It would seem that user scripts supports all this and could make ZFS behave like the unRaid array.  Would this make sense?

 

That's actually a really good idea and I doesn't thought of that, will implement that in my Container to avaid future problems on the next update.

Am I right that 'zpool export -a' is the right command to unmount all ZFS drives?

 

EDIT: @tr0910 If you are building a new Kernel/Images with my Container with ZFS drivers the import/export is now linked to the Array start/stop (also the prebuilt images in my thread are updated).

Share this post


Link to post

I have been testing on 6.9.0 b22 and have a ZFS mirror created on 2 spinners.  On this mirror, I have a Win10 VM created, and now twice unRaid has refused to start the VM after a VM shutdown.  The VM start command in unRaid just sits with the VM stopped and the red spinning arrows going in a circle attempting to restart the VM.  UnRaid is still reponsive as you can create another tab, and work with unRaid normally.

 

The VM had been running successfully after a fresh unRaid boot, but shutting down a Win10 VM and restarting the VM causes this issue.

Share this post


Link to post
I have been testing on 6.9.0 b22 and have a ZFS mirror created on 2 spinners.  On this mirror, I have a Win10 VM created, and now twice unRaid has refused to start the VM after a VM shutdown.  The VM start command in unRaid just sits with the VM stopped and the red spinning arrows going in a circle attempting to restart the VM.  UnRaid is still reponsive as you can create another tab, and work with unRaid normally.
 
The VM had been running successfully after a fresh unRaid boot, but shutting down a Win10 VM and restarting the VM causes this issue.

Anything in the syslog?


Sent from my iPhone using Tapatalk

Share this post


Link to post
Posted (edited)

At 10:26 there are a few zfs entries, but it is now 13:30 and the start VM icon is still spinning away.  About 12:00 would be the time that the VM was shut down.  Other than for this, the server continues to work normally.  ZFS list commands return expected results nothing seems broken, except the VM doesn't start.

 

On shutting down the server, unRaid claims the server is down, but a putty window shows the server is still live, and will respond to ZFS list etc.  What it will not do is ZPOOL export -a  This comes back with. 

cannot unmount '/mnt/MFS/VM/Win10ZFS': umount failed

 

cara-diagnostics-20200701-1327.zip

Edited by tr0910

Share this post


Link to post
Posted (edited)

Followup on the 6.9.0 b22 with Win10 ZFS install problems.  When the VM is shut down, and then attempt is made to restart it, the VM tab on the server will lock up.  Other than this, the server will continue to operate normally.  When the server is shut down from the GUI, to attempt to bring the VM back online, unRaid reports that shutdown is successful with system powered off.  A putty telnet window open gets the expected "The system is going down for system halt NOW!" message, but after unRaid reports the system is powered off, a press of the enter key shows putty still connected with a telnet session active.  Even the unRaid array remains mounted.  I don't know if this is unRaid's fault or ZFS?  I look forward to somebody else attempting to replicate this issue with 6.9.0 b22 and a Win10 vm.

Syslog entries after initating shutdown from GUI - It should be off, but its still live.

Jul  2 09:33:08 Cara shutdown[140908]: shutting down for system halt
Jul  2 09:33:08 Cara init: Switching to runlevel: 0
Jul  2 09:33:08 Cara init: Trying to re-exec init
Jul  2 09:33:12 Cara kernel: mdcmd (44): nocheck cancel
Jul  2 09:33:13 Cara emhttpd: Spinning up all drives...
Jul  2 09:33:13 Cara kernel: mdcmd (45): spinup 1
Jul  2 09:33:14 Cara emhttpd: Stopping services...
Jul  2 09:33:14 Cara emhttpd: shcmd (791): /etc/rc.d/rc.libvirt stop
Jul  2 09:34:42 Cara root: Status of all loop devices
Jul  2 09:34:42 Cara root: /dev/loop1: [2049]:4 (/boot/bzfirmware)
Jul  2 09:34:42 Cara root: /dev/loop2: [0044]:264 (/mnt/disk1/system/libvirt/libvirt.img)
Jul  2 09:34:42 Cara root: /dev/loop0: [0044]:260 (/mnt/disk1/system/docker/docker.img)
Jul  2 09:34:42 Cara root: Active pids left on /mnt/*
Jul  2 09:34:42 Cara root:                      USER        PID ACCESS COMMAND
Jul  2 09:34:42 Cara root: /mnt/MFS:            root     kernel mount /mnt/MFS
Jul  2 09:34:42 Cara root: /mnt/disk1:          root     kernel mount /mnt/disk1
Jul  2 09:34:42 Cara root: /mnt/disks:          root     kernel mount /mnt/disks
Jul  2 09:34:42 Cara root: /mnt/user:           root     kernel mount /mnt/user
Jul  2 09:34:42 Cara root:                      root      54064 f.c.. smbd
Jul  2 09:34:42 Cara root: /mnt/user0:          root     kernel mount /mnt/user0
Jul  2 09:34:42 Cara root: Active pids left on /dev/md*
Jul  2 09:34:42 Cara root: Generating diagnostics...
Jul  2 09:34:47 Cara avahi-daemon[4244]: Interface docker0.IPv4 no longer relevant for mDNS.
Jul  2 09:34:47 Cara avahi-daemon[4244]: Leaving mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1.
Jul  2 09:34:47 Cara avahi-daemon[4244]: Withdrawing address record for 172.17.0.1 on docker0.

 

Edited by tr0910

Share this post


Link to post

Followup on the 6.9.0 b22 with Win10 ZFS problems. If a Win10 VM is started and stopped, the server will shutdown successfully.  The attempt to restart a shutdown Win10 VM will fail and this is what is bringing ZFS and unRaid to a bad place where unRaid will not shutdown.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.