From what I've read, this xattr=sa setting, extensively reduces the array from doing writes to files on disk, instead writing to inodes.  If this is always the way it has worked (and I believe it is), changing this is only like to be a workaround of some other thing, impacted by lots of writes such as when creating a new VM image.  I had thought this was probably kernel related, but it doesn't happen on 6.8.3 and 6.9 if I recall.  Either way, I'm not convinced this is the end of it.

Upgrade to beta 4 has completely borked my system.  That's my current technical assessment anyway.  It took me ages to figure that out too.

 

All sorts of errors about missing shared libraries, and also doing odd things like removing the docker and virtual machine menus from the Unraid GUI, also noticed it's telling me docker can't start because VT-d is missing - well it's a dual Xeon I don't think so.

 

So far I don't recommend any of these betas and regret using them at all.

 

I've tried to downgrade to beta 3 using the above method which is saying ZFS modules are not loaded so something is not right.

 

This is all happening at a time when I have no time.  But I need it to run.  If I had the time I would reformat and downgrade to stable ZFS.

 

ZFS beta HIGHLY NOT RECOMMENDED on UNRAID is my current opinion.

 

The funny thing is that ZFS itself is running in the background - even when the menus are missing docker and vm's are running too (or were at least).  It's also causing other issues like randomly not being able to boot from USB stick.

 

My guess is that like the other assumption it's more of an issue to do with the implementation of ZFS as per how it's attached to unraid.  But I don't know.

 

Even the nerd pack is throwing up errors with this installed.  It's possible it's something else I haven't thought of, but when I disable ZFS everything else starts to run normally again.

 

Painful!

@steini84 Are you sure something in the new build of ZFS beta4 isn't overwriting parts of the system it shouldn't?  I've even got it complaining I've got less that 2G of memory in the GUI, yet in the console it's saying I have 96.  Check out my main screen. I've done a memory test too just in case.  Check out how docker and VM tabs are missing, but actually they're running.  Weird scripting / coding language showing up in the GUI.

I managed to downgrade to beta 3 on 6.8.3.  Same things happening so I guess it's co-incidence.  Not really what I want to hear, but sadly co-incidence must mean I have bigger issues.  No idea what though at this point.

Which mountpoints do you have/use for your zfs pools?

I use /mnt/x y z

 

I've just reformatted the key and this time not copied the config.  It was all working fine until I added ZFS.  I note that only the beta of ZFS is now being offered not stable, which I think is a huge mistake given it's the stable 6.8.3 I have installed.  @steini84 perhaps you could implement something so that stable / next can be chosen for ZFS.

 

Anyway, I'm back to believing it's a ZFS issue.  I don't know why it is.  I'm going to try the community kernel and see if that sheds any light.

 

Community Kernel (Unraid Kernel Helper) same result (using my normal config though, but I did prove it did this with none of my config also). This is a nightmare.  The only thing I can say for sure is it only appears when ZFS is installed.  But earlier known good versions of ZFS also exhibit the behaviour.

Edited October 23, 20205 yr by Marshalleq

4 hours ago, Marshalleq said:

I use /mnt/x y z

 

I've just reformatted the key and this time not copied the config.  It was all working fine until I added ZFS.  I note that only the beta of ZFS is now being offered not stable, which I think is a huge mistake given it's the stable 6.8.3 I have installed.  @steini84 perhaps you could implement something so that stable / next can be chosen for ZFS.

 

Anyway, I'm back to believing it's a ZFS issue.  I don't know why it is.  I'm going to try the community kernel and see if that sheds any light.

 

Community Kernel (Unraid Kernel Helper) same result (using my normal config though, but I did prove it did this with none of my config also). This is a nightmare.  The only thing I can say for sure is it only appears when ZFS is installed.  But earlier known good versions of ZFS also exhibit the behaviour.

Good idea, I completely forgot to separate the stable builds from the RC version. I upgraded the plugin so that by default you install the cached version if available. If not the plugin check for a stable version by default.

However you can touch a file on disk and enable unstable builds (like the 2.0.0 RC series) 

 

#Enable unstable builds
touch /boot/config/plugins/unRAID6-ZFS/USE_UNSTABLE_BUILDS
rm /boot/config/plugins/unRAID6-ZFS/packages/*
#Then reboot

#Disable unstable builds
rm /boot/config/plugins/unRAID6-ZFS/USE_UNSTABLE_BUILDS
rm /boot/config/plugins/unRAID6-ZFS/packages/*
#Then reboot

The builds are pretty much vanilla ZFS and you can check the build scripts on Github:

https://github.com/Steini1984/unRAID6-ZFS/blob/master/build.sh (to build latest ZFS)

https://github.com/Steini1984/unRAID6-ZFS/blob/master/build_github.sh (to build custom versions like RC)

 

I feel your pain and it´s incredibly frustrating when the server has issues! I remember having incredible problems with my server a few years back. Random disks failing, processes crashing and all around pain. Took me probably a week of debugging and the solution was a new PSU. Hope you can fix your problems and I really hope that at least ZFS in not the culprit

Thanks - I actually installed a new version of 6.8.3 on my USB using the Mac install tool, which in the process wipes the usb stick.  I copied over no config and installed ZFS from the App Store and somehow that ended up with the beta4 version of zfs.  So I don't know.  Anyway a bit of a learning for me!  I have to wonder what is on that disk that's so problematic - I only created it recently.  I have a suspicion it may have a duplicate mount point or something.

Using zfs-2.0.0-rc4-unRAID-6.9.0-beta30.x86_64.tgz with unRaid 6.9-beta30, the /dev/zvol/<zvols-by-name> are missing (even the /dev/zvol/ folder is missing) while they appear as /dev/zd* and are fully usable, but it is exhausting to check all the /dev/zd* with fdisk to find out wich one belongs to the corresponding zvol (and for volumes with same size and disk format this problem is only solvable by try and error).

 

Is this only a problem for me or a problem with the package?

 

Thanks and regards.

 

I have written a lil bash script that creats the links, you can use/modify it if you need this too:

#!/bin/bash
mkdir /dev/zvol/
for i in $(ls /dev/zd*)
do
        TMP=$(/lib/udev/zvol_id $i)
        TMP0=$(cut -d '/' -f1 <<<$TMP)
        TMP1=$(cut -d '/' -f2 <<<$TMP)
        mkdir /dev/zvol/$TMP0
        ln -s $i /dev/zvol/$TMP0/$TMP1
done

 

Edited November 2, 20205 yr by efschu

Built zfs-2.0.0-rc5 for 6.8.3 & 6.9.0-beta30

*see the first post to see how to enable unstable builds

Hey @steini84

Don't know if you've already seen it but I've opened a PullRequest on your ZFS plugin to import/export the ZFS pools on start/stop of the array.

 

I only thought this would be a nice feature.

If not simply delete it.

question, how do you guys monitor the disk smart when using ZFS?
unassgined devices plugin is not monitoring disks that are not mounted right?

 

27 minutes ago, segator said:

question, how do you guys monitor the disk smart when using ZFS?
unassgined devices plugin is not monitoring disks that are not mounted right?

 

Not using ZFS, but this is a nice solution as it collects smart data so you can get historic data.

Works for all drive devices, not just array/cache

 

question, how do you guys monitor the disk smart when using ZFS?

unassgined devices plugin is not monitoring disks that are not mounted right?

 

I use check_mk for overall monitoring of my servers and everything that has a network. It seems that the unassigned drives are being monitored by unraid :

 

 

Sent from my iPhone using Tapatalk

interesting not on mine   (those disks are a zfs pool)

shall I need to do something else?

interesting not on mine [emoji37]  (those disks are a zfs pool)
shall I need to do something else?

Check the Dashboard tab, you are in Main


Sent from my iPhone using Tapatalk

oh.. sorry what a newbie I am...
yes says smart ok, so I supose if in the future some issues are detected I will be notified as any disk in the array, right¿?

BTW someone tried ZFS over USB disks? is that a crazy idea? i am out of space on my rack server  and i'm considering buying a 4slots usb enclosure. any other idea if not?

I would not use USB for main storage, it's slow and can cause a mess if you disconnect the usb cable without exporting the pool.

 

I have used a buffalo diskstation over usb for zfs replication for years and that works perfectly fine after I put it on a UPS

Well the idea is to have it there 24/7 running to expand my array, some will be disks on unraid array (so i supose no problem as disks works independently, but i also plan to create another ZFS raid there(3disks).

 

 

I'm not trying to hush you, but any idea when version 6.9.0-beta35 will be supported? I had to rollback just now.

Building right now

 

... Running into some build issues. Hopefully I'll have time to figure it out over the weekend.

 

Sent from my iPhone using Tapatalk

On 11/8/2020 at 5:24 AM, segator said:

Well the idea is to have it there 24/7 running to expand my array, some will be disks on unraid array (so i supose no problem as disks works independently, but i also plan to create another ZFS raid there(3disks).

 

 

I've recently been wondering about the speed of rebuild's / checks on USB.  I have a 5 bay USB 3 box that came from a QNAP.  It's seen in unraid.  But I think it will be too slow.

8 hours ago, steini84 said:

Building right now

 

... Running into some build issues. Hopefully I'll have time to figure it out over the weekend.

 

Sent from my iPhone using Tapatalk

Did you read the beta 35 notes?  It seems that there's some new methods for deploying drivers and modules.  They actually include in tree AMD GPU too.  At first I was thinking this was going to save the day - then I realised we still have to compile it against the new kernel right?  I've found these particular new Unraid beta's to be the worst betas I've ever used for some reason, so I think I'll stay away until they at least put the new linux kernel in.

Did you read the beta 35 notes?  It seems that there's some new methods for deploying drivers and modules.  They actually include in tree AMD GPU too.  At first I was thinking this was going to save the day - then I realised we still have to compile it against the new kernel right?  I've found these particular new Unraid beta's to be the worst betas I've ever used for some reason, so I think I'll stay away until they at least put the new linux kernel in.

Yeah I just put it all together and pushing it out now


Sent from my iPhone using Tapatalk

1 minute ago, Dtrain said:

Beta 35

ZFS Plug could be installed but modules are not loaded   

I had to update the plugin to adjust to the new modules system and make it backwards compatable so try updating the plugin