Marshalleq

Already did that some time back. I don't expect it to come anytime soon given their stance on ZFS.

Yeah, shouldn't. Nevertheless, I do recall in the logs from joly0 that it was the loopback process that was stuck on 100% cpu while this issue was happening. Not that I investigated it deeply, that's just what I remember.

Also, I think it would be a good idea to keep zfs > 2.0.1 out of main until this is resolved?

For clarity, I do believe someone posted previously that 2.0.3 still exhibits this issue and the workaround is to ensure your docker.img file is not placed on a ZFS partition. I'm stuck on 2.0.1 because I no longer have a filesystem other than ZFS (and don't want one). We seem to be at a deadlock where we require some help from someone knowledgeable about the inner workings of docker on unraid. @ich777 Perhaps with your kernel helper there would be a way to build ZFS support into docker as a workaround as per Squid's comment below? Potentially we could then get rid of the docker.img entirely, which seems to be some issue with running loop device on ZFS > 2.0.1. @joly0 Just thinking about your report that the problem process was loopback - I wonder if we could test a number of different loopback devices to see if there's anything in that.

@Squid Is there a particular developer you could recommend that knows the ins and outs of docker that might be able to steer us on this issue? It would be great to have it sorted out so that @limetech doesn't have this as a blocker for any future zfs implementation. We are all looking forward to that and it would be disappointing to have another reason to delay our chances. 😡

@squid For those whom want the quick summary, this thread ends with, "There's different drivers for docker to be on different filesystems. AFAIK, there's only the 2 included in the OS -> btrfs and xfs. You could probably do a feature req to have the driver for zfs included for docker"

Good question - I should be more specific. I just found that there were files that I 'assume' are core to docker functioning on the boot drive. My bad for assuming, but I deleted them because I assumed they should not be there given I point docker at a different drive. That deletion removed my docker config and it was recreated, which I 'assume again sorry' is back in the boot drive. I should check that. So unless it's obvious to you, I assume docker requires some files in /boot somewhere additional to those specified in the docker config GUI page which I did double check was correctly configured to NOT point at the boot drive. It seemed logical that since my configuration was pointed elsewhere and deleting these files broke docker config, that these were core to docker (on unraid) functioning.

I've tried that. It doesn't work. Apparently there's something in Unraid that requires ZFS compiled in for it to work. Sounded strange but the explanation looked solid. That response is written either in here or in the general forum, probably the forum. Logged under me.

@Joly0 To do what I did for the GitHub ticket, I set up another pool on a spare NVME drive (figuring a pool would be more unraid workable than an unassigned disk). And noting it's a single NVME drive with no other drives in the pool, not even a parity. I pointed docker directly at that new pool - but for some reason I today find out that it put a lot of core docker files on my USB boot drive. I even double checked and it was definately still configured to be pointed at the NVME drive. I found this out because I thought it was a hangover from something else and deleted the usb drive files (which were just in an app data folder). I honestly don't know what is wrong with unraid here. Perhaps it's always had these files on the usb, but I've never noticed them before. I'll recreate on an unassigned disk and try again. After a day or so though, otherwise docker is working if the docker image is not placed on a ZFS drive as per your experiment.

This shouldn't really need to be 'handled' at all. Unraid forces the use of nobody.users and I can see why TBH. But if they're going to do that, they should really default all the built in writing to their chosen permissions format. File sharing has never been the strong suit of unraid unfortunately, but with a bit of tweaking it can work OK.

I actually forgot all about this - have been away for work. My last post when it was working was January 7. I see it's been working well until January 18, upon which it stopped again (that's 7 days ago). @steini84 Is there some way to turn on extra logging to a file? Thanks.

@Joly0 you don't happen to be running xattr=sa do you? Or any other changes from the norm? Was reading the issues list and there are a few BSD related things. Just trying to think of any similarities. @steini84 on the thread above, they're suggested we do a staggered approach of all the commits to see where in the code history this happened between 2.0.0 and 2.0.1. I might just give that a go - might need to just confirm some things with you first.

This may be our issue: https://github.com/openzfs/zfs/issues/11476

Yeah the zfs array unmounts separately. I guess the same problem is causing the unraid array to not unmount properly, which in turn is meaning things are still active running on the ZFS array. Only other thing I could think of would be to reboot in maintenance mode, or to set the unraid array not to start at boot. But I'm not sure if that will give @steini84 the information he wants. I am definitely not running that particular docker BTW. It could be an 'other' common docker component I suppose. At least now we have two of us with the issue - maybe someone else will get it too and we can start to narrow it down. I might just have time to try run those commands tonight if I'm lucky.

Yes if you unmount the unraid array, then it shouldn't give you that error. The unmounting of the unraid array would stop all the things that are preventing the zfs array from running that command. Out of interest, which docker? Perhaps I'm running the same one and it's related to that. Thanks.

Oh right. I think first you should stop the unraid array (go to the main screen of the unraid going, scroll down and stop it there). This will stop docker, vm's shares and such like so that nothing is accessing ZFS - then that should allow the zpool command to run. Then run the command in console - zfs import -a which I think will try to auto unmount the ZFS parts of the ZFS array as part of the command. I've never run this command before, but in case you can't get your array back, the command would be the opposite e.g. zpool import -a Yeah - docker seems more impacted. There were other screens that were an issue too. Docker was running in the background if I recall - but I do have 128GB RAM in my system which might have been hiding that particular issue.

@Joly0 can you run the commands requested by @steini84 above? I'm not going to be able to get to it for around a week. e.g. Stop the array Open up a terminal and type in: 'zpool export -a' and wait for it to finish Then click reboot And report back. Probably good to run and post a diagnostics too. Thanks.

I've just built 6.9 RC2 with zfs support. However I don't seem to be able to do GPU passthrough with my NVIDIA 1070TI (EVGA). It seems to either complain with the error you get if you have an AMD card, or it boots to a black screen. This was working in 6.9RC2 with the zfs plugin, but not now with this compiled version. I'm not 100% it's the compiled version either. Every now and then I get issues, which I'm suspicious are resolved by moving the card out of slot 1. Going to try that now anyway. EDIT: Moved my card out of the primary slot and it seems to be working now. What isn't working on ZFS 2.0.1 is rebooting. I'll see how it goes, but suspect I'll roll back to 2.0.0.

@steini84 Just an update that today I've compiled zfs version 2.0.1 on 6.9 RC2 using the unraid kernel helper. That seems to be working perfectly so far. So this is good as it points to some other issue, that should be fixable. I'd be interested to know if anyone else has any issue with the 2.0.1 plugin. EDIT: It actually looks like there is at least one thing still happening - and that is that I can't actually reboot without hard rebooting by holding the power button down. So that to me confirms there is some issue with 2.0.1 or some feature it's introduced that's incompatible with something in unraid. I'll keep testing. EDIT2: Nope my docker tab now is also not loading. So right now I definitely don't recommend 2.0.1 given this happens on two machines and two separate builds of ZFS.

So for my main box I got around it by using the community kernel direct download here to get it going, in case anyone else gets stuck. Until this plugin is fixed or whatever is happening. Note the direct download is 6.8.3 though, with ZFS 2.0.0.

Another curiosity, along with the zfs-2.0.0-unRAID-6.9.0-rc2.x86_64.tgz going missing, the zfs plugin get's uninstalled, both at reboot.

@steini84 There's something wrong with 2.0.1. I removed unassigned devices plugin altogether, hard rebooted (because I have no choice - saved by CoW). Still same issue, so then did: cd /boot/config/plugins/unRAID6-ZFS/packages/ rm * wget https://github.com/Steini1984/unRAID6-ZFS/blob/master/packages/zfs-2.0.0-unRAID-6.9.0-rc2.x86_64.tgz.md5 wget https://github.com/Steini1984/unRAID6-ZFS/blob/master/packages/zfs-2.0.0-unRAID-6.9.0-rc2.x86_64.tgz Hard rebooted again because no choice and the server now reboots correctly. However I must have done something wrong as ZFS is not working. Looking in the packages directory only the md5 file remains and zfs file is missing. I tried the whole procedure again and the file is missing again. Zfs command doesn't work. It's like the tgz file is just being deleted.

There were two plugin updates at the same time, unassigned devices and nerd pack. I assume it'll be some issue with unassigned devices rather than ZFS but I am still investigating. After upgrade the docker tab loads indefinitely, the main tab is either slow or only partially loads. It also won't reboot via GUI or command line reboot requests, which is the most concerning thing. This happening on both systems I just updated with ZFS and the two plugins.

I got zfs-2.0.1-1 zfs-kmod-2.0.1-1 However, for reasons unknown, on both my machines the GUI is now mostly unresponsive. Anyone installed it and got it working?

@Joly0 Can you point at what the specific fix is from here? I run lancache too, but I haven't noticed (nor tested) for any issues such as you describe because I don't understand how there could create an issue with a single docker container - it sounds like you're saying ZFS / RC2 prevented writes - but we'd be seeing that across the board. Either way, there are some good performance fixes in ZFS 2.01. Regarding updates, I'm not sure if it needs to be manually compiled first by @steini84, but either way it's supposed to apply when you reboot.