ZFS plugin for unRAID


steini84

Recommended Posts

9 minutes ago, BVD said:

Negative - the plugin's current design checks on reboot so he doesn't have to generate a new plugin each time a new ZFS release comes out.

This is also not possible on other systems.

(Maybe it is possible but if you restart after building the new Kernel module you are way quicker than issuing all commands and unloading all your ZFS pools and so on... I think you get what I mean)

  • Like 1
Link to comment
19 minutes ago, BasWeg said:

Ah ok. It was possible in the past, but since zfs is a kernel Module it is not possible anymore. 

As said one comment above it is still possible but you are way quicker rebooting and it would basically be the same as if you reboot.

 

I think you get the point.

AFAIK you have to reboot after updating ZFS on other systems too and this is also the recommended and from my oppinion the right way to do it.

  • Thanks 1
Link to comment
11 hours ago, ich777 said:

Just to let you guys know, a update for ZFS is available ZFS v2.1.4 (Changelog - not available yet but should be tomorrow)

 

The new ZFS packages are currently building for Unraid versions:

  • 6.9.2
  • 6.10.0-rc4

 

Should be done in about 20 minutes.

 

To pull the update please restart your server or upgrade to Unraid 6.10.0-RC4 if you want to try the Unraid RC series (if you restart your server make sure that your Server has a active internet connection on boot and is actually able to connect to the internet).

 

@steini84

 

Man, I should have waited for this before moving to RC4, which caused me some grief. Back on RC3, will move once Extended Self Test is complete for all HDDs.

Link to comment
18 hours ago, BVD said:

After looking at @dmacias's repo, seems like it'd be easy enough to build. Guess I know what I'll be doing this weekend lol

You can get away with a oneliner in the go file / User Scripts: 

wget -P /usr/local/sbin/ "https://raw.githubusercontent.com/jimsalterjrs/ioztat/main/ioztat" && chmod +x /usr/local/sbin/ioztat

 

But packing this up as a plugin should be a fun project

Link to comment
20 minutes ago, steini84 said:

But packing this up as a plugin should be a fun project

Doesn't even saw that it's written in Pyhton... :D

 

Really don't know if you really need a package if it's that simple...

What's also possible @BVD is that you download and put it on the boot device in a dedicated folder (maybe "scripts") and copy it over to /usr/bin and make it executable through the go file like @steini84 mentioned.

  • Like 1
Link to comment
20 minutes ago, sabertooth said:

How do we debug this?

I think first of all your Diagnostics would be great.

 

20 minutes ago, sabertooth said:

Within half an hour or so the pool was again in SUSPENDED state.

Can only speak for myself but RC4 and ZFS 2.1.4 working great, but I'm only not the hardcore ZFS user (only having one mirrored Pool).

Link to comment
1 hour ago, ich777 said:

I think first of all your Diagnostics would be great.

 

Can only speak for myself but RC4 and ZFS 2.1.4 working great, but I'm only not the hardcore ZFS user (only having one mirrored Pool).

 

Finally, was able to replicate the issue with RC3 as well.

Please find the diagnostics as attached.

 

unraidzfs-diagnostics-20220324-1932-zfs-suspended.zip

 

Edited by sabertooth
Adding diagnostics.
Link to comment
2 hours ago, ich777 said:

Doesn't even saw that it's written in Pyhton... :D

 

Really don't know if you really need a package if it's that simple...

What's also possible @BVD is that you download and put it on the boot device in a dedicated folder (maybe "scripts") and copy it over to /usr/bin and make it executable through the go file like @steini84 mentioned.

 

After finding that my go file was getting out of hand, I ended up setting up a script that kicks off at first array start instead which walks through symlinking all the various stuff I had in there instead, just so I could keep the go file limited to things that actually have to be ran at init (or at least 'things that make the most sense at init' lol).

At this point, it links my bash and tmux profiles, iterm shell integration, some cron, several other things - this way at least the UI is relatively tidy lol. I'd found some areas where just having everything on the NVME zpool and linking it afterwards just seemed to help with initial startup times and the like.

 

Maybe if user scripts ever gets the ability to pul things in folders I'll change it up, but I spend most of my time in the cli anyway, so I guess we'll just see what happens 🤷‍♂️

  • Like 1
Link to comment

Hi everyone,

 

It's been a while I noticed it, but some of the dockers datasets display a yellow icon, even the snapshot as we can see in the screenshot. Those managed by docker itself, not mines.

 

image.thumb.png.b058a928e824e4992712a59a73859de6.png

 

I imagine that it means that the dataset or snapshot is damaged if the colors rules are the same as for the pools.

 

Everything is working whatever. I'd simply ask to you how you manage this on your own servers or if you let it sink?

Edited by gyto6
Link to comment
2 hours ago, gyto6 said:

Hi everyone,

 

It's been a while I noticed it, but some of the dockers datasets display a yellow icon, even the snapshot as we can see in the screenshot. Those managed by docker itself, not mines.

 

image.thumb.png.b058a928e824e4992712a59a73859de6.png

 

I imagine that it means that the dataset or snapshot is damaged if the colors rules are the same as for the pools.

 

Everything is working whatever. I'd simply ask to you how you manage this on your own servers or if you let it sink?

 

I think you probably meant to post that for the ZFS Master plugin? This one just adds the ZFS driver. Maybe there's a hover over help menu or something? Not sure.

To the question at hand though, it looks like you're using the docker ZFS driver (using folders instead of .img) - personally, I'd recommend against that. The data within these directories are just the docker layers that get rebuilt each time you update a container. Doing it this way just makes managing ZFS a mess, as you end up with all sorts of unnecessary junk volumes and/or snapshots listed out every time you do a zfs list / zfs get / etc. Plus, it creates so danged many volumes that, once you get to a decent number of containers (or if you've automated snapshots created for them), filesystem latencies can get pretty stupid.

Link to comment
2 hours ago, BVD said:

 

I think you probably meant to post that for the ZFS Master plugin? This one just adds the ZFS driver. Maybe there's a hover over help menu or something? Not sure.

To the question at hand though, it looks like you're using the docker ZFS driver (using folders instead of .img) - personally, I'd recommend against that. The data within these directories are just the docker layers that get rebuilt each time you update a container. Doing it this way just makes managing ZFS a mess, as you end up with all sorts of unnecessary junk volumes and/or snapshots listed out every time you do a zfs list / zfs get / etc. Plus, it creates so danged many volumes that, once you get to a decent number of containers (or if you've automated snapshots created for them), filesystem latencies can get pretty stupid.

Indeed, I posted in the wrong topic once again... My mistake.

 

Indeed, using the directory for docker is actually a mess due to all theses datasets and snapshots as you described very well.
 

I'll manage to migrate to the .img docker. I did a screenshot from every containers configuration as I won't get them back from Previous App and put the .img file in a 1M recordsize dataset. I don't thing that there's too much configuration for a dataset due to the low use of the docker files itself and the NVMe drives do most of the performances.

 

Thanks again! @BVD

Edited by gyto6
Link to comment

You've got a couple options - 

1. Create a zvol instead, format it, and keep docker writing there (which is now a block storage devices)

2. Your plan, creating a fileset and using a raw disk image

 

In either case, couple things you can do - 

* redundantmetadata=most - containers are throwaway, no real reason to have doubly redundant metadata when you can just pull the container back down anyway; wasted resources

* primarycache=none (or metadata at most) - containers might be (probably imperceptably given you're on NVME and not SATA) slower to initially start, but once they do, the OS is already handling memory caching anyway, leaving ZFS often duplicating efforts (and using memory by doing so)

* sync=standard - again, containers, who cares if it needs to be re-pulled

* atime=off - (I do this for everything)

I've got a whole whack of performance tuning notes lying around for one thing or another - if the forum ever gets markdown support, I'll post em up, maybe then can be helpful to someone else lol

Edited by BVD
edited after checking my own settings to add a couple more
Link to comment
32 minutes ago, BVD said:

You've got a couple options - 

1. Create a zvol instead, format it, and keep docker writing there (which is now a block storage devices)

2. Your plan, creating a fileset and using a raw disk image

 

In either case, couple things you can do - 

* redundantmetadata=most - containers are throwaway, no real reason to have doubly redundant metadata when you can just pull the container back down anyway; wasted resources

* primarycache=none (or metadata at most) - containers might be (probably imperceptably given you're on NVME and not SATA) slower to initially start, but once they do, the OS is already handling memory caching anyway, leaving ZFS often duplicating efforts (and using memory by doing so)

I've got a whole whack of performance tuning notes lying around for one thing or another - if the forum ever gets markdown support, I'll post em up, maybe then can be helpful to someone else lol

Well, thank you for all these precious advice!

 

The problem to me by now is that I have to mount the ZVOL to Unraid. I'm figuring out how to do this on Unraid without ISCSI. The docker .img is indeed not functional (tons of bugs) and trying this on an USB key is not a good idea at all.

 

Thanks again. ;)

Link to comment

Thanks again @BVD ! I'm finally able to run the docker.img inside of a ZVOL.

 

For those interested:

 

zfs create -V 20G pool/docker # -V refers to create a ZVOL
cfdisk /dev/pool/docker # To create easily a partition
mkfs.btrfs -q /dev/pool/docker-part1 # Simple to format in the desired sgb
mount /dev/pool/docker-part1 /mnt/pool/docker # The expected mount point

 

 

Link to comment
12 hours ago, gyto6 said:

Hi everyone,

 

It's been a while I noticed it, but some of the dockers datasets display a yellow icon, even the snapshot as we can see in the screenshot. Those managed by docker itself, not mines.

 

image.thumb.png.b058a928e824e4992712a59a73859de6.png

 

I imagine that it means that the dataset or snapshot is damaged if the colors rules are the same as for the pools.

 

Everything is working whatever. I'd simply ask to you how you manage this on your own servers or if you let it sink?

 

Well, there is no ZFS Master support topic yet... so I answer that question right here :P; the color of the camera icon is about the last snapshot date; blue means everything is fine, grey means that there is no snapshots for the dataset; and yellow/orange means that the last snapshot is older than the amount of days that you configured in the settings. BTW, ZVols are not supported yet; I hope to get some time off in the coming months, so I can get hands on all the functionality in the plugin.

 

 

image.png

Edited by Iker
Link to comment
7 hours ago, BVD said:

Create a zvol instead, format it, and keep docker writing there (which is now a block storage devices)

I run everything (Docker, libvirt, system) on my ZFS Mirror without any formatting.

 

The only difference is that I first created a partition and them created a mirror with the corresponding partitions like /dev/sdx1 /dev/sdy1

 

Is it possible that this is the reason why everything runs without a hitch on ZFS on my system?

Link to comment
6 hours ago, ich777 said:

I run everything (Docker, libvirt, system) on my ZFS Mirror without any formatting.

 

The only difference is that I first created a partition and them created a mirror with the corresponding partitions like /dev/sdx1 /dev/sdy1

 

Is it possible that this is the reason why everything runs without a hitch on ZFS on my system?

Do you means that with ZFS Master, you can't see tons of datasets with snapshot generated by Docker?

 

Referring to the @BVD interventions, I didn't suffer of any performance troubleshoot. The only matter was thatr all those dataset generated by docker was always flooding the GUI and any zfs list command.

 

Thanks for the precision @Iker, the ZVOL support could be interesting. Meanwhile, I simply mounted the ZVOL within a dataset in order to get it listed in the GUI.

Edited by gyto6
Link to comment
15 minutes ago, gyto6 said:

Do you means that with ZFS Master, you can't see tons of datasets with snapshot generated by Docker?

Sure do I have datasets created by Docker but this is also the case if you run it on a BTRFS volume, this is just normal and these are the actual Docker Images.

  • Upvote 1
Link to comment
On 3/19/2022 at 2:59 AM, ich777 said:

On what Unraid version are you?

Was a Parity check triggered too?

This will be only triggered if the shutdown was somehow unclean.

Please always attach your Diagnostics if you experience such issues (pull the Diagnostics of corse after it happened and don't reboot in between).

 

Do you had any SSH connections or some terminal windows with an active connection to Unraid open?

Please go forward and try to upgrade to the latest RC version and see if the same happens.

A scrub won't hurt your pool in any way.

RC2, the unraid parity was not triggered just the zfs. was a while ago so no data. will be upgrading to rc4 once things get caught up.

Link to comment
39 minutes ago, anylettuce said:

RC2, the unraid parity was not triggered just the zfs. was a while ago so no data. will be upgrading to rc4 once things get caught up.

This is really strange since the Parity check and ZFS Scrub are using the same way how it‘s triggered.

Link to comment

Hi everyone, I’m new to Unraid (moving from Synology) and I just finished the parity check on my 2x 18TB array with 512GB NVMe cache connected via 2.5Gb. Would it be a good idea for me to use this ZFS plugin since I’m starting clean? I read about needing to use a separate USB drive in XFS since Unraid doesn’t support this natively. I’m running the latest RC and keep reading about imminent native support, would there be an easy upgrade path if/when Unraid supports ZFS natively so I don’t have to wipe my drives?

Link to comment
On 3/25/2022 at 12:23 AM, ich777 said:

I run everything (Docker, libvirt, system) on my ZFS Mirror without any formatting.

 

The only difference is that I first created a partition and them created a mirror with the corresponding partitions like /dev/sdx1 /dev/sdy1

 

Is it possible that this is the reason why everything runs without a hitch on ZFS on my system?

 

Are you running 6.10? I've not tested it since 6.8.3, so perhaps there were updates to the docker zfs driver in between that've made a difference, I'm not sure. I'd say it's more likely though that you're just not 'pushing the limits', so any inefficiencies never really bubble to the surface.

I will say though, I likely would've been in a similar situation with btrfs. At the time, I was running some ~160 containers on one system (testing for some work stuff in addition to my home environment), and it got to the point where any time one of the containers was restarted, there were significant latency spikes, even though the hardware layer still had loads of overhead available. I tracked it as far as a long running TXG (but before attempting to commit the TXG) before I realized I not only didn't have time for this, but was way the hell out of my league lol. Something funky going on in spacemap, but I'd no idea what.

 

To me though, it makes some sense - why expend the resources necessary to create a snapshot of every single layer of every single container every single time for no benefit really given the containers are disposable (comparatively higher transactional expense to both CPU and filesystem), when you can instead just give it a single entrypoint and let the lower level IO scheduler handle it with an image or block device? 

 

Honestly for most folks it probably doesn't matter though - a single current gen NVME device can handle enough IO these days that even with any number of inefficiencies in configuration, they likely wouldn't notice. And if they do, it'll likely be some higher level code efficiency that crops up before the storage starts to give em any headaches lol.

"Back in my day, we had to pool 80 drives, manually align blocks to page sizes, and add a SLOG just to get enough IOPs that the DBA's didn't burn us at the stake! Don't get me wrong, they were still out for blood, and if you saw them at the coffee machine when you went for your refill, you still turned around and came back later... But at least they just wanted you harmed, and not dead!"

🤣

Edited by BVD
Link to comment
4 minutes ago, BVD said:

Are you running 6.10?

Yes.

 

4 minutes ago, BVD said:

so perhaps there were updates to the docker zfs driver in between that've made a difference, I'm not sure

Many people reporting issues with ZFS and Docker on 6.10.0

 

 

 

My main concern is why so many people having issues but I don't have... I run everything form ZFS (Docker, libvirt,...). Maybe it's because I've created a partition before or because I use the full path to the partition /dev/sdx1 who knows, would be really interesting, have to do some tests when I got more time on my test system.

 

I've seen also some Github Issues that Docker fails in different scenarios while having everything on ZFS.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.