2 minutes ago, ich777 said:

Because you are using /mnt/user/... please always use the real file path not /mnt/user/... when specifying a path other then /mnt/user/... in a plugin <- that's actually your issue.

 

You misunderstood. I never use the FUSE "/mnt/user" path on my systems for anything (absolutely every share on every server I'm running uses pool storage directly). I was only showing what it looked like in there after using only the pool. I edited the previous post as I isolated this as an Unraid bug.

 

Any folder created inside the root of the pool causes this issue.

 

 

Edited September 22, 20241 yr by Espressomatic

1 minute ago, Espressomatic said:

I edited the previous post as I isolated this as an Unraid bug.

Already saw that but I don't think that this is a bug or at least not easy to solve because Unraid has no idea where the "source" share comes from and simply has no idea what's it's main storage is so it defaults to Array, which in your case is wrong.

I don't know how easy it'll be to fix, but the issue is caused by making a directory (on the ZFS pool), whereas using the Shares UI to create a new share creates a ZFS Dataset (which doesn't show the issue).

 

I'm going to try this on a server where I have some extra disks and where I can make an array device - interested to see what happens.

Edited September 22, 20241 yr by Espressomatic

2 minutes ago, Espressomatic said:

on the ZFS pool

I mean I know that this is also true if you do that on a BTRFS pool but this was never a bug to me because of the above explained reason (share was not created by Unraid and it default‘s to the Array).

 

3 minutes ago, Espressomatic said:

which doesn't show the issue

Because Unraid creates the config file for the share on it‘s own when you create a share through the GUI, which the LXC plugin can‘t do.

Or at least it never bothered me because it was clear to me that I have to manually set it‘s main storage pool in the share settings.

1 hour ago, ich777 said:

I hope you are using ZFS as the Storage Backing Driver in the LXC plugin settings and not Directory...

 

I am now.   I took the opportunity to create the lxc share via the Unraid dashboard on the pool so it's created as a dataset. Everything running well as expected.

 

 

3 hours ago, Espressomatic said:

I am now.   I took the opportunity to create the lxc share via the Unraid dashboard on the pool so it's created as a dataset. Everything running well as expected.

What? This is not what you should do, there is a simple setting for that in the plugin...

 

The plugin will create it's own dataset (alongside with the LXC directory to edit the actual files), please read the help text, I've even implemented a feature to Unraid where you can convert existing containers to it's specific Backing Storage Type.

No worries. The result is the same. ZFS is listed as the backer in the plugin and the container storage is a dataset. The container is working correctly, auto-start, stop, etc. The plugin is working correctly, including the ability to make snapshots.

 

Kudos on the extensive help. I wasn't aware it existed as I didn't even think to check - so many ppl skip that so my first instinct is to assume it doesn't exist.

Edited September 22, 20241 yr by Espressomatic

7 hours ago, Espressomatic said:

The result is the same.

Not quite...

If you are not using the LXC ZFS Backing Storage Driver then each container will be still be handled as if it was a directory and not use the ZFS Snapshot feature will be way quicker (seconds instead of minutes depending on the size of the container).

 

I would strongly recommend that you switch the Backing Storage to ZFS in your scenario because LXC then will use all the ZFS features natively which in it's current for it wont.

As said above, I also have implemented the script "lxc-dirtozfs" which you can call from the command line and in combination with the container name it will automatically convert the existing directory to native ZFS.

7 hours ago, ich777 said:

I would strongly recommend that you switch the Backing Storage to ZFS in your scenario because LXC then will use all the ZFS features natively which in it's current for it wont.

 

It is. The only difference in setup was that I created the dataset before selecting the option in lxc plugin settings.

 

 

Edited September 23, 20241 yr by Espressomatic

14 minutes ago, Espressomatic said:

It is. The only difference in setup was that I created the dataset before selecting the option in lxc plugin settings.

Can you please share your Diagnostics?

What does happen when you create a snapshot (in terms of the folder structure).

Diags attached below

 

 

trillian-diagnostics-20240923-1006.zip

Quote

It will create it's own dataset (nvme_pool/zfs_lxccontainers/...) on the ZFS pool where your Default LXC storage path is located

 

I played around with the convert and saw the above, which I'm not fond of. I was hoping for only one top-level dataset for LXC, and therefore only one entry in Unraid's Shares tab.

 

The LXC path created in the plugin settings is already a dataset. Does there need to be another dataset that links back to the first?

Edited September 23, 20241 yr by Espressomatic

2 hours ago, Espressomatic said:

It is.

The ultimate answer is that it is not because simply changing the backing storage type after you've already created containers won't change anything for existing containers and they actually won't use ZFS, they behave just like default containers without ZFS.

 

I just try to help and arguing that it is something it isn't is really frustrating for me...

 

You see, your containers uses the default directory entry (from what I see in your Diagnostics) :

lxc.rootfs.path = dir:/mnt/nvme_pool/lxc/NginxProxyManager/rootfs

where it should be something like:

 lxc.rootfs.path = zfs:nvme_pool/zfs_lxccontainers/NginxProyManager/NginxProxyManager

 

But since you've played around already with lxc-dirtozfs you see now a structure like that:

Where now LXC now will use only ZFS filesystem specific commands for creating a dataset, snapshots and so on...

 

I assume you already noticed that snapshots from container are now created immediately and don't take minutes...

 

40 minutes ago, Espressomatic said:

which I'm not fond of. I was hoping for only one top-level dataset for LXC, and therefore only one entry in Unraid's Shares tab.

That's not possible since you always have a dataset that is created by LXC which you need a mount point for (okay, you don't need it but if you don't have a mount point you wouldn't be able to copy files), thus two "shares", which the dataset is not really a share and you will actually not see anything in there.

 

However I could also implement that differently that you actually don't have a mount point for the dataset and that users have to manually mount the dataset to a directory, but that is not my way of doing things because that complicates things for users...

 

You see the directory /mnt/zfs/system/lxc/<CONTAINERNAME> is only the directory where the files from the dataset are mounted and therefore are visible.

 

40 minutes ago, Espressomatic said:

Does there need to be another dataset that links back to the first?

No, it doesn't need to be a dataset, these are just directories where the datasets are mounted.

ZFS should actually notice that when creating a snapshot that it won't include the files because another dataset (from LXC) is mounted in there.

 

Or is the only issue that you see another share in your Shares section within Unraid?

 

I hope that explains how everything works.

Not arguing, just trying to explain what's happening on my system.  Let me wipe everything clean so I can get some examples of what this looks like. There's been too many manual changes now so I want to make sure I can see and show what happens right from the start. Be back shortly.

 

I'll also explain how I anticipated this would work (as a former software designer and developer, how I believe it should work), if it's different.

Edited September 23, 20241 yr by Espressomatic

15 minutes ago, Espressomatic said:

Not arguing

This:

3 hours ago, Espressomatic said:

It is. The only difference in setup was that I created the dataset before selecting the option in lxc plugin settings.

 

 

15 minutes ago, Espressomatic said:

Let me wipe everything clean so I can get some examples of what this looks like. There's been too many manual changes now so I want to make sure I can see and show what happens right from the start. Be back shortly.

I know exactly what happens since I built the plugin or at least the GUI from the ground up and to summarize what you have to do or wh

 

1. Install the plugin
2. Go to the settings page enable LXC and set the path to somewhere on the ZFS Pool
   (At this point you can't change the setting for the storage backing type)
3. After you click apply you can now change the storage backint type to ZFS
   This will basically create another dataset called zfs_linuxcontainers on the same Pool that you've specified the path on
4. In the share settings you will now have two shares (zfs_linuxcontainers & the path for the LXC directory) that will probably default to Array and won't have to correct settings in them
   (You can only fix that simply by setting the main storage to the ZFS Pool)
5. Created the first container you will see that it created a dataset in zfs_linuxcontainers where it basically doubles the container name eg POOLNAME/zfs_linuxcontainers/CONTAINERNAME/CONTAINERNAME -> this dataset is then mounted to the directory that you've specified as the LXC path eg pool/lxc/CONTAINERNAME

Again, not arguing.

 

I know that you know how it works. What I'm trying to explain is how anyone else besides the person who created this sees it. There's a big difference.

 

 

Install plugin and then go to plugin settings.  As you know, no option for ZFS. But the difference between the author and a user here is that you know that something else CAN appear there after applying. No one else will know that.

 

 

 

After pressing Apply of course as you mentioned:

 

 

But no one installing this is going to look at that menu again the second time because they will never expect that it changes after applying the setup options. And that you have to make a second change and Apply a second time. After pressing Apply the first time, I suspect 99.9% of people will leave this setup page and go to the LXC tab to start creating a container. Just like with most of Unraid setup tabs/pages.

 

I can see that after the initial Apply, the nvme_pool/lxc path is created as a normal folder.

 

Because you've explained it here in the forum, now I choose ZFS and apply again. Then go to the LXC tab and create a container without starting it, called Debian. This creates 3 datasets as you mentioned, one of which points to the rootfs of the container.

 

 

So now let me explain how I would expect this to work, not knowing any of what you told me:

 

When the plugin starts, it checks the volumes and keeps track of their filesystems. The backing option is populated right away to include all relevant options. In my case that means dir and ZFS are there from the start.

 

Since it is not recommended to put the container folder in the FUSE path, I would not expect it pre-populated with the fuse path by default. Even leaving it empty is better here so the settings can’t be applied until the warning is read and a path entered manually.

 

So I specify the storage path outside /user fuse path, in my case on my pool, nvme_pool. I choose the ZFS backing option and then press apply.

 

Now the container path, which I have called “lxc” is created as a ZFS dataset exactly where I specified it should be be: nvme_pool/lxc

 

Next I go to the LXC tab and create a container (Debian) and press apply

 

The plugin creates a “nvme_pool/lxc/Debian” dataset and a rootfs dataset (if needed) under the Debian.

 

This would match the behavior of the VM system in Unraid, which (when using ZFS) also has a dataset parent and dataset to store the vdisks:

 

 

That's it. 🍻🍻

 

 

 

 

Edited September 23, 20241 yr by Espressomatic

12 minutes ago, Espressomatic said:

But no one installing this is going to look at that menu again the second time because they will never expect that it changes after applying the setup options. And that you have to make a second change and Apply a second time. After pressing Apply the first time, I suspect 99.9% of people will leave this setup page and go to the LXC tab to start creating a container. Just like with most of Unraid setup tabs/pages.

Can't do much about that for now...
I can't easily detect the filesystem before the path is set for LXC, however I would strongly recommend that you stay on the page because there are also other things to discover like the Unraid unique Backup feature. or disabling the dynamic update feature for containers on the LXC page and various other settings.

 

16 minutes ago, Espressomatic said:

When the plugin starts, it checks the volumes and keeps track of their filesystems. The backing option is populated right away to include all relevant options. In my case that means dir and ZFS are there from the start.

Thanks for the recommendation, I already have that on my todo list but it's way down at the bottom.

 

17 minutes ago, Espressomatic said:

Since it is not recommended to put the container folder in the FUSE path, I would not expect it pre-populated with the fuse path by default. Even leaving it empty is better here so the settings can’t be applied until the warning is read and a path entered manually.

I'm not a friend from leaving it empty since Docker or VM Settings are also defaulting to the FUSE path which I would also not recommend using the FUSE path and that's why I put the big red text that says to not use the FUSE path (the text will be always displayed if /mnt/user is used).

I think the red warning text is enough for now and leaving it empty is not that what I want to do.

 

21 minutes ago, Espressomatic said:

Now the container path, which I have called “lxc” is created as a ZFS dataset exactly where I specified it should be be: nvme_pool/lxc

 

Next I go to the LXC tab and create a container (Debian) and press apply

 

The plugin creates a “nvme_pool/lxc/Debian” dataset and a rootfs dataset (if needed) under the Debian.

Sorry that's not possible, please look at the screenshot form the datasets that I posted above:

 

Let's imagine this:

You have a container called Debian and another one called Arch like in the example above and you have for both the datasets:
nvme_pool/lxc/Debian & nvme_pool/lxc/Arch

What happens when you create a LXC Snapshot it will create Snapshot at the dataset:

nvme_pool/lxc/snap0

When you create another snapshot for Arch it tries to create a Snapshot at:

nvme_pool/lxc/snap0

will fail with an error message.

 

I can't do anything about that since that's how LXC does call the ZFS filesystem functions and how it creates the datasets for containers and snapshots on Linux.

It is also not possible to mount the dataset which LXC creates (nvme_pool/lxc/Debian) to a directory that lives under that dataset.

The main reason why I had to do it like that is because usually LXC always creates a dataset which you have to mount somewhere and usually the directory is /var/lib/lxc/CONTAINERNAME but since Unraid has no persistent storage at /var I can't do it like that and I have to mount the container somewhere that is visible to the user and that is directly a share.

 

Sure I could also create a Image with a selectable filesystem like it is the case for Docker but I really would like to avoid that since then you are maybe limited by the fixed size from that image and also is a waste of hard drive space, complicates the set up even more and not to speak of the performance penalty because mounting it as a loopback image.

The other possible solution is that a user only specifies the ZFS Pool where the datasets should be created and I don't mount them by default, you shouldn't mess with the files anyways from outside the container <- it is always better to mount a path from the host inside the container and copy files that way (or through SMB, NFS, SSH,...).

IIRC, LXC (since version 6.0 I think) doesn't mount the datasets anymore on the host if you are using ZFS as backing storage type.

 

I really appreciate the response and discussion, I also hope that explains that a bit more in detail why I did it how I did it.

15 minutes ago, ich777 said:

I really appreciate the response and discussion, I also hope that explains that a bit more in detail why I did it how I did it.

 

Cheers, yes I recognize you're dealing with limitations imposed by both LXC and Unraid. And again, thank you for putting this together and releasing it. Being able to put Tailscale together with NPM (so quickly and so painlessly) has made deploying all the services across my entire network securely to the WAN, SUPER EASY.

If you decide to refactor in the future, I would suggest trying to align the user experience to Unraid's VM system.

 

Right now all this deployment and testing has been on my "Edge" system where I run pfSense and only other Edge functions like DNS and Reverse Proxy. In that regard, this container doesn't have to be performant in snapshots nor coexist with other containers as I may only have one LXC  here.

If I can find other reasons to use LXC containers, then my Apps/Services system can handle many of them.

Edited September 23, 20241 yr by Espressomatic

3 minutes ago, Espressomatic said:

If you decide to refactor in the future, I would suggest trying to align the user experience to Unraid's VM system.

I'm rather thinking about removing the mount points for the ZFS datasets entirely since I just looked it up and the default is to not mount it when using ZFS.

 

The VM system has a unfair advantage, Unraid creates the shares automatically which I want to avoid since it should be up to the user where it should be located and the red text is enough for now I think since you've also read it and I assume every user that has set the path to /mnt/user/

 

4 minutes ago, Espressomatic said:

If I can find other reasons to use LXC containers, then my Apps/Services system can handle many of them.

Just look here: https://github.com/ich777/unraid_lxc_pihole

 

With that container you can run two instances of PiHole on two different machines, through the magic of keepalived with a virtual IP so that your DNS server never goes down and nobody yells at you why the Internet is not working when the main PiHole is down.

9 minutes ago, ich777 said:

so that your DNS server never goes down and nobody yells at you why the Internet is not working when the main PiHole is down

 

The yelling at me is always going to happen, I don't think I can prevent that even if I had a 5 person IT department working under me at my house with 99.9% uptime.

 

If I have to do a fast reboot on my router while setting something up, it's a chorus of "what's wrong with the wifi?"

 

I'm running AdGuard Home (in place of PiHole) on the same system as pfSense, so if the system goes down, it's everything gone. I'd like to have some redundancy for the entire system using other servers I already have, but at the moment it's a question of some extra cabling work and configuration on switches - which I need some time to figure out.

 

I'd otherwise be pretty nervous about relying on general-purpose hardware to run this stuff, but I've had amazing stability with Unraid for over 5 years, I had no reservations about setting up a dedicated system for routing/firewall/dns/proxy.

I try to setup "mirrors.tuna.tsinghua.edu.cn/lxc-images" as LXC mirror in LXC Container URL, but it wont allow me to save the setup

27 minutes ago, trott said:

I try to setup "mirrors.tuna.tsinghua.edu.cn/lxc-images" as LXC mirror in LXC Container URL, but it wont allow me to save the setup

What is the specific error message or what does the plugin do?

have you yet tried it without the suffix /lxc-images

6 hours ago, ich777 said:

What is the specific error message or what does the plugin do?

have you yet tried it without the suffix /lxc-images

it shows in Chinese, basically it says the Format is not match,  I guess it won't link "/" in the URL, but "mirrors.tuna.tsinghua.edu.cn/lxc-images" is a valid mirror,  without "/lxc-images" it will not working

8 hours ago, trott said:

it shows in Chinese, basically it says the Format is not match,  I guess it won't link "/" in the URL, but "mirrors.tuna.tsinghua.edu.cn/lxc-images" is a valid mirror,  without "/lxc-images" it will not working

I'll look into that, please give me a few days.

 

Please update the plugin, tested and working with the provided URL.