Reboot required to apply Unassigned Devices update + ZFS

[ich777]

3 hours ago, DaKarli said:

but led to problems with docker on my zfs share (I've read the warning but nevertheles tried it).

Do I understand that right that you've had issues with your Docker(s) when using the path /mnt/disks/whatever?

I now run basically everything from ZFS (mounted to /mnt/nvme), my Docker path for the Docker images, libvirt image, appdata, domains, system,... and had never had a single issue with it.

 

@steini84 also told me now that multiple people report they have issues doing it this way but I don't haven't got a single issue so far.

 

1 hour ago, DaKarli said:

Another puzzle completing the big picture.

I think since this is a plugin that extends Unraid in it's functions there is no right or even wrong way to do it.

It's always up to the user where you set your mount points for ZFS and from my perspective if it works for you, then leave it as it is...

[dlandon]

For the record, UD does not limit writes to deviices properly mounted at /mnt/disks/.  The protection is for incorrect writes directly to /mnt/disks/ that end up in the tmpfs.  Those writes would not be written to a device, but instead to ram file system.

[DaKarli]

19 hours ago, ich777 said:

Do I understand that right that you've had issues with your Docker(s) when using the path /mnt/disks/whatever?

I now run basically everything from ZFS (mounted to /mnt/nvme), my Docker path for the Docker images, libvirt image, appdata, domains, system,... and had never had a single issue with it.

 

@steini84 also told me now that multiple people report they have issues doing it this way but I don't haven't got a single issue so far.

 

I think since this is a plugin that extends Unraid in it's functions there is no right or even wrong way to do it.

It's always up to the user where you set your mount points for ZFS and from my perspective if it works for you, then leave it as it is...

 

Unfortunately, Docker on ZFS still gives me trouble (just tried again to confirm, see the following pictures and description).

VMs on ZFS work, even with the libvirt.img residing on ZFS but as a security measure I still keep the libvirt.img in the standard folder to ensure it is not the .img which causes trouble.

 

Setting Docker to a path on my ZFS pool the installation e.g. for Krusader fails/hangs at the shown point and the GUI does not respond anymore.

I can only access the machine on the host terminal and even shutdown -r -n now does not work, so a hard-reset is necessary.

 

I tried to install Krusader withContainer Path = /mnt/z-syspool/UnRaid/appdata/krusader/

Spoiler

My docker.cfg looks like this:

Spoiler

DOCKER_ENABLED="yes"
DOCKER_IMAGE_FILE="/mnt/z-syspool/UnRaid/system/docker/docker.img"
DOCKER_IMAGE_SIZE="20"
DOCKER_APP_CONFIG_PATH="/mnt/z-syspool/UnRaid/appdata/"
DOCKER_APP_UNRAID_PATH=""
DOCKER_CUSTOM_NETWORKS=" "
DOCKER_TIMEOUT="10"
DOCKER_LOG_ROTATION="yes"
DOCKER_LOG_SIZE="50m"
DOCKER_LOG_FILES="1"
DOCKER_AUTHORING_MODE="no"
DOCKER_USER_NETWORKS="remove"

The docker.log:

Spoiler

time="2022-03-21T12:17:07+01:00" level=warning msg="deprecated version : `1`, please switch to version `2`"

The error.log (IP addresses and names have been "x"ed):

By the way, the same error could be seen with IPv4 so IPv6 is not the reason.

Spoiler

2022/03/21 12:30:54 [error] 27760#27760: *8320 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2001:x:x:x:x, server: , request: "GET /plugins/dynamix.local.master/include/LocalMaster.php HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user/my-Krusader.xml"

2022/03/21 12:30:54 [error] 27760#27760: *8320 auth request unexpected status: 504 while sending to client, client: 2001:x:x:x:x, server: , request: "GET /plugins/dynamix.local.master/include/LocalMaster.php HTTP/1.1", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user
/my-Krusader.xml"

2022/03/21 12:30:54 [error] 27760#27760: *8329 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2001:x:x:x:x, server: , request: "GET /plugins/unassigned.devices.preclear/assets/sweetalert2.js?_=1647861614714 HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user/my-Krusader.xml"

2022/03/21 12:30:54 [error] 27760#27760: *8329 auth request unexpected status: 504 while sending to client, client: 2001:x:x:x:x, server: , request: "GET /plugins/unassigned.devices.preclear/assets/sweetalert2.js?_=1647861614714 HTTP/1.1", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user/my-Krusader.xml"

2022/03/21 12:30:56 [error] 27760#27760: *8356 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2001:x:x:x:x, server: , request: "GET /plugins/dynamix.my.servers/webComps/unraid.min.js?v=1647042912 HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user/my-Krusader.xml"

 

2022/03/21 12:30:56 [error] 27760#27760: *8356 auth request unexpected status: 504 while sending to client, client: 2001:x:x:x:x, server: , request: "GET /plugins/dynamix.my.servers/webComps/unraid.min.js?v=1647042912 HTTP/1.1", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/temp
lates-user/my-Krusader.xml"

2022/03/21 12:32:37 [error] 27760#27760: *7791 upstream timed out (110: Connection timed out) while reading upstream, client: 2001:x:x:x:x, server: , request: "POST /Apps/
AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user/my-Krusader.xml HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "ryzenbrain.xxx.xx", referrer: "http://ryzenbrain.xxx.xx/Apps/AddContainer?xmlTemplate=user:/boot/config/plugins/dockerMan/templates-user/my-Krusader.xml"

 

File permissions also look ok on the regular share and on the ZFS share:

Spoiler

 

Acccording to the error.log, I'd say it is not a problem with the .img files themself but more of a problem somewhere in the way PHP is accessing these files?

 

If there is something else you want me to try, just drop me a line.

 

Regards

DaKarli.

..

Edited March 21, 2022 by DaKarli
took out double picture

[ich777]

29 minutes ago, DaKarli said:

If there is something else you want me to try, just drop me a line.

As said above, I've run a Docker Path and not the Docker Image on my ZFS Pool (/mnt/nvme), also maybe try to use a pool that has no special characters in it's path, only letters and numbers, maybe that's causing the issue.

 

Here is my docker.cfg

Spoiler

DOCKER_ENABLED="yes"
DOCKER_IMAGE_FILE="/mnt/nvme/system/docker/"
DOCKER_IMAGE_SIZE="30"
DOCKER_APP_CONFIG_PATH="/mnt/nvme/appdata/"
DOCKER_APP_UNRAID_PATH=""
DOCKER_CUSTOM_NETWORKS=" "
DOCKER_LOG_ROTATION="yes"
DOCKER_LOG_SIZE="50m"
DOCKER_LOG_FILES="1"
DOCKER_AUTHORING_MODE="no"
DOCKER_USER_NETWORKS="preserve"
DOCKER_TIMEOUT="10"
DOCKER_IMAGE_TYPE="folder"
DOCKER_DHCP_ETH1="10.0.1.0/25"
DOCKER_NETWORK_TYPE="1"

 

36 minutes ago, DaKarli said:

shutdown -r -n now

I would strongly recommend that you use:

powerdown -r

for a restart or:

powerdown

for a shutdown from your server but anyways this wouldn't changed much in your case that the server hung up.

 

39 minutes ago, DaKarli said:

VMs on ZFS work, even with the libvirt.img residing on ZFS but as a security measure I still keep the libvirt.img in the standard folder to ensure it is not the .img which causes trouble.

Only having one "test" Windows 11 VM image on my ZFS Pool, maybe it's also related how you set up the pool. I'm only running a mirror without anything special but with that parameters in the go file:

Spoiler

# ZFS Tweaks, Autotrim & 8GB ARC memory limit
zpool set autotrim=on SSD
echo 8589934592 >> /sys/module/zfs/parameters/zfs_arc_max

 

I run it this since the first Unraid 6.10.0 RC series came out on my server without an issue so far, also have my Krusader container installed and it works without a hitch.

[DaKarli]

18 hours ago, dlandon said:

For the record, UD does not limit writes to deviices properly mounted at /mnt/disks/.  The protection is for incorrect writes directly to /mnt/disks/ that end up in the tmpfs.  Those writes would not be written to a device, but instead to ram file system.

dlandon, thanks once again for the further clarification.

 

So in fact what I've done was already fine and counts as a proper mount, right?:

#mount
...
syspool/UnRaid on /mnt/disks/z-syspool/UnRaid type zfs (rw,noatime,xattr,posixacl)
tmpfs on /mnt/disks type tmpfs (rw,relatime,size=1024k,inode64)
...

Regards

DaKarli

[DaKarli]

 

17 minutes ago, ich777 said:

As said above, I've run a Docker Path and not the Docker Image on my ZFS Pool (/mnt/nvme)

But looking at your docker.cfg it looks like your docker.img resides on your ZFS as well, so in the end we have the same setup regarding the docker.img and appdata path.

 

42 minutes ago, ich777 said:

DOCKER_IMAGE_FILE="/mnt/nvme/system/docker/"

Or do I get something wrong?

 

17 minutes ago, ich777 said:

also maybe try to use a pool that has no special characters in it's path, only letters and numbers,

This shouldn't be a problem as my pools had this name-scheme since long time ago: https://docs.oracle.com/cd/E23824_01/html/821-1448/gbcpt.html

Nevertheless I'll give it a try when I have a spare harddisk or maybe on a USB stick.

 

18 minutes ago, ich777 said:

Only having one "test" Windows 11 VM image on my ZFS Pool, maybe it's also related how you set up the pool. I'm only running a mirror without anything special but with that parameters in the go file:

  Hide contents

# ZFS Tweaks, Autotrim & 8GB ARC memory limit
zpool set autotrim=on SSD
echo 8589934592 >> /sys/module/zfs/parameters/zfs_arc_max

 

I also limited the memory use of ARC cache and my ZFS pools are created as mirrors like this:

zpool create -O compression=lz4 -o ashift=12 -o autotrim=on -O xattr=sa -O acltype=posixacl -O aclinherit=passthrough -O atime=off -O relatime=off -O dnodesize=auto -O mountpoint=/mnt/z-syspool -O normalization=formD syspool mirror nvme1 nvme2

The same for the dataset:

zfs create -o snapdir=visible -o recordsize=16k -o mountpoint=/mnt/z-syspool/UnRaid syspool/UnRaid

...so nothing special or out of scope for zfs here.

Nevertheless, I'll also try a plain flat standard ZFS pool if I have a spare disk.

 

55 minutes ago, ich777 said:

powerdown -r

Made no difference - a hard reset is still necessary. I only used "shutdown" because it is one of the linux standard tools you can use and it uses a shutdown procedure if you set one up.
 

Actually looking at what "powerdown" actually does, I found something very interesting:

/usr/local/sbin/powerdown     

#!/bin/bash
logger "/usr/local/sbin/powerdown has been deprecated"
if [[ "$1" == "-r" ]]; then
  /sbin/reboot
else
  /sbin/init 0
fi

So even this command does nothing else than reboot or init 0 a system - the same what shutdown does.

 

Due to the fact that the system does nothing and does not reboot, I use a hard method to bring the system down by using this (!! use with extreme caution on a work system !!)

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

As a result with this I don't have to push the Reset button...

 

To see some debug or whatever, I tried to run the docker run command above directly in a shell with the -D option, but with or without, I dont get any debug information and the system simply hangs at this screen:

Spoiler

At this point, my knowledge about Docker and what to do if it doesnt work, ends...

And I don't know if I have enogh time to dig deeper to find out what stops Docker to install/start my Docker image (in this case I've choosen Krusader).

 

As a last measure I will try to do all of this on a clean ZFS setup (as mentioned above) without my optimizations and see if that helps.

I'll come back here to tell 🙂

 

Regards

DaKarli.

[ich777]

6 minutes ago, DaKarli said:

But looking at your docker.cfg it looks like your docker.img resides on your ZFS as well, so in the end we have the same setup regarding the docker.img and appdata path.

I think you overlooked this line:

 DOCKER_IMAGE_TYPE="folder"

I don't have a docker.img

 

6 minutes ago, DaKarli said:

Or do I get something wrong?

In the above location are the files and folders that sit in your case in the docker.img but in my case without a image, they are stored directly in this path.

 

7 minutes ago, DaKarli said:

This shouldn't be a problem as my pools had this name-scheme since long time ago: https://docs.oracle.com/cd/E23824_01/html/821-1448/gbcpt.html

Yes sure this is the namig scheme from the ZFS documentation but maybe something on Unraid doesn't like that or Docker... Whatever the case may be, at least you can try it.

As said, I run Docker and everything from my ZFS Mirror without a issue so far.

 

8 minutes ago, DaKarli said:

Made no difference - a hard reset is still necessary. I only used "shutdown" because it is one of the linux standard tools you can use and it uses a shutdown procedure if you set one up.

Yes, but if I'm not mistaken this is the recommended way to shutdown Unraid, but I think that there are many ways to shutdown Unraid.

 

 

[DaKarli]

57 minutes ago, ich777 said:

I think you overlooked this line:

 DOCKER_IMAGE_TYPE="folder"

I don't have a docker.img

 

In the above location are the files and folders that sit in your case in the docker.img but in my case without a image, they are stored directly in this path.

 

Yes sure this is the namig scheme from the ZFS documentation but maybe something on Unraid doesn't like that or Docker... Whatever the case may be, at least you can try it.

As said, I run Docker and everything from my ZFS Mirror without a issue so far.

 

Yes, but if I'm not mistaken this is the recommended way to shutdown Unraid, but I think that there are many ways to shutdown Unraid.

 

 

Hey ich77,

yes I overlooked this line. Ok, now I understand why your Dockers are working... Because they are using the regular file system and are not mounted within a .img. That should be very fine with ZFS as well.

Finally I'll switch to exactly this as I see no advantage of using an .img file for Docker compared to a directory.

 

Regarding the shutdown you are right, one should use the given shutdown script which in this case is the powerdown cmd even though in fact it does nothing special.

For me testing this system I actually don't need to care as my only intention is to bring down the system by any means... 😈

 

In the meantime I found an empty USB stick and tried what I said above.

Did a clean zfs create without any special options and with testpool as its name.

 

In short without further explanation or screenshots: Still didn't work this way.

 

I believe it has something to do with the way UnRaid is writing to the docker.img file while this file sits on ZFS.

First I thought it may additionally has to do with the btrfs filesystem inside the docker.img file which causes trouble when on ZFS, but trying XFS didn't work either. Hangs at the very same point.

So for the moment this will be the end of my investigations because a working solution has been found which has no drawbacks in my opinion.

 

Thanks to all for being with me, with best regards

DaKarli

Edited March 21, 2022 by DaKarli

[DaKarli]

@ich777 Just a quick question:

After changing to "directory" instead of docker.img I see a lot of datasets created under my ZFS datapool.

Do you see/have the same outcome?

Spoiler

 

[ich777]

40 minutes ago, DaKarli said:

Do you see/have the same outcome?

Yes, because the Docker ZFS driver is used and they (inages) are now directly on your ZFS Pool.

 

The same happens when you are using BTRFS.