jortan

Well that confirms it - ZFS lacks sendfile syscall support, at least on Unraid. This should be configurable in nginx, it might be fairly simple to disable this as presumably that file will be stored in appdata. Look for nginx.conf and just change "sendfile = on" to "sendfile = off" I had to do some scripting to ensure sendfile is disabled in my lancache docker as the relevant configuration file was inside the docker image, not inside appdata, so my changes were overwritten every docker update. As an alternative, swag docker doesn't have this issue, though it doesn't have the nice front-end of nginx-proxy-manager

I also had unusual problems with certain dockers trying to use docker in directory mode on ZFS last time I tried it. Glad you got it working.

The df output for ZFS pools is incorrect

I have an unraid server for testing that uses a single 8GB thumb drive for the array. You don't need to assign a parity device. Keep in mind that by default the "system" share (libvirt image, docker image) is going to be placed on the array, as presumably you also won't have an unraid array cache device either. If you're going to use a thumb drive for your array + ZFS pool for storage, you will want to make sure all of these point to appropriate locations in your ZFS pool: Settings | Docker - Docker vDisk location - Default appdata storage location Settings | VM Manager - Libvirt storage location - Default VM storage path - Default ISO storage path Note that some dockers from Community Applications ignore the default appdata storage location, and will still default to: /mnt/user/appdata/xxxx Make sure you check these and change to a path within your pool when adding any new docker applications I'm no ZFS expert, but I'm not sure that this is a good idea. From what I understand, this setting could add to write amplification for asynchronous writes and cause other performance issues. For a dataset of bluray images, this makes sense. Not so much for dockers/vms. The ZFS ARC will use up to half your system memory for caching by default (I think?) - but is also very responsive to other memory demands from your system and will release any memory required by other processes. In most cases it's best to just let the ZFS ARC use whatever memory it can - unless you have large databases or other processes that could make better use of unallocated memory. You should still setup a user script to run "zpool trim poolname" manually in addition to this: https://askubuntu.com/questions/1200172/should-i-turn-on-zfs-trim-on-my-pools-or-should-i-trim-on-a-schedule-using-syste

Just helped someone diagnose something with very similar errors. This was caused by the NVME controller being configured for passthrough in Tools | System Devices - even though this device had never been configured for passthrough. This happened due to a single unrelated pcie device being removed - this changed pcie device assignments in some way that caused the wrong devices to be enabled for passthrough.

Don't want to labour the point, but it matters if your use case isn't huge streaming writes as in the case off chia plotting. For most people: Chia plotting = Abysmal ZFS pool for some dockers and VMs = Great The Firecuda 520 isn't in this graph, but most (consumer) nvme devices have similar reduction in write performance after their fast-cache fills up:

Agreed, Firecuda 520 not optimal for chia plotting. You're hitting SLC cache limits as well as potentially reduced write speed due to TLC / drive filling up: https://linustechtips.com/topic/1234481-seagate-firecuda-520-low-write-speeds/?do=findComment&comment=13923773 I'm doing some background serial plotting now, but just using an old 1TB SATA disk and -2 pointing to a 110GB ramdisk. Takes about 5 hours per plot, but Chia netspace growth has really levelled off now, so I'm less keen to burn through SSDs/nvmes: https://xchscan.com/charts/netspace I was more interested in TBW rating. Doesn't compare to enterprise SSD, but good ratings compared to other consumer nvme. I'm hoping these will run my dockers/VMs for 5 years or more.

For what workload? Outside of "370GB of sustained writes filling up the pSLC cache" scenarios, they seem to perform well I'm currently using 2 x Firecuda 520 nvmes in RAIDZ1 pool (for possible raidz expansion later) No issues encountered, though mine are sitting behind a pcie 3.0 switch.

Any shares configured in /boot/config/smb-extras.conf will appear after restarting samba service: /etc/rc.d/rc.samba restart

If you want to make the ZFS datasets open to everyone, on the unraid server with your zfs pool: chown -R nobody:users /zfs/movies chown -R nobody:users /zfs/music chown -R nobody:users /zfs/tv That should probably fix it, but if not you could try this as well: chmod -R 777 /zfs/movies chmod -R 777 /zfs/music chmod -R 777 /zfs/tv

I haven't watched space invader's video - how was the SMB share created? I shared a ZFS dataset by adding the the share to: /boot/config/smb-extra.conf [sharename] path = /zfs/dataset comment = zfs dataset browseable = yes public = yes writeable = yes vfs objects = If I remember correctly, you can then restart samba with: /etc/rc.d/rc.samba restart

They're saying, in the nicest possible way, that BTRFS is not stable in RAID5 mode: >>btrfs today is most reliable when configured in RAID 1 or RAID 10 Seems like all these features will make it in to unRAID eventually and they are just polling in order to set their priorities.

Vote for native ZFS support in unRAID here:

>> I guess just refers to its connection on the local network? Yes. This is common even in an enterprise environment (though they would generally have a more segmented internal network) One of the selling points of reverse proxies is "SSL offloading" meaning the reverse proxy handles the SSL workload and the application server behind it doesn't need to.

Consider using SWAG docker for this. This would give you a single place to configure LetsEncrypt certificate, a single port to forward and can be configured as a front-end to multiple non-SSL backend web servers. edit: just saw this "behind cloudflare and a reverse proxy" - if you have SSL on the reverse proxy, then it's not really necessary to have SSL enabled on Sonarr? Or is the reverse proxy outside your network?

545G scanned at 4.30M/s, 58.6G issued at 473K/s, 869G total This should give you some idea - 869G allocated in the array, 545G has been scanned and 58.6G written to the replacement disk so far. Hopefully this doesn't confuse the resilvering. Won't cause any problems, but it will slow down the resilvering process. There are some zfs tunables you can modify to change the io priority, but safest thing is probably just to let it complete. Consider turning off any high-io VMs/dockers that you don't need to have running.

It's because ZFS pools might not import on startup if the device locations have changed: https://openzfs.github.io/openzfs-docs/Project and Community/FAQ.html#selecting-dev-names-when-creating-a-pool-linux My not having any issues with this might be down to the fact that unRAID doesn't have a persistent zpool.cache (as far as I know). To each their own!

zpool replace poolname origdrive newdrive Just to clarify, "origdrive" refers to whatever identifier ZFS currently has for the failed disk. So yes, this is 3739555303482842933 (ZFS id, apparently the drive located here has failed to the point where it wasn't assigned a /dev/sdx device) So the command should be zpool replace MFS2 3739555303482842933 sdi As long as you understand that these are how you refer to drives when replacing disks using zpool, there's not much chance of replacing the wrong drive: I understand that's a common recommendation, but in my experience I just reference the normal drive locations /dev/sda, /dev/sdb. ZFS never seems to have any issue finding the correct disks, even when the order has changed. In my array,. ZFS has switched by itself to using drive id's - this may only occurs if the disk order has changed?

This person quite recently is having the same issue on ARM version of plex: https://forums.plex.tv/t/failed-to-run-packege-service-i-tried-many-solutions-but-did-not-work/726127/29 In the end they used a different version of Plex to do the install. Might be worth forcing an older version of the plex docker? edit: Failing that it might be worth a post on Plex forums with a reference to the above thread, noting that you seem to have the same issue on x86 docker. The few issues I've had with Dockers and ZFS seem to involve applications doing direct storage calls that ZFS doesn't support. Maybe the latest versions of Plex are doing something similar during new installs? Have you also tried to copy a working database created on non-ZFS storage?

I have a very similar setup to you (nested datasets for appdata and then individual dockers), I've never run in to this issue. I just checked and it's using read/write - I'm not aware of that causing any issues either? Are there any existing files in the plex appdata folder that you've copied from elsewhere? Could it be permissions related? chown -R nobody:users /mnt/Engineering/Docker/Plex Same issue with a empty /mnt/Engineering/Docker/Plex/ folder owned by nobody?

Nope, mine was also youtube-dl.subfolder.conf and I know I never enabled this as I only use *.subdomain.conf I think somehow in a previous version of swag docker a non-sample conf must have been pushed out. Possibly even from back before this docker was renamed? edit: judging by the file date, this happened early July 2020.

I have a very basic setup and I've just experienced this as well - all sites returning: refused to connect. Nothing logged in access.log or error.log something broke between: 1.17.0-ls70 and 1.17.0-ls71 For anyone else seeing this, edit swag docker and change repo to: linuxserver/swag:1.17.0-ls70 edit: Don't do the above, instead rename: swag/nginx/proxy-confs/youtube-dl.subfolder.conf to this swag/nginx/proxy-confs/youtube-dl.subfolder.conf-notused (unless you do actually use this config file, in which case remove the line containing: proxy_redirect off;

That did it - thank you! For anyone else with this issue, these lines were added: <boot order='1'/> <alias name='usbboot'/> You'll want to check any other instances of "boot order" setting in the XML and make everything else something other than "1"

I forked a script called borgsnap a while ago to add some needed features for Unraid and my use-case. This allows you to create automated incremental-forever backups using ZFS snapshots to a local and/or remote borgbackup repository. I've posted a guide here. This includes pre/post snapshot scripts so you can automate shutting down VMs briefly while the snapshot is taken.

This is a guide for users of ZFS Plugin who want to backup their ZFS pools using borgbackup. pre/post scripts can be used to briefly shutdown dockers/VMs while snapshots are taken. The backup can then proceed using the ZFS snapshot while the dockers/VMs are running. (personally I haven't bothered with this, a crash-consistent snapshot backup is good enough for me) Use case I use sanoid/syncoid but also wanted an encrypted copy of my non-encrypted zfs pool backed up incrementally off-site. I came across borgsnap that was further improved here. borgsnap wasn't suitable for large numbers of nested datasets so I've created a fork that adds recursive ZFS snapshots, some modifications for Unraid and other features here: https://github.com/jortan/borgsnap/blob/master/borgsnap borgsnap can backup to a local destination, a remote destination (via borg over SSH) or both. It will: - Read configuration file and encryption key file - Validate output directory exists and a few other basics - For each ZFS filesystem configured: Initialize borg repository if it doesn't exist Take a ZFS snapshot of the filesystem (recursively if enabled) Run borg for the local output if configured Run borg for the rsync.net output if configured Delete old ZFS snapshots (recursively if enabled) Prune local and remote borg backups if configured and needed Disclaimer I barely speak bash and have modified a script written by someone who does. Use at your own risk! Remove existing borgbackup from NerdPack The "borgbackup" package in NerdPack has broken a couple of times (including as of now). I recommend uninstalling borgbackup from NerdPack if you've installed this previously. The instructions below include downloading a linux binary of borgbackup from here - as far as I can tell works fine, though I'm not sure if it's relying on python / other packages I have installed via NerdPack. Please post any requirements you come across and I'll update this post. Create location for borg respository You can store the borgsnap repository on an array share, unassigned disk, remote mount, or another ZFS pool. You just need an empty folder somewhere. You don't need to create a borg respository, borgsnap will handle that for you. Create single disk ZFS pool for LOCAL borgsnap backup (optional) wipefs -a /dev/sdxx zpool create -o ashift=12 -m /mnt/borgpool borgpool /dev/sdxx zfs create borgpool/borgrepo Configure ZFS tunables to taste - borgbackup is going to compress backups, so no need to compress this pool/dataset. zfs set compression=zle atime=off recordsize=1m xattr=sa borgpool Download borgbackup/borgsnap Check for latest 1.x release here, copy URL for "borg-linux64" binary. Download the borgbackup binary: mkdir /boot/config/borgsnap wget https://github.com/borgbackup/borg/releases/download/1.1.16/borg-linux64 /boot/config/borgsnap/borg-linux64 Download borgsnap scripts to unRAID wget 'https://github.com/jortan/borgsnap/raw/master/borgsnap' /boot/config/borgsnap/ wget 'https://github.com/jortan/borgsnap/raw/master/borgwrapper' /boot/config/borgsnap/ Install binaries/scripts cp /boot/config/borgsnap/borg-linux64 /usr/local/sbin/borg cp /boot/config/borgsnap/borgsnap /usr/local/sbin/ cp /boot/config/borgsnap/borgwrapper /usr/local/sbin/ chmod +x /usr/local/sbin/borg chmod +x /usr/local/sbin/borgsnap chmod +x /usr/local/sbin/borgwrapper echo "alias borgwrap='/usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf'">>/etc/profile Install borgbackup/borgsnap on startup Add commands to /boot/config/go # borgbackup / borgsnap setup # cp /boot/config/borgsnap/borg-linux64 /usr/local/sbin/borg cp /boot/config/borgsnap/borgsnap /usr/local/sbin/ cp /boot/config/borgsnap/borgwrapper /usr/local/sbin/ chmod +x /usr/local/sbin/borg chmod +x /usr/local/sbin/borgsnap chmod +x /usr/local/sbin/borgwrapper echo "alias borgwrap='/usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf'">>/etc/profile Create a borg passphrase file IMPORTANT: borgbackup stores the decryption key with the backups (inside "config" file) but requires a passphrase to decrypt the key. You will not be able to restore your borg backups if you lose access to the passphrase Enter a long random passphrase here: /boot/config/borgsnap/passphrase.key Or generate a passphrase: cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 128 >/boot/config/borgsnap/passphrase.key Store a copy of this somewhere accessible in a DR scenario! No really, do it now! Create borgsnap config file. Note: I have only tested this with first-level zfs datasets (ie. poolname/dataset) /boot/config/borgsnap/borgsnap.conf FS="pool/appdata pool/data pool/vm" BASEDIR="/mnt/user/appdata/borgsnap" LOCAL="/mnt/borgpool/borgrepo" LOCAL_READABLE_BY_OTHERS=false LOCALSKIP=false RECURSIVE=true COMPRESS=zstd CACHEMODE="mtime,size" REMOTE="" REMOTE_BORG_COMMAND="" PASS="/boot/config/borgsnap/passphrase.key" MONTH_KEEP=1 WEEK_KEEP=4 DAY_KEEP=7 PRE_SCRIPT="" POST_SCRIPT="" IMPORTANT: All options must be present in the borgsnap.conf file, even if they have no value. BASEDIR is required for Unraid. This may take a few gigabytes, put this anywhere persistent. It's not required for a restore, but you don't want to lose borgbackup's cache every reboot. If you don't specify a "LOCAL" path, borgsnap will create a repository inside the dataset being backed up. You can skip doing LOCAL backups entirely and only create a REMOTE backup by setting LOCALSKIP=true If you want to create borg backups on rsync.net: REMOTE="[email protected]:myhost" In my case I installed borg on a remote Ubuntu machine, configured ssh key to access the remote server. From unraid: ssh-keygen ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] You can then specify the remote backup destination like this: REMOTE="[email protected]:/remote/path/borgrepo" By default, the remote borg command is configured for rsync.net - if using your own borg installation, you will probably need to set: REMOTE_BORG_COMMAND="borg" Run borgsnap! You can run borgsnap manually with this command: /usr/local/sbin/borgsnap run /boot/config/borgsnap/borgsnap.conf Note that if borgsnap is interrupted, it won't run again properly until the next day. You may want to add this to User Scripts and use "Run in background" so that closing your shell won't stop borgsnap. If something didn't work and you want to try again, you can use the "tidy" command - this will attempt to unmount borgsnap's ZFS snapshots and remove any local/remote backups for the current day. This isn't perfect but helps when initially trying to configure borgsnap: /usr/local/sbin/borgsnap tidy /boot/config/borgsnap/borgsnap.conf Pre/Post Scripts You can nominate a script to run before taking a ZFS snapshot and after a ZFS snapshot. Specify the full path to the script: PRE_SCRIPT="/mnt/user/appdata/borgsnap/prescript.sh" POST_SCRIPT="/mnt/user/appdata/borgsnap/postscript.sh" Note: You can't make files executable in /boot/config so these need to go somewhere else persistant. Keep in mind these scripts are going to be run as root so you may want to set appropriate permissions with something like: chmod 700 /mnt/user/appdata/borgsnap/prescript.sh chmod 700 /mnt/user/appdata/borgsnap/postscript.sh The same script is run for every FS (pool/dataset) configured in borgsnap.conf. The FS name is passed to the scripts as $1, so you can use this to run commands specific to a pool/dataset. Example: /mnt/user/appdata/borgsnap/prescript.sh #!/bin/bash if [[ "$1" = "pool/appdata" ]]; then echo pool/appdata - Stopping all dockers docker stop $(docker ps -aq) sleep 10 fi /mnt/user/appdata/borgsnap/postscript.sh #!/bin/bash if [[ "$1" = "pool/appdata" ]]; then echo pool/appdata - Starting all dockers docker start $(docker ps -aq) sleep 10 fi Note: This starts all containers, not just those configured to auto-start in the docker service. There's presumably a better command for this. User Scripts borgsnap-run Used to run borgsnap on a schedule. /usr/local/sbin/borgsnap run /boot/config/borgsnap/borgsnap.conf borgsnap-local-backup-summary Inside the LOCAL or REMOTE repo folder, borgsnap creates A folder for each zfs pool being backed up A sub-folder for each dataset being backed up Each of those sub-folders contains a borg repository. This script will parse LOCAL folder and give a summary of borg backups and repo sizes: #!/bin/bash source /boot/config/borgsnap/borgsnap.conf echo $'\n Summary of all borgsnap backups\n' echo " Source datasets: $FS" echo $'' echo " Backup repository: $LOCAL" echo $'' echo " Archives sizes: Original size, Compressed size, Deduplicated size" echo $'' for z in $LOCAL/*; do for f in $z/*; do if [ -d "$f" ]; then echo $'-------------------------------------------------------\n' /usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf list "$f" | cut -d" " -f1 | xargs -d "\n" -L1 echo $f:: | tr -d ' ' echo $'' /usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf info "$f" | awk 'NR==8' echo $'' fi done done Example output: Summary of all borgsnap backups Source datasets: pool/appdata pool/vm Backup repository: /mnt/borgpool Archives sizes: Original size, Compressed size, Deduplicated size ------------------------------------------------------- /mnt/borgpool/pool/appdata::month-20210328 /mnt/borgpool/pool/appdata::week-20210328 /mnt/borgpool/pool/appdata::day-20210329 All archives: 7.63 MB 134.86 kB 46.04 kB ------------------------------------------------------- /mnt/borgpool/pool/vm::month-20210328 /mnt/borgpool/pool/vm::week-20210328 /mnt/borgpool/pool/vm::day-20210329 All archives: 209.32 GB 105.99 GB 27.76 GB I'll add a REMOTE version of this script at some point. Run borgbackup commands interactively You can use "borgwrapper" to run arbitrary borg commands using the keyfile passphrase in borgsnap.conf: borgwrapper /boot/config/borgsnap/borgsnap.conf <borg commands> We added an alias in /boot/config/go to make this even easier to run (restart your shell if this doesn't work yet) borgwrap <borg commands> Example commands: List contents of a borg repository: borgwrap list /mnt/borgpool/pool/appdata