andrebrait

I see this on initialization sometimes. No idea what this is about.

Someone on Reddit linked this behaviour to the unRAID Connect plugin. Have you tried uninstalling it and seeing if this goes away?

Well, after some reboots and some more testing and trying to reproduce the issue, it seems it simply doesn't happen anymore. After more than 10 times reproducing it before reporting. Right now, I'm fairly convinced I stumbled upon a bug that only happens if the drives are all pre-cleared and you start an array from scratch with two parity drives and (?) one XFS (?) data drive + turbo writes (?)

Well, it completely went away. I'm not sure if there was some weird interaction between the disks being pre-cleared or something, but it did not crash anymore, despite me trying my best to reproduce the issue.

I know, but there isn't anything weird in it. Let me reproduce the crash and grab it for you, but my point was: it's probably crashing before it even has the chance to log anything. I'll try enabling logging to the USB drive.

Memtest86+ passed 6 consecutive times and there are no mentions of anything odd in syslog. Also, it seems the syslog server itself goes down with the entire machine. I tried plugging my KVM and, on the terminal, just tail -F /var/log/syslog and also dmesg -wH and there was nothing odd with either. Now it seems to be working with XFS. The only difference is that I named the share `array` instead of `data-array`, and I formatted the drives after trying btrfs. I'll try a clean reboot and array with the same share name to see if I can reproduce the crash. Are there other debug-like things I can do? I'll try enabling logging to the boot drive for a while as well.

Update 3: now it works with XFS and I have no idea why

Update 2: It seems it has no problem with the exact same setup, but using btrfs instead of XFS

Update: I have created a new ZFS Pool using the new 3 drives in RAIDZ1 mode, copied all 4TB of data I currently have over (simple copy and paste via the folder GUI), and scrubbed both pools afterwards - with zero errors found. SMART data looks good on all drives, old and new, and no UDMA errors or anything after the upgrade - years ago I had a bad cable so some HDDs have some UDMA errors, but they stopped increasing after I switches drivers. So I think this rules out the issue being connected to cables, any power-savings tuning or a bad HBA.

The USB boot drive from unRAID also has memtest86+, so you can try choosing that option. Also check the live memory testing plugin you can find in CA. I'm having a similar issue - the machine goes completely unresponsive - but perhaps different than you:

I have an unRAID server I recently upgraded to include an HBA (a Broadcom 9500-8i, whose firmware/BIOS/catalog I upgraded to the latest versions) and a handful more NVMe SSDs. This server has been running flawlessly for the past few weeks with a RAIDZ-2 pool (5x 6TB WD Enterprise HDDs) connected to the HBA + 2x ZFS RAID-1 pool (2x WD Red 2TB SATA SSDs) + 2x ZFS RAID-1 pool (2x Corsair 1TB NVMe SSDs - unRAID caching for the HDDs pool's share). I decided to switch to using an unRAID Array to save power, so I bought 3x 12TB HDDs (2x Seagate Enterprise Capacity, 1x WD HC520) and my plan was to: 1. Pre-clear the drives (OK!) 2. Create an array with dual-parity + 1 data HDD formatted as XFS (OK!) 3. Copy the data from the RAIDZ-2 share via the GUI (Crash randomly during that, machine goes completely unresponsive!) 4. Stop the array 5. Pre-clear 1x12TB drive I have and 4 of the other 6TB drives 6. Add those to the array 7. Have 48TB of dual-parity-protected storage space However, whenever those HDDs are set up as an unRAID array (at least dual-parity), I get a random crash - whether or not they're being used at all! (I once left the machine be, I was not even doing anything, and it froze) - and the machine goes completely unresponsive. The symptoms so far are: 1. Nothing weird in the screen, o dmesg and syslog, as far as I can tell 2. Unresponsive to input via USB 3. Network becomes unavailable - won't even respond to ping 4. As far as I can tell, the system is completely dead. But if I add those 3 HDDs as another ZFS RAIDZ-1 pool, it works without fault! Same HBA, same HDDs, same cables, nothing changed - but I'm using ZFS pools instead of an unRAID Array. Note: Docker and VMs are both disabled. I do have some plugins, but nothing too crazy, I think. The only thing I think could really cause this issue would be running powertop (still from NerdTools), but it doesn't make sense why it won't crash with ZFS only, but will with the array. Note 2: memtest86+ passed 6 times, so I assume my RAM is fine I have not tried safe mode for a long time yet, as I'm hoping someone can help me figure that out. ps.: I'm using turbo writes because the first copy is to the single disk + the 2 parity disks and it would be too slow in the normal mode. nas-diagnostics-20250426-1237.zip nas-diagnostics-20250426-1536.zip

@Alexstrasza I do have a share that is the root encrypted dataset itself. See the repository above, where I create the `appdata-crypt` share.

@Alexstrasza have you tried the strategy to automatically unlock the dataset I present here? https://github.com/andrebrait/unraid-native-zfs-encryption

Ah, great. My thinking was perhaps automating the builds using GitHub Actions, so it can be somewhat easily triggered / the process is more visible to anyone interested

Hey there. Software developer here, but inexperienced with building linux packages in general. Thinking here of maybe forking NerdTools and use your build system either directly or as a base for the fork. It would be nice if the fork could just be a GUI/Update notifier on top of something that could also be used from a CLI. Do you know if there's someone already working on something like that? And how feasible would it be to version the build system so that we could have packages following a couple major release branches of Unraid (e.g. 7.0 and 6.12 right now)? Thinking of selecting the version automatically via the GUI / also update packages on an Unraid upgrade (which I know is possible via some hooks in the plugins).

My guess is that the script is already running like that, from whatever is calling it, if that's the case. Adding that line is what made the service go from Stopped to Running when I reboot my machine (without it, it only goes to Running if the first PWM controller is enabled), so I always assumed it works.

In both 6.12.13 and 7.0.0-beta.3, it seems the Autofan plugin only starts working after I pretend to change something and click "Apply". Even ensuring the service is running by calling the following in /boot/config/go does not seem to be enough to make the fan control kick in. # Start “Dynamix Auto Fan Control” plugin` /usr/local/emhttp/plugins/dynamix.system.autofan/scripts/rc.autofan start &>/dev/null My entire array overheated this morning during a backup operation due to that. EDIT: I've moved on to using this fan control script, which works like a charm every time https://github.com/IDmedia/fan-control-script

So far, I've had great luck with the Transcend JetFlash 780 MLC line. I have them running on two unRAID machines for over a year now (specifically the 8GB version, TS8GJF780). Their flash is supposed to have higher endurance due to being MLC instead of TLC or QLC. https://www.amazon.com/Transcend-JetFlash-Flash-Drive-TS16GJF780 https://www.amazon.nl/Transcend-TS32GJF780-JetFlash-Stick-interface/dp/B006VAOM6G -> where I bought them

Hey there Whenever I boot up my machine, the plugin doesn't run (it says "Status: Stopped") until I change something in one of the configurations and click apply. Is that a bug or is it intended behavior?

When you encrypt via unRAID GUI, the encryption is done at a level above the filesystem, so it's expected it won't mount or give you any actual ZFS errors because the disk isn't available yet. This is different than creating an encrypted ZFS dataset, for which I have a tutorial here: https://github.com/andrebrait/unraid-native-zfs-encryption

This guide currently exists as a README in a GitHub repository. I will be porting it to this post in the coming days. Link here: https://github.com/andrebrait/unraid-native-zfs-encryption TL;DR unRAID does not provide native GUI support for encrypted ZFS datasets, which provide a more granular way of controlling encryption in ZFS pools. Alternatives do exist, but they usually rely o keyfiles located in some external media or location. This guide provides a way to get essentially equivalent GUI support in unRAID, without exposed keys in some external drive. Why use native ZFS encryption? unRAID already supports Full Disk Encryption (FDE) through LUKS in combination with ZFS pools. The support is solid and LUKS is a widely used tool with great performance. So, why use native ZFS encryption then? Here are a few pros and cons, starting with the cons: Cons of ZFS encryption relative to LUKS LUKS is transparent to the filesystem and does not depend on any filesystem-specific tooling and it's easy to find help on the internet, should you need it. LUKS is fully supported by unRAID, meaning the GUI and the interaction flows are thought out with it in mind. We will actually tap into this support in this document in order to bring a "similar" level of support for ZFS encryption, but it is still not officially supported, meaning unRAID might change something in the future. It's unlikely, but it could happen. ZFS encryption does not encrypt empty disk blocks, nor file metadata. LUKS encrypts everything, including file metadata and the filesystem structured themselves If absolute secrecy is your priority and you're willing to manage encryption on your backups as well, you might want to go with LUKS instead. However, ZFS encryption is not weak. Keep reading! Pros of ZFS encryption relative to LUKS Unlike LUKS, ZFS encryption can be turned on and tweaked at the dataset level, meaning for a single pool you can have a mix of encrypted and regular datasets. That means you can have both secrecy for your data and speed for your applications e.g. place an app's data in an encrypted dataset and its configuration files, database or log files in a regular dataset #### PENDING FORMATTING ###### * ZFS replication does **_not_** require decryption! * This is a major advantage. Not only can you do `zfs send` without loading the decryption key to your dataset, but you can also replicate your dataset in another host with complete secrecy. The remote host does not need to have your decryption key, nor does it need to receive the data decrypted for it to then encrypt again * All other filesystem features are still available and work as expected. * TRIM, compression and other features are all available with compression with no limitations or further loss of security implications. * Yes, LUKS does support TRIM as well, but it's disabled by default and it has security implications due to free space not being encrypted, defeating one of the advantages of FDE over ZFS's native encryption. * Enabling LUKS requires formatting the entire drive. Creating an encrypted ZFS dataset does not, and it's as easy as creating any regular dataset with a couple more options turned on. # The state of support in unRAID As of now (August 2024) unRAID lacks GUI support for ZFS encryption. The _ZFS Master_ plugin does so, but it requires the user to interact with it in order to unlock the dataset (via _password_ or _key file_). That means the encrypted dataset would only be available for unlocking after the array is started and it would take a while to do so, making it unusable for Docker mounts or VMs as those services start right after the array, if enabled. # The workaround ZFS, like LUKS, can store the decryption key in a _key file_. Normally, that's a complicated subject because it means you need some form of storage device (commonly a USB thumb drive or a network share) that is accessible at the time of mounting the dataset, but ideally not permanently attached to your unRAID machine (as it defeats the purpose of encryption) nor too easy to get your hands on, should someone steal the machine. For this reason, many people would prefer to rely on a password to decrypt the drives at startup, which does not decrypt any ZFS datasets using native encryption by default. Auto-mounting such ZFS datasets is possible, but it would require a key file, which leads us back to the "where do I store it?" problem. Well, what if we store it in a LUKS-encrypted drive that gets auto-mounted when the Array starts and then we auto-mount the dataset after that, but before anything starts? That way we get the GUI support, ensure the dataset is usable for Docker/VMs and we never have to worry about the key file because it is itself protected by our drive encrypted using our password! And that is exactly what we'll be doing here! 1. Your password guards the LUKS-encrypted drive 2. The LUKS-encrypted drive contains the ZFS decryption key 3. When starting the array, **you enter the password for decrypting the drives** 4. This makes the ZFS decryption key available 5. A script automatically mounts the encrypted dataset # Requirements This tutorial assumes you have: 1. **_At least one_** of the following: 1. An encrypted unRAID Array 2. An encrypted unRAID Pool 3. No unRAID array (ver. 7 or higher) + a spare drive 4. A dummy array (ver. 6.12 or lower) 2. A ZFS pool 3. Some experience with the command-line 4. unRAID 6.12 or later 5. ZFS Master plugin 6. User Scripts plugin # Recommendations These are not requirements, but are highly recommended: 1. Enable "Exclusive Shares" 1. Stop the array 2. Go to Settings -> Global Share Settings 3. Set "Permit exclusive shares" to "Yes" # Tutorial In this tutorial, we will: 1. Encrypt a dummy array to store the key for the ZFS dataset 2. Create an encrypted dataset: `appdata-crypt` 3. Create a script to automatically mounts it after the array starts 4. Move some parts of Immich, Nextcloud and Paperless-ngx to the new encrypted dataset ## Setup The setup I have is composed of: * A dummy array * Three ZFS Pools * `data-pool` for media storage * `cache-pool` as a cache for `data-pool` * `apps-pool` for `appdata`, `domain`, `system`, etc.   ## Encrypting an Array or Pool >_If you already have an encrypted **array** or **pool**, skip this section._ _This section assumes you either have **no array** or that you have a **dummy array**._ In order to encrypt the array, first stop it  Then select your array device  Choose any of the encrypted filesystem options. In this example, we'll be using XFS for its stability.  Observe that the array device now has a lock icon  Scroll down to where the Array Start button is. Type your password of choice and start the array.  The array is started, but the device is not yet encrypted! It must be formatted first.  Scroll down, tick the `Yes, I want to do that` box and format it.  Click OK to format the device. _THIS WILL ERASE ALL DATA ON IT!_  The device is now correctly formatted and unlocked  ## Creating the encrypted ZFS dataset First, let's find out the path for the encrypted drive we just created by running `ls /mnt`.\ In this case it's `/mnt/disk1`.  Now let's create the directory where we will store the key file for the encrypted ZFS dataset inside the encrypted disk. ```shell # Let's set the key directory in this variable so we can use it further down the line KEY_DIR="/mnt/disk1/.zfs-crypt-keys" mkdir -p "${KEY_DIR}" ``` Then create the key file and the dataset itself.\ Note that the dataset "knows" where its keyfile is and what its mountpoint is.\ You don't have to remeber that in the future. Please note you must choose *only one of the formats* for creating and keyfile/dataset: `raw` or `hex`.\ I find `hex` to be the best one because it's human-readable and you can then back it up to e.g. a password manager. ```shell # Set these so we can use them down below KEY_FILE="${KEY_DIR}/appdata-crypt.key" DATASET_LOCATION="apps-pool/appdata-crypt" # SELECT ONLY ONE OF THE FORMATS BELOW # Raw (notice different keyformat) # dd if=/dev/random of="${KEY_FILE}" bs=32 count=1 iflag=fullblock # zfs create -o encryption=on -o keyformat=raw -o keylocation="file://${KEY_FILE}" "${DATASET_LOCATION}" # Hex (notice different keyformat) od -Anone -x -N 32 -w64 /dev/random | tr -d [:blank:] > "${KEY_FILE}" zfs create -o encryption=on -o keyformat=hex -o keylocation="file://${KEY_FILE}" "${DATASET_LOCATION}" # Assuming you've used the 'hex' format # Backup your key file cat "${KEY_FILE}" # EXAMPLE OUTPUT. WRITE YOURS DOWN SOMEWHERE! # 316e548796d2307290353d94676269a629ebc9b14dcd5baf8910467b35c3f199 ``` The dataset should now be visible in `Main -> ZFS Master`!  Head over to `Shares` and set the newly crated `appdata-crypt` to be stored _only_ in the pool containing it  Now head back to `Main -> ZFS Master` and lock the dataset.  The icon changes to a locked drive.  Back to the command line, you'll noticed that `/mnt/apps-pool/appdata-crypt` exists even though the dataset is locked.\ To prevent any unintended files from being created there while the array does not start, let's make the directory immutable. ```shell # Make the mount point immutable. Not even root can modify it! chattr +i "/mnt/${DATASET_LOCATION}" ``` Back to `Main -> ZFS Master`, try unlocking the dataset.  If it prompts you for a password, do not enter anything and just click OK.  You should see this screen reporting it successfully unlocked the dataset!  Now that we know everthing is working, let's head back to the terminal and reduce the chances this key gets leaked. ```shell # Set the key location to only be readably by the 'root' user. chown -R root:root "${KEY_DIR}" chmod -R 500 "${KEY_DIR}" # Set key directory and files to be immutable so it's absolutely impossible to be modified chattr -R +i "${KEY_DIR}" ``` ## Setting up the startup script In order to have the encrypted dataset be unlocked at the same time the Array is started, head over to the `User Scripts` plugin and create a new script. Ensure its schedule is set to `At Startup of Array`, like so:  Edit the script's contents and paste the following: ```shell #!/bin/bash # Set this to 'yes' in order to receive notifications of success or failure to mount the encrypted dataset NOTIFY="no" # Discover all encrypted datasets we have, skipping over the prompt key locations declare -a encrypted_datasets=($(zfs list -r -t filesystem -H -o name,keylocation | grep -F '/' | grep -vE '\b(none|prompt)\b' | cut -d$'\t' -f1)) curr_date() { date +'%Y-%m-%d %H:%M:%S' } unraid_notify() { if [[ "${NOTIFY}" != "yes" ]]; then return fi local message="$1" local flag="$2" if [[ "$flag" == "success" ]]; then severity="normal" else severity="warning" fi /usr/local/emhttp/webGui/scripts/notify -s "Notification" -d "$message" -i "$severity" } echo "[$(curr_date)] Detected encrypted datasets: ${encrypted_datasets[@]}" for d in "${encrypted_datasets[@]}"; do echo "[$(curr_date)] Loading key for: ${d}" out=$(zfs load-key -r "${d}" 2>&1) if (($? == 0)); then echo "[$(curr_date)] Loaded key for ${d}" unraid_notify "Loaded key for ZFS dataset ${d}" "success" else echo "[$(curr_date)] Failed to load key for ${d}: ${out}" unraid_notify "Failed to load key for ZFS dataset ${d}: ${out}" "failure" exit 1 fi mountpoint="$(zfs list -H -o mountpoint ${d})" # Ensure the regular location is immutable so nothing can write to it unless we're mounted! out="$(mkdir -p "${mountpoint}" 2>&1 && chattr +i "${mountpoint}" 2>&1)" if (($? > 0)); then echo "[$(curr_date)] Failed to create directory for ${d}: ${out}" unraid_notify "Failed to create directory for ${d}: ${out}" "failure" exit 1 fi echo "[$(curr_date)] Mounting ${d} at ${mountpoint}" out="$(zfs mount "${d}" 2>&1)" if (($? == 0)); then echo "[$(curr_date)] Mounted ${d}" unraid_notify "Mounted ZFS dataset ${d}" "success" else echo "[$(curr_date)] Failed to mount ${d}: ${out}" unraid_notify "Failed to mount ZFS dataset ${d}: ${out}" "failure" exit 1 fi done ``` ## Testing the script Go to `Main -> ZFS Master` and lock the dataset again  Check that it displays the locked icon  Head back to `User Scripts` and run the script  You should see the output of the script, which should report success.  The script has an option to enable notifications. If set to `yes`, you'll also receive the results like so:  As a final test for the script, try stopping and starting the array again. It should mount the encrypted dataset right away, whe the Array starts. ## Done! That's it! From now on, you have an encrypted ZFS dataset in your unRAID server!\ Whenever the array stops, the dataset is automatically unmounted.\ When it starts, you'll be asked to fill in the password for the encryted array, just like always.\ The script will then ensure the dataset is unlocked and mounted before Docker and VMs start up. # Moving confidential parts of an application [WIP]  1. Stop docker compose. ```shell # 1. Identify what to transfer # In this case, we want to transfer these while maintaining the rest unencrypted # /mnt/user/appdata/immich/photos to /mnt/user/appdata-crypt/immich/photos # /mnt/user/appdata/nextcloud/data to /mnt/user/appdata-crypt/nextcloud/data # /mnt/user/appdata/paperless/server/media to /mnt/user/appdata-crypt/paperless/server/media # /mnt/user/appdata/paperless/server/export to /mnt/user/appdata-crypt/paperless/server/export # /mnt/user/appdata/paperless/server/consume to /mnt/user/appdata-crypt/paperless/server/consume # Some applications require extra attention # nextcloud: mount some non-encrypted directory (e.g. /mnt/user/appdata/nextcloud/logs) as log directory and add this to php config # 'log_type' => 'file', # 'logfile' => '/var/log/nextcloud/nextcloud.log', # 'log_type_audit' => 'file', # 'logfile_audit' => '/var/log/nextcloud/audit.log', OLD_DATASET="apps-pool/appdata" NEW_DATASET="${DATASET_LOCATION}" declare -a PATHS_TO_TRANSFER=( "immich/photos" "nextcloud/data" "paperless/server/media" "paperless/server/export" "paperless/server/consume" ) for p in "${PATHS_TO_TRANSFER[@]}"; do rsync -aRv "/mnt/${OLD_DATASET}/./${p}" "/mnt/${NEW_DATASET}/" done # or, to move it instead of copying (BUT BE CAREFUL): # for p in "${PATHS_TO_TRANSFER[@]}"; do # rsync -aRv --remove-source-files "/mnt/${OLD_DATASET}/./${p}" "/mnt/${NEW_DATASET}/" && rm -rf "/mnt/${OLD_DATASET}/${p}" # done # change directories in docker templates and compose, start the apps again, test tha they're ok # remove each one of the old folders for p in "${PATHS_TO_TRANSFER[@]}"; do rm -rf "/mnt/${OLD_DATASET}/${p}" done ``` Make adjustments to your `docker-compose.yaml` files.     

One big thing about the approach I used there is that the ZFS key is never exposed unencrypted.