opixo

Hey, take a look here, might be interesting for you: (2198) Unraid 7.3 Beta: Set Up a New Server with Internal Boot - YouTube and this: Unraid 7.3 Beta: Convert Your Server to Internal Boot

Well, with a bit of help from Opus 4.7 here are a few reasonable things to check out, my main issue was that the USB drive (a very old one actually but very reliable) was partitioned in a way that Unraid 7.3.0 didn't like (strangely since I had same USB key since 2017 and never had any issues) however apparently yours might be related so something different. Also, my setup is a little bit more complex, I am using a Q-Nap JBOD and have had: 1. 8-year-old superfloppy USB — this is the headline. Most newer flash sticks were already MBR-partitioned, which is why this regression isn't hitting everybody. 2. Modded go script with custom boot-time stuff 3. Custom kernel cmdline (pcie_aspm=off pcie_port_pm=off for ASM1164) 4. /boot/extra/ with pulseway (manual package install from 2018) 5. All-ZFS topology (no Unraid btrfs cache pool, no parity disk for media) Couple of things worth checking — your symptom is actually a bit different from mine. Mine failed after the kernel was running (emhttpd rejecting the flash); yours is freezing visibly at loading /bzroot...ok, which is much earlier. That said, you might be hitting a display-only issue while the system is actually booting fine. The 7.3.0 kernel jumped from 6.12.x to 6.18.x and the new i915/DRM stack can blank the console on some Intel iGPU systems after bzroot even when boot proceeds normally. Please try these in order: 1. Power on, wait a full 5 minutes, then from another machine on the same network try: - ping <tower-ip> - Open http://<tower-ip> in a browser - ssh root@<tower-ip> If any of those respond, your system is booting fine and it's just a display issue — see step 3. 2. If nothing responds, take the flash to your Windows laptop and verify the rollback actually took: - On the USB root, check that bzimage, bzroot, bzroot-gui, bzfirmware, bzmodules have dates matching your old 7.2.3 (not 7.3.0). Easy to mistakenly copy the wrong direction. - Open bzimage.sha256 in Notepad and confirm the hash matches bzimage (you can compute it in PowerShell with Get-FileHash bzimage -Algorithm SHA256). Any mismatch and the kernel will refuse / hang. - Open syslinux/syslinux.cfg — the kernel /bzimage and initrd /bzroot,/bzfirmware,/bzmodules lines should not reference anything in /previous/. 3. If you can SSH in but the console is blank, you're hitting the iGPU/framebuffer issue. From SSH, edit /boot/syslinux/syslinux.cfg and add nomodeset video=vesafb:off i915.modeset=0 to the append line of the Unraid OS label. Reboot — console should come back. This is purely a display workaround, doesn't affect anything else. 4. Also check what my thread is really about (separate from your boot freeze): in Windows, open Disk Management, find the USB, and look at the partition layout. If it shows "Removable - no partition table" with a single FAT32 volume spanning the whole disk, you have the superfloppy layout my post addresses — and even if you get past your current boot freeze, you'll hit ENOFLASH4 next. If it shows a proper Partition 1: FAT32, that part of the layout is already fine and the issue is purely kernel/display. Report back what ping/ssh shows and what the partition layout looks like, happy to keep digging.

Please check here, is the USB partitioning basically [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs - General Support - Unraid

Thanks Jorge, yes correct

I did but I was more comfortable to do it manually, i have a modded go file, just pers preference but I think main take away is 7.3.0 is picky about USB partitioning, mine worked since 2017 thru all the upgrades up to this point

Hey, please look at this might apply to you, worth a shot: [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs - General Support - Unraid

Hi guys, had same problem, posting here as well might help other users, documented in the following post: [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs - General Support - Unraid

Please see my post here, might be related, 7.3.0 is picky about how the USB drive is formatted: [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs - General Support - Unraid I also had this issue with NGINX error after upgrading from 7.2.6 to 7.3.0 (worked perfectly fine since 2017 up to 2 days ago when upgraded to 7.3.0)

Please see my post here, might be related, 7.3.0 is picky about how the USB drive is formatted: [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs - General Support - Unraid I also had this issue with NGINX error after upgrading from 7.2.6 to 7.3.0 (worked perfectly fine since 2017 up to 2 days ago when upgraded to 7.3.0)

Please see my post here, might be related, 7.3.0 is picky about how the USB drive is formatted: [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs - General Support - Unraid I also had this issue with NGINX error after upgrading from 7.2.6 to 7.3.0 (worked perfectly fine since 2017 up to 2 days ago when upgraded to 7.3.0)

# [DIAGNOSED + WORKAROUND] 7.3.0: emhttpd ENOFLASH4 — flash rejected, GUID = (null) on legacy whole-device FAT (superfloppy) USBs **TL;DR** — If you upgraded 7.2.6 → 7.3.0 and your boot USB is suddenly rejected (`emhttpd: import flash device: (sda) ... Error: (/dev/sda) not known`, license shows `(null)`, GUI returns HTTP 500, array won't start), the cause is almost certainly that your USB has the legacy "superfloppy" layout — a FAT filesystem written directly to `/dev/sda` with no partition table — which 7.2.x and earlier accepted but **7.3.0 rejects**. The fix is to re-flash the same USB with a proper MBR + single FAT32 partition layout using the current Unraid USB Creator, then restore your `/config` from backup. **Same USB, same license, same GUID — it just needs the partition table.** Full procedure below. If this saves anyone the panic I had, mission accomplished. --- ## Symptoms (what 7.3.0 looks like when it rejects your flash) After the standard in-GUI Update OS to 7.3.0 stable and reboot: - Kernel boots, USB enumerates fine (`lsusb`, `dmesg` happy, `sda` present) - `emhttpd` starts, fails to classify the flash device: ``` emhttpd: import flash device: (sda) emhttpd: Error: shcmd ... mount /boot ... not known emhttpd: shcmd ... ENOFLASH4 ``` - `/var/local/emhttp/var.ini` is created but every classifier-derived value is `(null)`: ``` flashGUID="(null)" flashVendor="(null)" flashProduct="(null)" regGUID="(null)" regTy="(null)" ``` - WebUI loads the static login page (HTTP 302), but every authenticated request returns **HTTP 500** because `emhttpd` never created `/var/run/php-fpm.sock` - Array never starts, no pools imported, Docker daemon never started, no containers - SSH still works (root login is independent of `emhttpd`), which is your lifeline The `Plus.key`, `super.dat`, and the rest of `/boot/config/` are **not corrupted** — only the boot images (`bzimage`, `bzroot`, `bzfirmware`, `bzmodules`, `memtest`) were swapped by the upgrade. ## Root cause 7.3.0's flash classifier is stricter and **requires a proper MBR with a single primary FAT partition**. It rejects the legacy whole-device FAT (a.k.a. "superfloppy") layout that older Unraid releases were happy with. You can see the difference yourself. Old, rejected layout (this is what triggers the failure): ``` $ blkid /dev/sda /dev/sda: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="XXXX-XXXX" BLOCK_SIZE="512" TYPE="vfat" # ↑ filesystem is on the raw disk, no PTUUID/PTTYPE, no /dev/sda1 $ fdisk -l /dev/sda Disk /dev/sda: 29.11 GiB, 31260704768 bytes, 61056064 sectors Disklabel type: dos # the partition entries fdisk shows are garbage — random bytes in the MBR area # being mis-interpreted; sector counts overflow the physical disk size ``` New, accepted layout (what every modern Unraid USB Creator produces): ``` $ blkid /dev/sda /dev/sda1 /dev/sda: PTUUID="xxxxxxxx" PTTYPE="dos" /dev/sda1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="XXXX-XXXX" TYPE="vfat" PARTUUID="..." # ↑ proper partition table on /dev/sda, FAT filesystem on /dev/sda1 $ fdisk -l /dev/sda Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 61054975 61052928 29.1G c W95 FAT32 (LBA) ``` Many older USBs prepared with the original `make_bootable` (especially SanDisk Cruzer Force and similar long-serving sticks) shipped/were-flashed in superfloppy mode. They booted fine for years on Unraid 6 and 7.0–7.2 because the classifier accepted both layouts. 7.3.0 no longer does. ## Proof it's purely the partition layout (not the kernel, not the license, not the USB hardware) Same physical SanDisk Cruzer Force, same `/boot/config/` (restored byte-for-byte from backup), same `Plus.key` (md5 unchanged), same machine, same kernel cmdline. The **only** thing that changed between "7.3.0 rejects" and "7.3.0 accepts" was the partition layout (superfloppy → MBR + sda1). After re-flashing the layout, 7.3.0 booted, `emhttpd` immediately classified the USB, computed the correct flashGUID, license validated as Plus, array started, all 6 ZFS pools imported, all 23 containers came up. Important: the `flashGUID` is derived from the USB controller's vendor/product/serial descriptors, which are unchanged by reformatting. **Your existing license stays bound to the same USB.** No Replace Key, no re-registration, no involvement from sales/support. ## The fix (procedure I followed — 30 minutes downtime, no key transfer required) > ⚠️ Read the whole thing before starting. **Take a full byte-level backup of the USB first** — if anything goes wrong, you can `dd` it back and you're exactly where you started. ### What you need - A working Linux machine to take a backup over SSH (or just plug the USB in directly) - A Windows or macOS machine to run the official Unraid USB Creator - Your most recent `/boot/config/` backup (the in-GUI **Tools → Backup/Flash** zip works, or a tarball you take manually). If you don't have one, take it now from 7.2.6 before doing anything else. ### Step 1 — Roll back to 7.2.6 (so you can use the GUI to take a clean backup) If you're stuck at the failing 7.3.0 boot but `/boot/previous/` exists (Unraid keeps the prior version's boot files there during an upgrade), copy these from `/boot/previous/` over `/boot/` via SSH and reboot: ``` bzimage bzimage.sha256 bzroot bzroot.sha256 bzroot-gui bzroot-gui.sha256 bzfirmware bzfirmware.sha256 bzmodules bzmodules.sha256 memtest memtest.sha256 ``` Don't touch `/boot/config/`, `/boot/syslinux/syslinux.cfg`, `Plus.key`, `super.dat`, or anything else. Reboot — you should be back on 7.2.6 with everything working. ### Step 2 — Take backups from working 7.2.6 From the GUI: **Tools → Backup/Flash → Backup**. Save the resulting zip somewhere off the USB. If you want a belt-and-braces byte-level backup of the USB itself (recommended), pull it from another machine over SSH: ```bash # from a Linux box, USB in the Tower as /dev/sda — DO NOT run this on the Tower itself ssh root@tower 'dd if=/dev/sda bs=4M status=progress 2>/tmp/dd.log | gzip -1 -c' \ > sandisk-backup.img.gz ``` (The `2>/tmp/dd.log` matters — if you let `dd`'s stderr into the pipe, gzip will ingest the progress messages as data and your backup will be silently corrupted. Don't ask me how I know.) ### Step 3 — Pull the USB and reformat it with the official Creator 1. Stop the array, shut down the Tower cleanly (Main → Stop, then Tools → Shutdown) 2. Pull the USB and put it into your Windows/macOS machine 3. Download the **Unraid USB Creator** from `unraid.net/download` (current release — make sure it's the new one, not the old `Unraid.USB.Creator` / `make_bootable.bat` from 2017) 4. Choose: **OS Version: 7.3.0**, **Server Name:** (your existing name), **Network Mode:** DHCP (or whatever you had), keep "format the USB device" enabled, write 5. After it finishes, on Windows it'll prompt you to run `make_bootable.bat` — **do that** (it installs the syslinux MBR for legacy BIOS boot) 6. The drive letter will change (Windows treats the new partition layout as a new volume). That's expected and harmless. ### Step 4 — Restore your `/config` and customizations The Creator wrote a fresh, default `/config/` directory (with a generic `network.cfg`, `go`, `ident.cfg`, etc). You need to overwrite it with your real config from the backup. From your `/boot/config/` backup: - **Delete** the auto-generated `\config\` on the new USB - **Copy** your backup's `config/` directory onto the USB so it lives at `\config\` (with all subfolders: `plugins/`, `pools/`, `shares/`, `ssh/`, `modprobe.d/`, etc — and the files `Plus.key`, `super.dat`, `network.cfg`, `ident.cfg`, `go`, `disk.cfg`, etc) - If you had anything in `/boot/extra/` (extra packages installed at boot — e.g. Pulseway), restore that to `\extra\` too If you had a customized `syslinux.cfg` (custom kernel args, default GUI mode, etc): - Take the **new 7.3.0 `syslinux/syslinux.cfg`** the Creator wrote as your base (in case Limetech changed something) - Add your custom kernel args back to the `Unraid OS` and `Unraid OS GUI Mode` `append` lines (e.g. `pcie_aspm=off pcie_port_pm=off`) - If you had `menu default` on a non-default label, re-add it - Make the same edits to **both** `\syslinux\syslinux.cfg` (BIOS) and `\EFI\boot\syslinux.cfg` (UEFI) - Do **not** copy your old `syslinux/*.c32` binaries over the new ones — keep the new 7.3.0 ones Verify before unplugging: - `\config\Plus.key` exists and md5 matches your backup - `\config\super.dat` exists - `\config\go` is your customized one (not the 1-line default) - `\config\plugins\` is populated with your plugins - `\config\pools\` (if you had one) is populated Safely eject the USB. ### Step 5 — Boot Put the USB back in the Tower, power on. Within ~3 minutes you should see: - Console gets to a login prompt - WebUI is up at `https://<tower-ip>` (HTTP 302 to the login page) - License shows correctly (Tools → Registration) - Array starts (it'll likely run a parity check because the previous shutdown wasn't graceful — this is normal, let it complete) - Containers come back Verify via SSH (handy one-liner): ``` cat /etc/unraid-version grep -E '^(flashGUID|flashVendor|flashProduct|regGUID|regTy|mdState|fsState)=' \ /var/local/emhttp/var.ini fdisk -l /dev/sda | head -15 ``` You should see `version="7.3.0"`, a non-`(null)` `flashGUID`, `regTy="Plus"` (or Pro/Trial), and `Disklabel type: dos` with a single `/dev/sda1` of type `W95 FAT32 (LBA)`. ## What I checked / ruled out before landing on the partition-layout theory - **Not the kernel cmdline** — `syslinux.cfg` was unchanged - **Not internal-boot vs flash-boot** — never opted into Internal Boot or TPM - **Not the license file** — `Plus.key` md5 unchanged before/after, works fine on 7.2.6 post-rollback and post-fix - **Not the `go` script** — `go` runs after `emhttpd`, so it can't cause `ENOFLASH4` - **Not a reverse-proxy issue** — the 500s come from `php-fpm.sock` not existing, which is downstream of `emhttpd` failing - **Not a hardware fault** — USB enumerates fine, reads/writes normally, healthy SMART - **Not a USB descriptor / VID:PID issue** — same descriptors before and after, same flashGUID computed both before the upgrade (on 7.2.6) and after the fix (on 7.3.0) ## Asks for Limetech (if anyone from staff reads this) The fact that the upgrade silently bricks the boot of users with the older USB layout is rough. Two suggestions, either would be a meaningful improvement: 1. **Pre-flight check in the in-GUI Update OS flow:** detect superfloppy layout before applying the upgrade, warn the user, and link to a "how to migrate to MBR layout" doc. The upgrade should not proceed without acknowledgement. 2. **Loosen the 7.3.x classifier** to accept the legacy whole-device FAT layout the way 7.2.x did, even if only as a deprecated fallback for one more minor release cycle. If neither is feasible, at minimum a release-note entry and a pinned thread documenting the symptom + workaround would save people a lot of time. ## Diagnostics available I have the original failing 7.3.0 boot files preserved, full syslogs from both the failing 7.3.0 boot and the working post-fix 7.3.0 boot, and a byte-level backup of the USB in its original superfloppy layout. Happy to share any of it with staff for analysis. ## System (for reference) - **Hardware:** Dell OptiPlex 7050 MFF, Intel i7-7700, 32 GB DDR4 - **Storage controllers:** onboard Intel + ASM1164 PCIe HBA - **Boot USB:** SanDisk Cruzer Force 32 GB, in service ~8 years - **License:** Plus / Lifetime (registered 2017) - **Working version before:** 7.2.6 (kernel 6.12.x) - **Failing version:** 7.3.0 stable (kernel 6.18.x) — *until* USB was re-partitioned, after which it works perfectly Hope this helps somebody. 🍻

I know its an old thread but it was bugging me for sol long, after 2 full days with AI on the task this is what it came up with, the below is written by the AI so take it with a pinch of salt but it works for me apparently. ZFS Pool HDD Spindown Fix — Unraid 7.xPlatform: Unraid 7.2.4 (ZFS pools on HDDs) | Problem: ZFS pool HDDs wake from standby within 30s of hdparm -y, despite zero user I/O ✅All 4 root causes identified and fixed — drives now stay spun down indefinitely. The ProblemAfter setting up a ZFS mirror pool on two HDDs in Unraid 7.x, the drives refused to stay spun down. Manual spindown via hdparm -y /dev/sdX would put them in standby, but they'd wake within 15–30 seconds every time. Array disks (XFS) had no issues spinning down — only the ZFS pool was affected. Root Causes (4 Found)1ZFS Master Plugin — 30-Second Polling LoopThe ZFS Master plugin (zfs.master) runs an nchan poller in an infinite loop. The default refresh_interval is 30 seconds — it calls zpool status, zfs list, and related commands which access the pool's vdevs, waking any spun-down drives. 🔍 Diagnosis: After killing the ZFS Master poller process, drives still woke (proving additional culprits existed), but this was confirmed as one source. ✅ Fix: Set refresh to manual-only: # /boot/config/plugins/zfs.master/zfs.master.cfg [general] refresh_interval=0The plugin stays installed and fully functional — it just only polls when you manually open the ZFS Master page in the WebUI. 2SMART Polling (poll_attributes) — 30-Second DefaultUnraid's emhttpd polls SMART attributes on all disks at the interval set by poll_attributes in /boot/config/disk.cfg. The default is 30 seconds. SMART queries send ATA commands that wake spun-down drives. Array disks appear to have standby-aware skip logic, but ZFS pool disks do not benefit from this — they get polled regardless of power state. 🔍 Diagnosis: /proc/diskstats monitoring showed drives waking with zero write sectors — pointed to ATA-level commands (SMART) rather than filesystem I/O. The wake interval perfectly matched the 30-second polling. ✅ Fix: Change from 30 to 7200 (2 hours): # /boot/config/disk.cfg poll_attributes="7200"Requires reboot to take effect. Drives will wake once every 2 hours for a SMART check, then spin back down after the configured idle timeout (e.g., 15 minutes). A value of 0 disables SMART polling entirely (not recommended). 3ZFS Transaction Group Timeout (txg_timeout) — 5-Second DefaultZFS flushes transaction groups every zfs_txg_timeout seconds (default: 5), even when there's zero pending data. This causes periodic metadata writes that keep drives active. ✅ Fix: Set to 300 seconds (5 minutes) via /boot/config/go so it persists across reboots: # Add to /boot/config/go (before the last line) echo 300 > /sys/module/zfs/parameters/zfs_txg_timeoutThis is safe for a backup/archive pool. For pools with active writes, a lower value (30–60) may be more appropriate. 4Cache Dirs Plugin (dynamix.cache.dirs) — PeriodicfindThe Dynamix Cache Dirs plugin runs periodic find commands to warm directory listings in RAM. If any share on your ZFS pool is in the plugin's include list, it will periodically walk the entire directory tree, generating heavy read I/O on the pool disks. In our case, a backup share containing thousands of nested iCloud directories was in the include list — each scan read extensively from the ZFS pool drives. 🔍 Diagnosis: Used kernel ftrace to catch the exact process: echo 1 > /sys/kernel/debug/tracing/events/block/enable cat /sys/kernel/debug/tracing/trace_pipe | grep "sd[gh]" # Caught: find-91408 reading blocks from sdg✅ Fix: Remove ZFS pool shares from the cache_dirs include list: # /boot/config/plugins/dynamix.cache.dirs/dynamix.cache.dirs.cfg# Remove your ZFS pool share names from both 'include=' and 'options=' linesThen kill and restart the cache_dirs process: pkill -f "scripts/cache_dirs" # It will auto-restart, or rebootAdditional Fixes (May Also Help)Disable autotrim for HDD pools: zpool set autotrim=off <pool> — TRIM is useless for HDDs Set atime=off: zfs set atime=off <pool> — prevents access time updates on reads Check for btrfs cron jobs on ZFS pools: Unraid's Scheduler may generate btrfs balance/scrub cron jobs for ZFS pools (wrong filesystem type). Check /boot/config/plugins/dynamix/balance_*.cron and scrub_*.cron — delete any that reference your ZFS pool. Also check /etc/cron.d/root for active entries. How to Diagnose Spindown IssuesStep 1: Baseline Testhdparm -y /dev/sdX # Spin down sleep 30 hdparm -C /dev/sdX # Check: "standby" = good, "active/idle" = something woke itStep 2: Monitor Block I/O# Watch /proc/diskstats for your drives (e.g., sdg) while true; do grep "sdg " /proc/diskstats | awk '{print strftime("%H:%M:%S"), "reads:"$4, "writes:"$8, "read_KB:"$6/2, "write_KB:"$10/2}' sleep 15 doneIf drives wake with zero write sectors, the cause is likely ATA commands (SMART polling, power management queries), not filesystem I/O. Step 3: Pool Export Test (Definitive)zpool export <pool> # Detach pool from ZFS hdparm -y /dev/sdX # Spin down sleep 120 # Wait 2 minutes hdparm -C /dev/sdX # If standby: ZFS/related processes are the cause zpool import <pool> # Re-import when doneIf drives stay in standby with the pool exported, the wake source is definitely something interacting with ZFS — not hardware or BIOS. Step 4: Kernel Ftrace (Catch the Exact Process)# Enable block-level tracing (no extra tools needed on Unraid) echo 1 > /sys/kernel/debug/tracing/events/block/enable cat /sys/kernel/debug/tracing/trace_pipe | grep "sdX" # Output shows: processname-PID reading/writing blocks# Press Ctrl+C when done, then: echo 0 > /sys/kernel/debug/tracing/events/block/enableStep 5: Check zpool I/Ozpool iostat <pool> 5 # 5-second intervals# If all zeros but drives still wake → ATA-level commands, not I/OBIOS Considerations⚠️ Note: If you've disabled CPU C-States, Deep Sleep, or ASPM (e.g., for HBA/JBOD card compatibility), be aware these keep SATA controllers fully active at all times. While they don't directly wake drives, they remove the power-saving buffer that might otherwise mask brief wake events. Summary# Culprit Default Wake Interval Fix 1 ZFS Master plugin 30s refresh Every 30s refresh_interval=0 2 SMART polling 30s poll Every 30s poll_attributes="7200" 3 ZFS txg_timeout 5s flush Every 5s Set to 300 via /boot/config/go 4 cache_dirs plugin Varies Periodic Remove ZFS pool shares from include list Available Diagnostic Tools on Unraid❌ Don't Work on Unraidblktrace, strace, auditctl, fatrace, iotop ✅ Do Work on Unraidhdparm -C, /proc/diskstats, zpool iostat, kernel ftrace, inotifywait Tested on Unraid 7.2.4 with ZFS mirror pool (2×2TB HDDs) on a QNAP QXP-800eS JBOD card. After applying all 4 fixes, drives confirmed staying in standby for 5+ minutes (previously woke within 15–30 seconds every time).

This is a very old post, but maybe will help someone in same situation as me. Today, i finally had some time to troubleshoot it and togheder with my friend the AI who is also writing the solution below we manage to fix it. Here is what we did: 🔧 [RESOLVED] Dell OptiPlex 7050 + QNAP TL-D800S JBOD — Complete Fix GuideHey everyone, follow-up to my original post from August 2024. After a LOT more troubleshooting (and almost losing my mind), I've fully resolved the cold boot drive failure issue with the QNAP QXP-800eS-A1164 PCIe card. First — thank you to @Vr2Io and @Frank1940 for the suggestions in the original thread. You were both pointing in the right direction: @Vr2Io: The PCIe AER Fatal errors WERE the key symptom. Couldn't force PCIe 2.0 in the Dell BIOS, but the fix was disabling power management features that were destabilizing the PCIe link. @Frank1940: Disabling array autostart + waiting before starting was absolutely the right instinct. The go script solution below is essentially an automated version of that, plus NCQ disable. Posting this in detail because I found very few resources for this specific combination and hopefully it saves someone else days of headaches. 🖥️ HardwareServer Dell OptiPlex 7050 Tower (i7-7700, 32GB DDR4, BIOS v1.27.0) JBOD QNAP TL-D800S (8-bay, SFF-8088 connection) PCIe Card QNAP QXP-800eS-A1164 (ASM2812 switch + 2x ASM1164 AHCI controllers) OS Unraid 7.2.3, kernel 6.12.54-Unraid Array Drives 4x Seagate ST4000NE001 4TB (1 parity + 3 data) Pool Drives 2x Hitachi 2TB (ZFS mirror) + 2x HGST 10TB (ZFS stripe) 💥 The ProblemEvery cold boot, array drives on ASM1164 controller #1 would fail. Two types of errors depending on what was going wrong: Type 1 — PCIe AER Fatal errors (from original post, Aug 2024): pcieport 0000:00:1d.0: AER: Multiple Uncorrected (Fatal) error received from 0000:03:00.0 pcieport 0000:04:02.0: device [1b21:2812] error status/mask=00100000/04400000 pcieport 0000:04:02.0: [20] UnsupReq (First) ahci 0000:05:00.0: AER: can't recover (no error_detected callback) pcieport 0000:00:1d.0: AER: device recovery failedType 2 — SATA/AHCI errors (after PCIe stabilized with BIOS fixes): ata5.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x6 frozen ata5.00: failed command: WRITE FPDMA QUEUED ata5: hard resetting link ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)Both types would crash ALL 4 ports on the same ASM1164 chip simultaneously, cascade into hard reset loops, and disable all drives. The array would hang during start and require SysRq reboot (echo b > /proc/sysrq-trigger) because ZFS import processes get stuck in uninterruptible D-state. Warm reboots sometimes worked. Safe mode always worked. 100% reproducible on cold boot under any I/O load. 🔍 Root Cause — THREE Issues Stacked Together1. ASM1164 NCQ BugThe ASM1164 has a known bug with Native Command Queuing (NCQ). Commands sent as WRITE FPDMA QUEUED (NCQ) would timeout and crash the controller. This is documented across multiple platforms — Raspberry Pi Linux, openSUSE forums, various NAS communities. 2. Dell BIOS Power ManagementThe OptiPlex 7050's default BIOS settings include aggressive power saving that destabilizes PCIe devices: Deep Sleep Control: enabled by default (cuts power to PCIe in S4/S5) C-States: enabled (CPU power states affect PCIe link stability) Block Sleep: disabled (allows S3 sleep which kills PCIe) AC Recovery: Off (no auto-restart after power loss) 3. PCIe Remove/Rescan Hack Breaking BIOS InitializationThis was the hardest to find. I had a workaround in /boot/config/go that did: echo 1 > /sys/bus/pci/devices/0000:04:00.0/remove sleep 5 echo 1 > /sys/bus/pci/rescanThis was supposed to "reset" the card. What it ACTUALLY did was destroy the proper firmware initialization that BIOS performed during POST. The Linux PCIe rescan does NOT perform the full firmware handshake — it just re-enumerates the device. The controller appeared to work for light I/O but would crash under sustained mixed read+write workloads (like parity rebuild). The smoking gun was in dmesg timestamps: all drives were detected at 3–9 seconds (BIOS init), then the go script removed the card at ~35s and re-scanned at ~147s, causing drives to be detected a SECOND time with degraded initialization. ✅ The Complete FixAll three issues needed to be addressed: Fix 1: JBOD Firmware UpdateUpdated TL-D800S firmware: 1.4.1 → 1.5.0 Download from QNAP website. I had to temporarily move the QXP card to a Windows PC to run the QNAP firmware update utility. The JBOD firmware update is separate from the QXP card firmware. Fix 2: Dell BIOS Settings (F2 at boot)Section Setting Value Priority Power Management Deep Sleep Control DISABLED ⚠️ Critical Power Management AC Recovery Last Power State Recommended Power Management Block Sleep ENABLED Recommended Performance Intel SpeedStep DISABLED Recommended Performance C-States Control DISABLED ⚠️ Critical POST Behavior Fastboot Thorough Recommended These settings prevent the motherboard from doing anything that could affect PCIe link stability during boot or runtime. Fix 3: Go Script — NCQ Disable Only (NO PCIe hack!)File: /boot/config/go #!/bin/bash # === ASM1164 / QXP-800eS Workaround v4 === # BIOS fixes (Deep Sleep off, C-States off, ASPM off) + JBOD firmware 1.5.0 # now allow proper cold boot detection. PCIe remove/rescan REMOVED — it was # breaking the BIOS-initialized controller state. QXP_BRIDGE="0000:04:00.0" ASM_VENDOR="1b21:1164" if lspci -d $ASM_VENDOR &>/dev/null; then logger "go: ASM1164 QXP card detected" # Phase 1: Let drives fully spin up and settle (BIOS already detected them) logger "go: Phase 1 — 30s drive settle..." sleep 30 # Phase 2: Disable NCQ for all drives on the QXP card # ASM1164 has known issues with NCQ (WRITE FPDMA QUEUED timeouts) logger "go: Phase 2 — Disabling NCQ for ASM1164 drives..." for dev in /sys/block/sd*; do devpath=$(readlink -f "$dev") if echo "$devpath" | grep -q "$QXP_BRIDGE"; then echo 1 > "$dev/device/queue_depth" 2>/dev/null logger "go: NCQ disabled for $(basename $dev) (queue_depth=1)" fi done logger "go: ASM1164 workaround complete — starting emhttp" else logger "go: No ASM1164 detected — normal boot" fi # Start the Management Utility /usr/local/sbin/emhttpKey points about the go script: 🚫 DO NOT do PCIe remove/rescan — it breaks the BIOS initialization queue_depth=1 forces commands to DMA mode instead of FPDMA (NCQ). Verify with: dmesg | grep "ata[5-8]" — commands should show READ DMA / WRITE DMA EXT, NOT WRITE FPDMA QUEUED 30s settle time gives drives time to fully spin up after BIOS detection The script auto-detects drives behind the QXP bridge, so it won't affect onboard SATA drives Kernel Boot ParametersIn syslinux config (append line): pcie_aspm=off pcie_port_pm=offDisables PCIe Active State Power Management at the kernel level. Belt and suspenders with the BIOS ASPM disable. 🎉 After the FixDid a New Config (Tools → New Config → Preserve All → Apply) to get a clean super.dat, then started the array normally (not maintenance mode, parity NOT already valid). Results: ✅ All 8 drives detected cleanly in 3–9 seconds during boot ✅ Zero ATA errors in dmesg ✅ Parity rebuild running at 224 MB/s with zero errors ✅ All ZFS pools mounted with zero checksum errors ✅ SMB shares working, Docker containers running ✅ Array fully stable under sustained I/O for the first time ❌ What Didn't Work (save yourself the time)SATA link power management (max_performance) — no effect PCIe device remove + rescan in go script — MADE IT WORSE Longer delays alone (90s, 125s) — didn't help without BIOS fixes Starting in maintenance mode — controller still crashed after ~30s Direct dd write tests to diagnose — just kills drives when controller crashes 🔎 How to Diagnose If You Have This IssueCheck if errors say WRITE FPDMA QUEUED — that's the NCQ bug Check dmesg for duplicate drive detection (drives appearing twice = PCIe rescan issue) Check if ALL ports on one ASM1164 die simultaneously — that's the controller, not individual drives Test drives individually with dd reads when array is stopped — if they read fine, the drives are healthy Look for hostbyte=0x04 (DID_ERROR) — that means host adapter error, not drive error ZFS checksum verification: zpool status shows 0 errors = your data is intact regardless of md errors 🛠️ Useful CommandsCheck NCQ is disabled (should show queue_depth=1): for d in /sys/block/sd*; do echo "$(basename $d): $(cat $d/device/queue_depth 2>/dev/null)" doneCheck for ATA errors: dmesg | grep -iE "ata[0-9]+.*(error|fail|reset|frozen)" | tail -20Check parity rebuild progress: mdcmd status | grep -E "mdResyncPos|mdResyncSize|rdevNumErrors"Emergency reboot when array hangs (D-state processes block normal reboot): echo b > /proc/sysrq-trigger💡 Hardware NotesThe ASM1164 is a known problematic controller. If you're building new, avoid it and get an LSI 9211-8i or similar HBA instead. But if you already have the QNAP QXP-800eS + TL-D800S combo, the fix above makes it work reliably. The Dell OptiPlex 7050 is a solid Unraid server otherwise — the i7-7700 handles transcoding, VMs, and Docker well. Just fix the BIOS power management settings. Hope this helps someone. Happy to answer questions! 🤙

Anyways, thank you so much. will write after the test on the other computer.

I will try tomorrow to put the PCIe card in my main PC and try to boot UnRaid there, see if this happens there as well, might be that the Optiplex has an issue with PCIe. I recently bought it from eBay...