August 11, 20232 yr Hello, I'm getting a ZFS pool suspended error. I recently upgraded to 6.12.3 and followed the onscreen advice to convert my cache drive (an NVMe drive) from XFS to a ZFS pool. I had heard the praises of zfs and felt why not try it. However, now I'm not sure if this drive is bad, going bad, or if there is a problem with Unraid. This particular drive is about four years old. I repurposed it from a laptop I purchased in 2019. About eighteen months ago, I put it in my Unraid server to serve as the cache drive. Looking through my past notifications, the drive temperature rarely exceeded 48 C, and only went over 50 C on one occasion. I did not, and still do not, have a heat sink on this drive. In the past, this drive was installed to an m.2 slot where it was located between the first and second PCIe slots on my motherboard. I had a GTX960 installed above it. Perhaps the fans on that device moved enough airflow to keep it cool. After upgrading to 6.12.3, I changed my configuration around. I added more dedicated hardware for GPU and USB pass through to support more VMs. I relocated this drive from its m.2 slot to the first 16x PCIe slot with a 8x4x4 PCIe adapter board with two m.2 slots. In the 8x slot I put a GT1030 (above it). In the other slot (on the other side), I put a Renesas USB controller. This put the nvme drive in a vertical orientation. It now faced the CPU/memory slots on my motherboard. Also, I ran some additional cables to some USB 2.0 ports in the PCI slot. I actually thought this would reduce temperatures. After the ZFS and position change, Unraid began sending notifications that the drive was hitting 51-54 C regularly. It never went above 54C. These temperatures should be well within the specification limits Tested: Does Your M.2 NVMe SSD Need a Heatsink? Then, after getting these errors my VMs would become unresponsive, the VM page in the webGUI also became unresponsive (the loading bar never finished), and CPU usage skyrocketed. On the webGUI's Drives page, I believe the pool drive started showing up as missing and/or showed a size of 0 GB (should be 512 GB). I don't think I have a screenshot. I could still ssh in from a client - not the webGUI, but trying to shutdown the system didn't work. I had to manually hold down the power button for six seconds. I was able to collect one message from the drive page that the ZFS pool was suspended: pool: pool_one state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC config: NAME STATE READ WRITE CKSUM pool_one ONLINE 0 0 0 /dev/nvme0n1p1 ONLINE 2 24.9M 0 errors: 11 data errors, use '-v' for a list I could not recover or run this command after reboot. Wild Guesses on the Root Cause The reduced airflow and higher temperatures caused a the problem. Maybe the temperatures are causing the PCB to flex and weaken the connection or the NVME drive is more temperamental at higher temperatures. The 8x4x4 PCIe adapter could be poorly made and could be introducing signal integrity issues or other interference. My motherboard could have bifurcation issues. I haven't run the first PCIe slot (1) in 4x4x4x4 mode frequently in the past, but the other 16x slot (3) happily ran in 4x4x4x4 mode. The last thing is why I'm posting: Something is going wrong with Unraid's zfs pool implementation. I'm not sure how I should proceed. I don't want to re-add this drive if the same behavior will occur. I relocated the drive to the original m.2 slot and began manually backing up the /domains and /"appdata backup" folder created by the appdata backup plug-in in case this is a sign the drive is dying. Ironically, it reached 60 C when I was copying data off of it at 200 MB/s to a USB drive. I'm not sure what follow-up help I can offer. I'm not sure how many logs are left or where to find them. I was already running all 2.5" SSDs on the array off of a SAS HBA to a SAS expander in another chassis, and just treating this as a JBOD without parity drives... so the 'cache' or 'pool' usage is not that critical. Here's an output of the smartctl... I don't see any issues. smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.38-Unraid] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: KINGSTON RBUSNS8154P3512GJ1 Serial Number: <Redacted> Firmware Version: E8FK12.3 PCI Vendor/Subsystem ID: 0x2646 IEEE OUI Identifier: 0x0026b7 Total NVM Capacity: 512,110,190,592 [512 GB] Unallocated NVM Capacity: 0 Controller ID: 0 NVMe Version: 1.2 Number of Namespaces: 1 Namespace 1 Size/Capacity: 512,110,190,592 [512 GB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 0026b7 6830511ae5 Local Time is: Thu Aug 10 21:32:54 2023 PDT Firmware Updates (0x12): 1 Slot, no Reset required Optional Admin Commands (0x0007): Security Format Frmw_DL Optional NVM Commands (0x001e): Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Log Page Attributes (0x04): Ext_Get_Lg Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 84 Celsius Critical Comp. Temp. Threshold: 88 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 7.90W 0.0790W - 0 0 0 0 600 600 1 + 7.90W 0.0790W - 0 0 0 0 600 600 2 + 7.90W 0.0790W - 0 0 0 0 600 600 3 - 0.1000W 0.0790W - 3 3 3 3 1000 1000 4 - 0.0050W 0.0790W - 4 4 4 4 400000 90000 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 1 1 - 4096 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 48 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 7% Data Units Read: 31,228,641 [15.9 TB] Data Units Written: 59,681,139 [30.5 TB] Host Read Commands: 488,266,303 Host Write Commands: 842,469,025 Controller Busy Time: 3,185 Power Cycles: 889 Power On Hours: 19,886 Unsafe Shutdowns: 36 Media and Data Integrity Errors: 0 Error Information Log Entries: 2,409 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 2: 48 Celsius Error Information (NVMe Log 0x01, 16 of 16 entries) Num ErrCount SQId CmdId Status PELoc LBA NSID VS 0 2409 0 0x001c 0x0005 - 6 0 - 1 2408 0 0x0015 0x0005 - 12 0 - 2 2407 0 0x5010 0x0005 - 6 0 - 3 2406 0 0x0011 0x0005 - 12 0 - 4 2405 0 0x0010 0x0005 - 6 0 - 5 2404 0 0x0009 0x0005 - 12 0 - 6 2403 0 0x000a 0x0005 - 6 0 - 7 2402 0 0x0001 0x0005 - 12 0 - 8 2401 0 0x5010 0x0005 - 6 0 - 9 2400 0 0x0011 0x0005 - 12 0 - 10 2399 0 0x5014 0x0005 - 6 0 - 11 2398 0 0x0015 0x0005 - 12 0 - 12 2397 0 0x0000 0x0005 - 6 0 - 13 2396 0 0x0005 0x0005 - 12 0 - 14 2395 0 0x0001 0x0005 - 6 0 - 15 2394 0 0x0004 0x0005 - 12 0 -
August 12, 20232 yr Author False alarm. Thank you for prompting me to do this. I reconfigured the machine to its problematic configuration and began collecting diagnostics, however I tried using btrfs on the cache drive. I ended up encountering the same issue of the pool drive disappearing. I think this confirms my suspicion that this is a hardware-related issue.
August 12, 20232 yr Community Expert If the NVMe device is dropping there's one thing that can help, but need diags to confirm if it is.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.