I recently had a string of bad luck with my sata cache SSD drives.  So I thought I would order 2 NVME drives and start fresh. I've been having trouble doing that though, I tried at first to just create a pool and add both drives, but they wouldn't format, so then I restarted, pre-cleared both drives, then tried creating the pool and adding them, but they still wouldn't format.  They are both brand new, detected in bios and passed the MSI bios nvme test tool as well as pre-clear and SMART tests.  But I can't get either of them to format as a pool device.  Did I just have bad luck and get 2 bad drives? Or is something else wrong?

Thanks for taking a look

-e

Apr 18 21:33:09 Pantainos emhttpd: creating volume: cache (btrfs)
Apr 18 21:33:09 Pantainos emhttpd: shcmd (155): /sbin/wipefs -af --lock /dev/nvme1n1p1
Apr 18 21:33:09 Pantainos emhttpd: shcmd (156): /sbin/blkdiscard /dev/nvme1n1p1
Apr 18 21:33:39 Pantainos kernel: nvme nvme1: I/O 897 (I/O Cmd) QID 3 timeout, aborting
Apr 18 21:33:39 Pantainos kernel: nvme nvme1: I/O 898 (I/O Cmd) QID 3 timeout, aborting
Apr 18 21:33:39 Pantainos kernel: nvme nvme1: I/O 899 (I/O Cmd) QID 3 timeout, aborting
Apr 18 21:34:10 Pantainos kernel: nvme nvme1: I/O 897 QID 3 timeout, reset controller
Apr 18 21:37:18 Pantainos kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
Apr 18 21:37:18 Pantainos kernel: nvme nvme1: Abort status: 0x371
Apr 18 21:37:18 Pantainos kernel: nvme nvme1: Abort status: 0x371
Apr 18 21:37:18 Pantainos kernel: nvme nvme1: Abort status: 0x371

 

pantainos-diagnostics-20250418-2146.zip

Edited April 19, 20251 yr by Happydish

6 hours ago, Happydish said:

Apr 18 21:33:39 Pantainos kernel: nvme nvme1: I/O 897 (I/O Cmd) QID 3 timeout, aborting
Apr 18 21:33:39 Pantainos kernel: nvme nvme1: I/O 898 (I/O Cmd) QID 3 timeout, aborting
Apr 18 21:33:39 Pantainos kernel: nvme nvme1: I/O 899 (I/O Cmd) QID 3 timeout, aborting
Apr 18 21:34:10 Pantainos kernel: nvme nvme1: I/O 897 QID 3 timeout, reset controller

This looks like a bad device, or maybe a bad firmware, that is not correctly handling the full trim command (blkdiscard).

 

 

 

 

A little more info, I tried a fresh reboot, and formatting through unassigned devices.  It seems like it works just fine, formatted them to btrfs and added them both to a new 2 slot pool and one of them worked  (nvme0).  So I was thinking that maybe just nvme1 was bad?  Shut down the array, removed the pool, created a new pool with just nvme0, and now it won't format either.

Apr 19 08:51:37 Pantainos unassigned.devices: Formatting device '/dev/nvme0n1'.
Apr 19 08:51:37 Pantainos unassigned.devices: Device '/dev/nvme0n1' block size: 2000409264.
Apr 19 08:51:37 Pantainos unassigned.devices: Clearing partition table of disk '/dev/nvme0n1'.
Apr 19 08:51:37 Pantainos unassigned.devices: Clear partition result: 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0147597 s, 142 MB/s
Apr 19 08:51:40 Pantainos unassigned.devices: Successfully reloaded partition table on device '/dev/nvme0n1' with partprobe.
Apr 19 08:51:41 Pantainos unassigned.devices: Creating Unraid compatible mbr partition on disk '/dev/nvme0n1'.
Apr 19 08:51:41 Pantainos kernel: nvme0n1: p1
Apr 19 08:51:41 Pantainos unassigned.devices: Create mbr partition table result: write mbr signature done
Apr 19 08:51:44 Pantainos unassigned.devices: Successfully reloaded partition table on device '/dev/nvme0n1' with partprobe.
Apr 19 08:51:45 Pantainos unassigned.devices: Format device '/dev/nvme0n1' with 'btrfs' filesystem.
Apr 19 08:51:45 Pantainos unassigned.devices: Format command: /sbin/mkfs.btrfs -f '/dev/nvme0n1p1' 2>&1
Apr 19 08:51:46 Pantainos kernel: BTRFS: device fsid 0212fb2b-2468-4334-b27c-57895f5ec89f devid 1 transid 8 /dev/nvme0n1p1 scanned by mkfs.btrfs (38409)
Apr 19 08:51:50 Pantainos unassigned.devices: Successfully reloaded partition table on device '/dev/nvme0n1' with partprobe.
Apr 19 08:52:07 Pantainos unassigned.devices: Formatting device '/dev/nvme1n1'.
Apr 19 08:52:07 Pantainos unassigned.devices: Device '/dev/nvme1n1' block size: 2000409264.
Apr 19 08:52:07 Pantainos unassigned.devices: Clearing partition table of disk '/dev/nvme1n1'.
Apr 19 08:52:07 Pantainos unassigned.devices: Clear partition result: 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0165658 s, 127 MB/s
Apr 19 08:52:10 Pantainos unassigned.devices: Successfully reloaded partition table on device '/dev/nvme1n1' with partprobe.
Apr 19 08:52:11 Pantainos unassigned.devices: Creating Unraid compatible mbr partition on disk '/dev/nvme1n1'.
Apr 19 08:52:11 Pantainos kernel: nvme1n1: p1
Apr 19 08:52:11 Pantainos unassigned.devices: Create mbr partition table result: write mbr signature done
Apr 19 08:52:14 Pantainos unassigned.devices: Successfully reloaded partition table on device '/dev/nvme1n1' with partprobe.
Apr 19 08:52:15 Pantainos unassigned.devices: Format device '/dev/nvme1n1' with 'btrfs' filesystem.
Apr 19 08:52:15 Pantainos unassigned.devices: Format command: /sbin/mkfs.btrfs -f '/dev/nvme1n1p1' 2>&1
Apr 19 08:52:15 Pantainos kernel: BTRFS: device fsid dad2b098-9a6c-402a-9862-9c62a1b0eeeb devid 1 transid 8 /dev/nvme1n1p1 scanned by mkfs.btrfs (42655)
Apr 19 08:52:20 Pantainos unassigned.devices: Successfully reloaded partition table on device '/dev/nvme1n1' with partprobe.

The problem is occurring on both devices, when Unraid runs blkdiscrd (full device trim), UD doesn't do that before formatting, but the devices should support that command, it's a standard command, and I've seen another user with a same brand device with the exact same issue, so I think it's a device problem.