NVMe Cache Dropping Offline ~ 5min after boot

[Siwat2545]

I have 2 crucial p5 1TB NVMe SSD that I just installed to replace my 4 860 Evos

However, after I format the cache pool, one of the NVMe Drive drop offline and generate the error message 

Dec 10 01:19:03 MainServer kernel: nvme nvme0: Device not ready; aborting reset
Dec 10 01:19:03 MainServer kernel: nvme nvme0: Removing after probe failure status: -19
Dec 10 01:19:04 MainServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Dec 10 01:19:06 MainServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Dec 10 01:19:08 MainServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Dec 10 01:19:09 MainServer kernel: nvme nvme0: Device not ready; aborting reset
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 244190608, async page read
Dec 10 01:19:09 MainServer kernel: print_req_error: I/O error, dev nvme0n1, sector 64
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 244190608, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 0, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 0, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 4, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 8, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 16, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 32, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 64, async page read
Dec 10 01:19:09 MainServer kernel: Buffer I/O error on dev nvme0n1p1, logical block 128, async page read
Dec 10 01:19:09 MainServer root: Starting diskload
Dec 10 01:19:09 MainServer kernel: nvme nvme0: failed to set APST feature (-19)

Unraid Cache disk message: 10-12-2020 01:19
Warning [MAINSERVER] - Cache pool BTRFS missing device(s)
CT1000P5SSD8_20292A3294C4 (nvme0n1)

I already tried adding nvme_core.default_ps_max_latency_us=0 to my syslinux to no avails

The ssd is attached to an ASUS PCI-e 16x to 4 NVMe card

My motherboard is intel s2600cw2r

This behavior happen when I use unassigned device too (But a lot more rare), this might indicate a hardware issue, idk

Unassigned device format to BTRFS Log:

Dec 10 01:32:33 MainServer kernel: nvme1n1: p1
Dec 10 01:32:33 MainServer kernel: BTRFS: device fsid 1f5b7280-dad4-48cf-a438-26ef489d2dbe devid 2 transid 73 /dev/nvme1n1p1
Dec 10 01:34:08 MainServer emhttpd: CT1000P5SSD8_20292A3294C4 (nvme1n1) 512 1953525168
Dec 10 01:34:08 MainServer emhttpd: import 30 cache device: (nvme1n1) CT1000P5SSD8_20292A3294C4
Dec 10 01:35:34 MainServer emhttpd: CT1000P5SSD8_20292A3294C4 (nvme1n1) 512 1953525168
Dec 10 01:35:39 MainServer emhttpd: CT1000P5SSD8_20292A3294C4 (nvme1n1) 512 1953525168
Dec 10 01:35:41 MainServer emhttpd: CT1000P5SSD8_20292A3294C4 (nvme1n1) 512 1953525168
Dec 10 01:35:44 MainServer emhttpd: CT1000P5SSD8_20292A3294C4 (nvme1n1) 512 1953525168
Dec 10 01:36:03 MainServer unassigned.devices: Don't spin down device '/dev/nvme1n1'.
Dec 10 01:36:33 MainServer unassigned.devices: Adding disk '/dev/nvme1n1p1'...
Dec 10 01:36:33 MainServer unassigned.devices: Mount drive command: /sbin/mount -t btrfs -o rw,auto,async,noatime,nodiratime,discard '/dev/nvme1n1p1' '/mnt/disks/CT1000P5SSD8_20292A3294C4'
Dec 10 01:36:33 MainServer unassigned.devices: Successfully mounted '/dev/nvme1n1p1' on '/mnt/disks/CT1000P5SSD8_20292A3294C4'.
Dec 10 01:36:33 MainServer unassigned.devices: Don't spin down device '/dev/nvme1n1'.
Dec 10 01:38:41 MainServer unassigned.devices: Don't spin down device '/dev/nvme1n1'.
Dec 10 01:38:41 MainServer unassigned.devices: Unmounting '/dev/nvme1n1p1'...
Dec 10 01:38:41 MainServer unassigned.devices: Unmount cmd: /sbin/umount '/dev/nvme1n1p1' 2>&1
Dec 10 01:38:41 MainServer unassigned.devices: Successfully unmounted '/dev/nvme1n1p1'
Dec 10 01:38:48 MainServer unassigned.devices: Removing partition '1' from disk '/dev/nvme1n1'.
Dec 10 01:38:53 MainServer kernel: nvme1n1:
Dec 10 01:39:36 MainServer unassigned.devices: Device '/dev/nvme1n1' block size: 1953525168
Dec 10 01:39:36 MainServer unassigned.devices: Clearing partition table of disk '/dev/nvme1n1'.
Dec 10 01:39:36 MainServer unassigned.devices: Reloading disk '/dev/nvme1n1' partition table.
Dec 10 01:39:36 MainServer unassigned.devices: Reload partition table result: /dev/nvme1n1: re-reading partition table
Dec 10 01:39:36 MainServer unassigned.devices: Creating Unraid compatible mbr partition on disk '/dev/nvme1n1'.
Dec 10 01:39:36 MainServer unassigned.devices: Reloading disk '/dev/nvme1n1' partition table.
Dec 10 01:39:36 MainServer kernel: nvme1n1: p1
Dec 10 01:39:36 MainServer kernel: nvme1n1: p1
Dec 10 01:39:36 MainServer unassigned.devices: Reload partition table result: /dev/nvme1n1: re-reading partition table
Dec 10 01:39:36 MainServer unassigned.devices: Formatting disk '/dev/nvme1n1' with 'btrfs' filesystem.
Dec 10 01:39:36 MainServer kernel: BTRFS: device fsid 5854ac6a-e331-452a-bd17-5b0337196c18 devid 1 transid 5 /dev/nvme1n1p1
Dec 10 01:39:36 MainServer unassigned.devices: Format disk '/dev/nvme1n1' with 'btrfs' filesystem result: btrfs-progs v5.4.1 See http://btrfs.wiki.kernel.org for more information. Detected a SSD, turning off metadata duplication. Mkfs with -m dup if you want to force metadata duplication. Label: (null) UUID: 5854ac6a-e331-452a-bd17-5b0337196c18 Node size: 16384 Sector size: 4096 Filesystem size: 931.51GiB Block group profiles: Data: single 8.00MiB Metadata: single 8.00MiB System: single 4.00MiB SSD detected: yes Incompat features: extref, skinny-metadata Checksum: crc32c Number of devices: 1 Devices: ID SIZE PATH 1 931.51GiB /dev/nvme1n1p1
Dec 10 01:39:39 MainServer unassigned.devices: Reloading disk '/dev/nvme1n1' partition table.
Dec 10 01:39:39 MainServer kernel: nvme1n1: p1
Dec 10 01:39:39 MainServer unassigned.devices: Reload partition table result: /dev/nvme1n1: re-reading partition table
Dec 10 01:39:50 MainServer unassigned.devices: Adding disk '/dev/nvme1n1p1'...
Dec 10 01:39:50 MainServer unassigned.devices: Mount drive command: /sbin/mount -t btrfs -o rw,auto,async,noatime,nodiratime,discard '/dev/nvme1n1p1' '/mnt/disks/CT1000P5SSD8_20292A3294C4'
Dec 10 01:39:50 MainServer kernel: BTRFS info (device nvme1n1p1): turning on discard
Dec 10 01:39:50 MainServer kernel: BTRFS info (device nvme1n1p1): disk space caching is enabled
Dec 10 01:39:50 MainServer kernel: BTRFS info (device nvme1n1p1): has skinny extents
Dec 10 01:39:50 MainServer kernel: BTRFS info (device nvme1n1p1): flagging fs with big metadata feature
Dec 10 01:39:50 MainServer kernel: BTRFS info (device nvme1n1p1): enabling ssd optimizations
Dec 10 01:39:50 MainServer kernel: BTRFS info (device nvme1n1p1): checking UUID tree
Dec 10 01:39:50 MainServer unassigned.devices: Successfully mounted '/dev/nvme1n1p1' on

 

Currently only nvme0n1 drop offline (nvme1n1 dropped offline once before) 

On the latest boot I ran CrystalDiskMark on nvme1n1 without issues

 

but nvme0n1 is already gone after 15 min in unassigned drive without me even doing anything to it

As it stand, I will see if nvme1n1 drop offline if it leave it overnight, IDK what to do with nvme0n1 (already tried swapping the slot), It will take a while until I trust this thing to store my data .....

mainserver-diagnostics-20201210-0122.zip

Edited December 9, 2020 by Siwat2545