This is the second time in a week that this has happened with this drive. Crucial P2 500GB.
I thought it was an issue with the Hyper M.2 PCI-Express card that has previously been suspicious so I ended up rebuilding the server with a fresh motherboard so I could have the M.2 drives on the motherboard itself. So it has failed like this on both an old hardware configuration and a fresh one.
It is time to replace this drive or could there be something else that I am missing?
May 7 04:44:46 Emilia kernel: nvme nvme1: I/O 24 QID 5 timeout, aborting
May 7 04:45:17 Emilia kernel: nvme nvme1: I/O 24 QID 5 timeout, reset controller
May 7 04:45:47 Emilia kernel: nvme nvme1: I/O 10 QID 0 timeout, reset controller
May 7 04:48:29 Emilia kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
May 7 04:48:29 Emilia kernel: nvme nvme1: Abort status: 0x371
May 7 04:50:37 Emilia kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
May 7 04:50:37 Emilia kernel: nvme nvme1: Removing after probe failure status: -19
May 7 04:52:45 Emilia kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
May 7 04:52:45 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
May 7 04:52:45 Emilia kernel: blk_update_request: I/O error, dev nvme1n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
May 7 04:52:45 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 0, flush 1, corrupt 0, gen 0
May 7 04:52:45 Emilia kernel: BTRFS warning (device nvme1n1p1): chunk 393078636544 missing 1 devices, max tolerance is 0 for writable mount
May 7 04:52:45 Emilia kernel: BTRFS: error (device nvme1n1p1) in write_all_supers:3845: errno=-5 IO failure (errors while submitting device barriers.)
May 7 04:52:45 Emilia kernel: BTRFS info (device nvme1n1p1): forced readonly
May 7 04:52:45 Emilia kernel: BTRFS warning (device nvme1n1p1): Skipping commit of aborted transaction.
May 7 04:52:45 Emilia kernel: BTRFS: error (device nvme1n1p1) in cleanup_transaction:1942: errno=-5 IO failure
May 7 04:52:45 Emilia kernel: BTRFS warning (device nvme1n1p1): Skipping commit of aborted transaction.
May 7 04:52:45 Emilia kernel: BTRFS: error (device nvme1n1p1) in cleanup_transaction:1942: errno=-5 IO failure
May 7 04:52:45 Emilia kernel: nvme nvme1: failed to set APST feature (-19)
May 7 04:54:01 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 1, flush 1, corrupt 0, gen 0
May 7 04:54:01 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 2, flush 1, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 125920 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 650208 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 125664 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 125824 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 125984 op 0x1:(WRITE) flags 0x1800 phys_seg 48 prio class 0
May 7 04:54:06 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 649952 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 650112 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 650272 op 0x1:(WRITE) flags 0x1800 phys_seg 48 prio class 0
May 7 04:54:06 Emilia kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2377: errno=-5 IO failure (Error while writing out transaction)
May 7 04:54:06 Emilia kernel: BTRFS info (device loop2): forced readonly
May 7 04:54:06 Emilia kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
May 7 04:54:06 Emilia kernel: BTRFS: error (device loop2) in cleanup_transaction:1942: errno=-5 IO failure
May 7 04:54:06 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 27288 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
May 7 04:54:07 Emilia kernel: btrfs_dev_stat_print_on_error: 11 callbacks suppressed
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 3, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de6a78 len 4096 err no 10
May 7 04:54:07 Emilia kernel: blk_update_request: I/O error, dev loop2, sector 1297496 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 4, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 5, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7228 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 6, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7230 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7238 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 7, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 8, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 9, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7300 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7308 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7310 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 10, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7310 len 4096 err no 10
May 7 04:54:07 Emilia kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 19, rd 1, flush 0, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 11, flush 1, corrupt 0, gen 0
May 7 04:54:07 Emilia kernel: BTRFS warning (device nvme1n1p1): direct IO failed ino 263 rw 0,0 sector 0x3de7310 len 4096 err no 10