Hey all,

 

I upgraded my nvme drive on a HP Prodesk 400 G6 mini pc to a 1TB XPG S70. Since doing so, the drive will be fine for a bit, but then start cluttering the logs with errors and then becoming unavailable. Upon stopping the array, the drive vanishes. A reboot "fixes" things until the errors start again. I've tried reseating the drive once, thinking maybe that was the problem. I've also done a BIOS update, hoping that would help. Thoughts?

Jan 16 13:24:42 Tower kernel: nvme nvme0: I/O 115 QID 2 timeout, aborting
Jan 16 13:24:50 Tower kernel: nvme nvme0: I/O 480 QID 1 timeout, aborting
Jan 16 13:24:50 Tower kernel: nvme nvme0: I/O 481 QID 1 timeout, aborting
Jan 16 13:24:50 Tower kernel: nvme nvme0: I/O 482 QID 1 timeout, aborting
Jan 16 13:25:12 Tower kernel: nvme nvme0: I/O 115 QID 2 timeout, reset controller
Jan 16 13:25:42 Tower kernel: nvme nvme0: I/O 2 QID 0 timeout, reset controller
Jan 16 13:27:05 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jan 16 13:27:05 Tower kernel: nvme nvme0: Abort status: 0x371
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan 16 13:27:55 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jan 16 13:27:55 Tower kernel: nvme nvme0: Removing after probe failure status: -19
Jan 16 13:28:46 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 1001944288 op 0x1:(WRITE) flags 0x1000 phys_seg 4 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 1002150864 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 1001530688 op 0x1:(WRITE) flags 0x1000 phys_seg 4 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 1001196896 op 0x1:(WRITE) flags 0x1000 phys_seg 4 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 25776 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 1000205688 op 0x1:(WRITE) flags 0x1000 phys_seg 2 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 1000205666 op 0x1:(WRITE) flags 0x1000 phys_seg 1 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 500657648 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 500818448 op 0x1:(WRITE) flags 0x1000 phys_seg 4 prio class 0
Jan 16 13:28:46 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 25768 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 1075631154, offset 1212416, sector 1002150872
Jan 16 13:28:46 Tower kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_buf_ioend+0x12d/0x284 [xfs]" at daddr 0x3bb86ce0 len 32 error 5
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 537586352, offset 0, sector 500788960
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 537403597, offset 0, sector 500657656
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 537586346, offset 0, sector 500832176
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 537586348, offset 0, sector 500788960
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 1075631154, offset 1212416, sector 1002150864
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 1611570215, offset 0, sector 1501284352
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 1611570215, offset 16384, sector 1501284392
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 537586350, offset 0, sector 500786856
Jan 16 13:28:46 Tower kernel: nvme0n1p1: writeback error on inode 232, offset 761856, sector 25760
Jan 16 13:28:46 Tower kernel: XFS (nvme0n1p1): log I/O error -5
Jan 16 13:28:46 Tower kernel: XFS (nvme0n1p1): xfs_do_force_shutdown(0x2) called from line 1196 of file fs/xfs/xfs_log.c. Return address = 00000000b5b54af3
Jan 16 13:28:46 Tower kernel: XFS (nvme0n1p1): Log I/O Error Detected. Shutting down filesystem
Jan 16 13:28:46 Tower kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s)
Jan 16 13:28:46 Tower kernel: XFS (nvme0n1p1): log I/O error -5
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 16 13:28:46 Tower kernel: nvme nvme0: failed to set APST feature (-19)

 

tower-diagnostics-20220116-1600.zip

Edited January 16, 20224 yr by bmrowe

I just added:

 

nvme_core.default_ps_max_latency_us=0 

 

as is mentioned in a few posts and here: https://johnespiritu.dev/blog/random-nvme-crash-manjaro/. Not sure if thats the root cause but wanted to update the thread.

It might help, look also for a BIOS/firmware update.

5 hours ago, JorgeB said:

It might help, look also for a BIOS/firmware update.

Thanks. Did the BIOS update. Any tricks on firmware updates for an nvme m.2 drive? The tool provided by ADATA is a windows app. You don't really have to pull the drive and install it in a windows machine to update the firmware, do you?

9 minutes ago, bmrowe said:

You don't really have to pull the drive and install it in a windows machine to update the firmware, do you?

If they don't provide a way of doing it using Linux you're limited to Windows.