- 7.2.0-beta.3
- Solved
Not a direct bug in beta2, but a possible bug fix/mitigation for an ongoing NVME issue.
Some folks (me included) have had problems with NVME SSDs dropping off and requiring a hard power cycle to recover them.
I was just looking through the kernel change log and saw this:
It looks like that was released in 6.12.43, and we're currently on 6.12.42 in 7.2 beta 2
Could we move beta 3 to 6.12.43/44 to pick up this fix?
The errors mentioned are exactly what I see when the drive drops off and doesn't come back.
e.g.
an 26 10:31:17 unraid kernel: nvme nvme1: I/O 840 (I/O Cmd) QID 1 timeout, aborting
Jan 26 10:31:17 unraid kernel: nvme nvme1: I/O 537 (I/O Cmd) QID 5 timeout, aborting
Jan 26 10:31:17 unraid kernel: nvme nvme1: I/O 797 (I/O Cmd) QID 9 timeout, aborting
Jan 26 10:31:17 unraid kernel: nvme nvme1: Abort status: 0x0
Jan 26 10:31:17 unraid kernel: nvme nvme1: Abort status: 0x0
Jan 26 10:31:17 unraid kernel: nvme nvme1: Abort status: 0x0
Jan 26 10:31:18 unraid kernel: nvme nvme1: I/O 980 (I/O Cmd) QID 15 timeout, aborting
Jan 26 10:31:18 unraid kernel: nvme nvme1: Abort status: 0x0
Jan 26 10:31:18 unraid kernel: nvme nvme1: I/O 536 (I/O Cmd) QID 5 timeout, aborting
Jan 26 10:31:47 unraid kernel: nvme nvme1: I/O 840 QID 1 timeout, reset controller
Jan 26 10:32:18 unraid kernel: nvme nvme1: I/O 0 QID 0 timeout, reset controller
Jan 26 10:33:08 unraid kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
Jan 26 10:33:08 unraid kernel: nvme nvme1: Abort status: 0x371
Jan 26 10:33:29 unraid kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
Jan 26 10:33:29 unraid kernel: nvme nvme1: Removing after probe failure status: -19
Jan 26 10:33:49 unraid kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
Jan 26 10:33:49 unraid kernel: nvme1n1: detected capacity change from 3907029168 to 0
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114661376 size=61440 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114137088 size=12288 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114006016 size=49152 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114530304 size=131072 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908113088512 size=131072 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=1804943826944 size=131072 flags=1808b0
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114399232 size=131072 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=1804943695872 size=131072 flags=1808b0
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908110991360 size=12288 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=1804943564800 size=131072 flags=1808b0
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908113874944 size=28672 flags=180880
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 01:46:15 2025] docker0: port 3(vethbb8ba86) entered forwarding state
[Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 821 (I/O Cmd) QID 2 timeout, aborting
[Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 822 (I/O Cmd) QID 2 timeout, aborting
[Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 260 (I/O Cmd) QID 14 timeout, aborting
[Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0
[Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0
[Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0
[Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 888 (I/O Cmd) QID 3 timeout, aborting
[Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0
[Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 931 (I/O Cmd) QID 7 timeout, aborting
[Sun Jun 1 10:36:25 2025] nvme nvme1: I/O 821 QID 2 timeout, reset controller
[Sun Jun 1 10:36:56 2025] nvme nvme1: I/O 3 QID 0 timeout, reset controller
[Sun Jun 1 10:37:47 2025] nvme nvme1: Device not ready; aborting reset, CSTS=0x1
[Sun Jun 1 10:37:47 2025] nvme nvme1: Abort status: 0x371
[Sun Jun 1 10:38:07 2025] nvme nvme1: Device not ready; aborting reset, CSTS=0x1
[Sun Jun 1 10:38:07 2025] nvme nvme1: Removing after probe failure status: -19
[Sun Jun 1 10:38:28 2025] nvme nvme1: Device not ready; aborting reset, CSTS=0x1
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1825897586688 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610704896 size=53248 flags=180880
[Sun Jun 1 10:38:28 2025] nvme1n1: detected capacity change from 3907029168 to 0
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610442752 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610311680 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917606903808 size=12288 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=97925271552 size=131072 flags=1808b0
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610573824 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917607559168 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1825897717760 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=97925140480 size=131072 flags=1808b0
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=97925009408 size=131072 flags=1808b0
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1906971295744 size=45056 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610049536 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1825897848832 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610180608 size=131072 flags=180880
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
[Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480
Ref: