(solved) (6.8.2) nvme drive dying?

[munit85]

Hi all. experiencing some crashing.  My dockers died and vm's died.  logs show continuous errors.

Aug 28 06:48:07 Jefflix kernel: print_req_error: I/O error, dev loop2, sector 180736
Aug 28 06:48:07 Jefflix kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 4050, flush 0, corrupt 0, gen 0

The rest of the array seems to be online and fine. and the vm's show "One or more paths do not exist"

 

I also already deleted my docker image and recreated it. that worked for a few hours, but the next day it was in this error state again.

 

log attached.

jefflix-diagnostics-20200828-1633.zip

Edited September 1, 2020 by munit85

[JorgeB]

Yes, it's a problem with the NVMe device:

 

Aug 28 04:36:36 Jefflix kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Aug 28 04:36:36 Jefflix kernel: print_req_error: I/O error, dev nvme0n1, sector 585987336
Aug 28 04:36:36 Jefflix kernel: print_req_error: I/O error, dev nvme0n1, sector 23321408
Aug 28 04:36:36 Jefflix kernel: print_req_error: I/O error, dev nvme0n1, sector 167618128
Aug 28 04:36:36 Jefflix kernel: print_req_error: I/O error, dev nvme0n1, sector 320839472
Aug 28 04:36:36 Jefflix kernel: print_req_error: I/O error, dev nvme0n1, sector 320839488
Aug 28 04:36:36 Jefflix kernel: nvme 0000:01:00.0: Refused to change power state, currently in D3
Aug 28 04:36:36 Jefflix kernel: nvme nvme0: Removing after probe failure status: -19

 

But it doesn't necessarily mean it's dying, some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, between "append" and before "initrd=/bzroot"

 

nvme_core.default_ps_max_latency_us=0

 

Reboot and see if it makes a difference.

 

 

 

 

 

[munit85]

12 hours ago, johnnie.black said:

But it doesn't necessarily mean it's dying, some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, between "append" and before "initrd=/bzroot"

 

nvme_core.default_ps_max_latency_us=0

 

Reboot and see if it makes a difference.

testing now.  Flash now reads

 

append nvme_core.default_ps_max_latency_us=0 initrd=/bzroot,/bzroot-gui

I'll report back with what happens later today.  thanks

 

//edit  it's been hours and so far all is well.  If it doesn't have issues by tomorrow I'll consider this solved.

 

Quote

 

 

 

 

 

 

Edited August 29, 2020 by munit85

[munit85]

hmm just checked and errors returned.

 

attaching new diagnostics

jefflix-diagnostics-20200829-1919.zip

[JorgeB]

Similar problem, assuming it's well seated best bet would be to try the latest Unraid beta, newer kernel might help, if it doesn't get a different model NVMe device, if possible.

[munit85]

On 8/30/2020 at 3:08 AM, johnnie.black said:

Similar problem, assuming it's well seated best bet would be to try the latest Unraid beta, newer kernel might help, if it doesn't get a different model NVMe device, if possible.

well i did what you said and it worked. no errors to be found. Ordered a different drive as a backup, but either reseating or updating to the beta fixed it.

 

thanks so much