Hello,

 

I got some problems yesterday evening.

First my NVME Cache went Read Only. I copied all the files to my Array and wanted to reformat it from BTRFS in to XFS.

My Array did not stop so I forced a reboot with sudo reboot -f

After the reboot my Cache started working fine again

I see some errors related to my OS flash drive, but I dont know what to do with them.

How bad is this error? Everything seems to run fine at the moment.

 

I read a lot of posts that BTRFS works really bad on NVME and gets easily corrupt. Is this true?

Should I reformat my cache? With Sata SSD Cache I had no problems in 3 years.

 

Cache error log before reboot:

Quote

Jan 11 21:20:54 unraid nginx: 2024/01/11 21:20:54 [error] 10117#10117: *3212683 upstream prematurely closed connection while reading response header from upstream, client: 192.168.178.110, server: , request: "GET /logterminal/jellyfin.log/ws HTTP/1.1", upstream: "http://unix:/var/tmp/jellyfin.log.sock:/ws", host: "192.168.178.200"

Jan 11 21:21:15 unraid kernel: loop: Write error at byte offset 93716480, length 4096.

Jan 11 21:21:15 unraid kernel: loop: Write error at byte offset 40042496, length 4096.

Jan 11 21:21:15 unraid kernel: I/O error, dev loop2, sector 183040 op 0x1:(WRITE) flags 0x1800 phys_seg 20 prio class 2

Jan 11 21:21:15 unraid kernel: I/O error, dev loop2, sector 78208 op 0x1:(WRITE) flags 0x1800 phys_seg 20 prio class 2

Jan 11 21:21:15 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

Jan 11 21:21:15 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0

Jan 11 21:21:15 unraid kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction)

 

OS Flash error:

Quote

Jan 12 02:14:18 unraid kernel: critical medium error, dev sda, sector 59595752 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current] 

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0 

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 03 8d 5d e0 00 00 40 00

Jan 12 02:14:18 unraid kernel: critical medium error, dev sda, sector 59596256 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 2

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current] 

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0 

Jan 12 02:14:18 unraid kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 03 8d 5d 60 00 00 08 00

Jan 12 02:14:18 unraid kernel: critical medium error, dev sda, sector 59596128 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

 

Another error I cant fix:

Quote

ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1

Jan 12 08:34:01 unraid crond[1860]: failed parsing crontab for user root: Invalid frequency setting of  /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1

 

unraid-diagnostics-20240112-0902.zip

29 minutes ago, Pascal51882 said:

Jan 11 21:21:15 unraid kernel: I/O error, dev loop2, sector 183040 op 0x1:(WRITE) flags 0x1800 phys_seg 20 prio class 2

Jan 11 21:21:15 unraid kernel: I/O error, dev loop2, sector 78208 op 0x1:(WRITE) flags 0x1800 phys_seg 20 prio class 2

Jan 11 21:21:15 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

Jan 11 21:21:15 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0

Jan 11 21:21:15 unraid kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out

These are about the docker image, you can recreate it.

 

30 minutes ago, Pascal51882 said:

Jan 12 02:14:18 unraid kernel: critical medium error, dev sda, sector 59595752 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

If these continue you should replace the flash drive, or at least try to re-format it to see if it helps.

 

 

 

 

On 1/12/2024 at 3:38 PM, JorgeB said:

These are about the docker image, you can recreate it.

 

If these continue you should replace the flash drive, or at least try to re-format it to see if it helps.

Hello the problem came back out of no where.

I did reformat the Cache drive and created Docker from scratch.

Took couple of months to get the error again.

 

Do the errors say anything? What should I try next?

It would have been best to run a system diagnosis but I didn't think about that at the time.

Quote

Jul  9 21:25:21 unraid kernel: BTRFS error (device nvme0n1p1: state EA): bdev /dev/nvme0n1p1 errs: wr 36, rd 197082, flush 0, corrupt 0, gen 0

Jul  9 21:25:21 unraid kernel: BTRFS error (device nvme0n1p1: state EA): bdev /dev/nvme0n1p1 errs: wr 36, rd 197083, flush 0, corrupt 0, gen 0

Jul  9 21:25:21 unraid kernel: BTRFS error (device nvme0n1p1: state EA): bdev /dev/nvme0n1p1 errs: wr 36, rd 197084, flush 0, corrupt 0, gen 0

Jul  9 21:25:21 unraid kernel: BTRFS error (device nvme0n1p1: state EA): bdev /dev/nvme0n1p1 errs: wr 36, rd 197085, flush 0, corrupt 0, gen 0

Jul  9 21:25:21 unraid kernel: BTRFS error (device nvme0n1p1: state EA): bdev /dev/nvme0n1p1 errs: wr 36, rd 197086, flush 0, corrupt 0, gen 0

Jul  9 21:25:21 unraid kernel: BTRFS error (device nvme0n1p1: state EA): bdev /dev/nvme0n1p1 errs: wr 36, rd 197087, flush 0, corrupt 0, gen 0

 

Edited July 16, 20241 yr by Pascal51882

That suggests the NVMe device dropped offline, see here for more info:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582

 

Looks like the nvme ssd is bad? Its strange that it takes almost 6 months every time.

Its strange that its always between 21:00-22:00.

I have very frequent reboots every few weeks.

 

Could also be the Port of the mainboard?

I could update the Bios and try another port. 

But so far no reports of bad Bios version 4403 on TUF GAMING X570-PLUS.

Could be all that, this sometimes also helps:

 

Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

Reboot and see if it makes a difference.