Upgrading from 6.6.7 to 6.9.1 broke after a while


Recommended Posts

I am seeing this on my cache drive:

 


Mar 21 22:26:12 Tower emhttpd: read SMART /dev/nvme0n1
Mar 21 22:52:12 Tower emhttpd: shcmd (1683): mount -t xfs -o noatime /dev/nvme0n1p1 /mnt/cache
Mar 21 22:52:12 Tower kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Mar 21 22:52:12 Tower kernel: XFS (nvme0n1p1): Ending clean mount
Mar 22 03:01:36 Tower root: /mnt/cache: 190.3 GiB (204309774336 bytes) trimmed on /dev/nvme0n1p1
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_imap_to_bp+0x5c/0xa2 [xfs]" at daddr 0xf4afae0 len 32 error 5
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_imap_to_bp+0x5c/0xa2 [xfs]" at daddr 0x2cf0d200 len 32 error 5
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_da_read_buf+0x9e/0xfe [xfs]" at daddr 0x2cf8c7b0 len 8 error 5
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 42077208 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 88096136 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 809978401, offset 0, sector 754835864
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 500518589 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 0
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 12732352 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 256356112 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 16747910, offset 37883904, sector 42077232
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 502525672 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 0
Mar 22 19:40:33 Tower kernel: blk_update_request: I/O error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): log I/O error -5
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 274737869, offset 32768, sector 256356120
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 274737869, offset 36864, sector 256356136
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 4573547, offset 24576, sector 4572576
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 4573547, offset 16384, sector 4572552
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 4573547, offset 0, sector 4572520
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): xfs_do_force_shutdown(0x2) called from line 1196 of file fs/xfs/xfs_log.c. Return address = 000000007f16e358
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): Log I/O Error Detected. Shutting down filesystem
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s)
Mar 22 19:40:33 Tower kernel: nvme0n1p1: writeback error on inode 277207800, offset 528384, sector 258101072
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): log I/O error -5
Mar 22 19:40:33 Tower kernel: XFS (nvme0n1p1): log I/O error -5

It is full of all these errors. See aimage.thumb.png.36cc0b07312c220019bcf31b5724304f.pngttach screenshot.

Edited by CyberMew
set title
Link to comment
  • CyberMew changed the title to Upgrading from 6.6.7 to 6.9.1 broke after a while

Cache device dropped offline:

 

Mar 22 19:37:30 Tower kernel: nvme nvme0: I/O 393 QID 6 timeout, aborting
Mar 22 19:37:30 Tower kernel: nvme nvme0: I/O 394 QID 6 timeout, aborting
Mar 22 19:37:30 Tower kernel: nvme nvme0: I/O 395 QID 6 timeout, aborting
Mar 22 19:37:30 Tower kernel: nvme nvme0: I/O 396 QID 6 timeout, aborting
Mar 22 19:37:33 Tower kernel: nvme nvme0: I/O 397 QID 6 timeout, aborting
Mar 22 19:37:33 Tower kernel: nvme nvme0: I/O 398 QID 6 timeout, aborting
Mar 22 19:37:35 Tower kernel: nvme nvme0: I/O 1007 QID 2 timeout, aborting
Mar 22 19:37:41 Tower kernel: nvme nvme0: I/O 675 QID 11 timeout, aborting
Mar 22 19:38:00 Tower kernel: nvme nvme0: I/O 393 QID 6 timeout, reset controller
Mar 22 19:38:31 Tower kernel: nvme nvme0: I/O 16 QID 0 timeout, reset controller
Mar 22 19:39:32 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Mar 22 19:39:32 Tower kernel: nvme nvme0: Abort status: 0x371
### [PREVIOUS LINE REPEATED 7 TIMES] ###
Mar 22 19:40:03 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Mar 22 19:40:03 Tower kernel: nvme nvme0: Removing after probe failure status: -19
Mar 22 19:40:33 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

 

A power cycle should bring it back, but look for a BIOS update for your board to see if it doesn't happen again, sometimes disabling NVMe power states also helps.

Link to comment

Thank you for the reassurance! Only seeing these yellow warnings:

Mar 22 22:19:24 Tower kernel: tsc: Fast TSC calibration failed
Mar 22 22:19:24 Tower kernel: ACPI: Early table checksum verification disabled
Mar 22 22:19:24 Tower kernel: pci 0000:27:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit]
Mar 22 22:19:24 Tower kernel: pci 0000:27:00.0: BAR 7: failed to assign [mem size 0x00040000 64bit]
Mar 22 22:19:24 Tower kernel: floppy0: no floppy controllers found
Mar 22 22:19:24 Tower kernel: random: 6 urandom warning(s) missed due to ratelimiting
Mar 22 22:19:27 Tower rpc.statd[1933]: Failed to read /var/lib/nfs/state: Success

 

But so far so good, hope it stays that way.. thanks a lot!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.