Just noticed that log was at 100% so I had a quick look through it and the last hour was riddled with BTRFS errors. I currently have a mix of xfs drives and reiserfs drives (eliminating the later right now, just moving things around).

Any insight as to why these errors are popping up?

 

Of note - unbalance is currently running, moving data off reiserfs drives onto xfs drives before they are either pulled or reformatted.

 

The NVME cache has been in use for a few weeks.

Attached diagnostics.

 

 

tower-diagnostics-20210121-1140.zip

Edited January 21, 20215 yr by benyaki

NVMe device dropped offline:

 

Jan 21 10:23:41 Tower kernel: nvme nvme0: I/O 979 QID 6 timeout, aborting
Jan 21 10:23:41 Tower kernel: nvme nvme0: I/O 980 QID 6 timeout, aborting
Jan 21 10:23:41 Tower kernel: nvme nvme0: I/O 981 QID 6 timeout, aborting
Jan 21 10:23:50 Tower kernel: nvme nvme0: I/O 422 QID 5 timeout, aborting
Jan 21 10:23:50 Tower kernel: nvme nvme0: I/O 423 QID 5 timeout, aborting
Jan 21 10:24:11 Tower kernel: nvme nvme0: I/O 979 QID 6 timeout, reset controller
Jan 21 10:24:41 Tower kernel: nvme nvme0: I/O 7 QID 0 timeout, reset controller
Jan 21 10:25:24 Tower kernel: nvme nvme0: Device not ready; aborting reset
Jan 21 10:25:24 Tower kernel: nvme nvme0: Abort status: 0x7
### [PREVIOUS LINE REPEATED 4 TIMES] ###
Jan 21 10:25:46 Tower kernel: nvme nvme0: Device not ready; aborting reset
Jan 21 10:25:46 Tower kernel: nvme nvme0: Removing after probe failure status: -19
Jan 21 10:26:09 Tower kernel: nvme nvme0: Device not ready; aborting reset

 

Any suggestions to fix or troubleshoot? Or is it just something that occurs randomly?

So I let it run (moving some data around) and came home, could not access any dockers. So I stopped and restarted the array. Cache drive is missing (not a visible device). Restarted - same thing. Looks like the drive has disappeared ...

 

Attached diagnostics

tower-diagnostics-20210121-2011.zip

Try power cycling the server to see if it comes back.