I keep having issues with devices in the cache pool disappearing from the system. This is a new server build from December, so first I got a replacement M.2 nvme to see if that would fix it, but it did not and both the other SSDs dropped multiple times.
Next I had a replacement motherboard sent and just got that setup, and within 2 hours one of my SSDs just dropped out again...
Jan 24 19:14:55 Media ntpd[1957]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Jan 24 19:15:38 Media kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Jan 24 19:15:38 Media kernel: caller _nv000907rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs
Jan 24 19:39:53 Media kernel: nvme nvme0: I/O 514 QID 4 timeout, aborting
Jan 24 19:40:23 Media kernel: nvme nvme0: I/O 514 QID 4 timeout, reset controller
Jan 24 19:40:53 Media kernel: nvme nvme0: I/O 22 QID 0 timeout, reset controller
Jan 24 19:41:45 Media kernel: nvme nvme0: Device not ready; aborting reset
Jan 24 19:41:45 Media kernel: nvme nvme0: Abort status: 0x7
Jan 24 19:41:57 Media nginx: 2020/01/24 19:41:57 [error] 6086#6086: *6391 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.103, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.0.7", referrer: "http://192.168.0.7/Dashboard"
Jan 24 19:42:06 Media kernel: nvme nvme0: Device not ready; aborting reset
Jan 24 19:42:06 Media kernel: nvme nvme0: Removing after probe failure status: -19
Jan 24 19:42:12 Media nginx: 2020/01/24 19:42:12 [error] 6086#6086: *6872 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.103, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.0.7", referrer: "http://192.168.0.7/Dashboard"
Jan 24 19:42:27 Media nginx: 2020/01/24 19:42:27 [error] 6086#6086: *6229 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.103, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.0.7", referrer: "http://192.168.0.7/Dashboard"
Jan 24 19:42:27 Media kernel: nvme nvme0: Device not ready; aborting reset
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 104580456
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 97666840
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 96589704
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 97655216
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 136028264
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 97654272
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 97655440
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 104523792
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 96588568
Jan 24 19:42:27 Media kernel: print_req_error: I/O error, dev nvme0n1, sector 104555408
Jan 24 19:42:27 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:27 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:27 Media kernel: nvme nvme0: failed to set APST feature (-19)
Jan 24 19:42:27 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
### [PREVIOUS LINE REPEATED 2 TIMES] ###
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:27 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:27 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:27 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
### [PREVIOUS LINE REPEATED 2 TIMES] ###
Jan 24 19:42:27 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
### [PREVIOUS LINE REPEATED 47 TIMES] ###
Jan 24 19:42:32 Media kernel: btrfs_dev_stat_print_on_error: 1806 callbacks suppressed
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1759, rd 5, flush 53, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1760, rd 5, flush 53, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1761, rd 5, flush 53, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1761, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: btrfs_end_buffer_write_sync: 67 callbacks suppressed
Jan 24 19:42:32 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1762, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1763, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1764, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1765, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:32 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1766, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1767, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1768, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1769, rd 5, flush 54, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1769, rd 5, flush 55, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1770, rd 5, flush 55, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1771, rd 5, flush 55, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1772, rd 5, flush 55, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1772, rd 5, flush 56, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1773, rd 5, flush 56, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1774, rd 5, flush 56, corrupt 0, gen 0
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:37 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:37 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:39 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:39 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:41 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:41 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:42 Media kernel: BTRFS warning (device nvme0n1p1): lost page write due to IO error on /dev/nvme0n1p1
Jan 24 19:42:42 Media kernel: BTRFS error (device nvme0n1p1): error writing primary super block to device 3
Jan 24 19:42:42 Media kernel: btrfs_dev_stat_print_on_error: 42 callbacks suppressed
media-diagnostics-20200124-1948.zip