I just realized my cache disk is mounted read-only. Not sure what happened, but I need to know the proper procedure here. I have not rebooted and I'm attaching my diagnostics output. What's the right steps to troubleshoot/repair the problem here?

storage-diagnostics-20190516-1409.zip

There was an error with the cache drive, btrfs went read-only to protect the filesystem:

 

May 16 09:36:47 Storage kernel: sd 1:0:0:0: [sdb] tag#31 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06
May 16 09:36:47 Storage kernel: sd 1:0:0:0: [sdb] tag#31 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
May 16 09:36:47 Storage kernel: print_req_error: I/O error, dev sdb, sector 0
May 16 09:36:47 Storage kernel: BTRFS error (device dm-14): bdev /dev/mapper/sdb1 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
May 16 09:36:47 Storage kernel: BTRFS warning (device dm-14): chunk 1048576 missing 1 devices, max tolerance is 0 for writeable mount

 

A reboot should fix it for now, you might want to take the opportunity to replace cables to rule them out.

 

Also some errors with two other disks, likely also cable related:

 

May 14 11:05:34 Storage kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 14 11:05:34 Storage kernel: ata1.00: failed command: FLUSH CACHE EXT
May 14 11:05:34 Storage kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 22
May 14 11:05:34 Storage kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
May 14 11:05:34 Storage kernel: ata1.00: status: { DRDY }
May 14 11:05:34 Storage kernel: ata1: hard resetting link
May 14 11:05:39 Storage kernel: ata1: link is slow to respond, please be patient (ready=0)
May 14 11:05:43 Storage kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
May 14 11:05:43 Storage kernel: ata1.00: configured for UDMA/133
May 14 11:05:43 Storage kernel: ata1.00: retrying FLUSH 0xea Emask 0x4
May 14 11:05:43 Storage kernel: ata1: EH complete

 

May 16 04:07:10 Storage kernel: ata6.00: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
May 16 04:07:10 Storage kernel: ata6.00: irq_stat 0x08000000, interface fatal error
May 16 04:07:10 Storage kernel: ata6: SError: { UnrecovData Handshk }
May 16 04:07:10 Storage kernel: ata6.00: failed command: WRITE DMA EXT
May 16 04:07:10 Storage kernel: ata6.00: cmd 35/00:40:88:78:8a/00:05:37:00:00/e0 tag 5 dma 688128 out
May 16 04:07:10 Storage kernel:         res 50/00:00:87:78:8a/00:00:37:00:00/e0 Emask 0x10 (ATA bus error)
May 16 04:07:10 Storage kernel: ata6.00: status: { DRDY }
May 16 04:07:10 Storage kernel: ata6: hard resetting link
May 16 04:07:10 Storage kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
May 16 04:07:10 Storage kernel: ata6.00: configured for UDMA/133
May 16 04:07:10 Storage kernel: ata6: EH complete
May 16 04:08:54 Storage kernel: ata6.00: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
May 16 04:08:54 Storage kernel: ata6.00: irq_stat 0x08000000, interface fatal error
May 16 04:08:54 Storage kernel: ata6: SError: { UnrecovData Handshk }
May 16 04:08:54 Storage kernel: ata6.00: failed command: WRITE DMA EXT
May 16 04:08:54 Storage kernel: ata6.00: cmd 35/00:40:18:d7:dd/00:05:38:00:00/e0 tag 23 dma 688128 out
May 16 04:08:54 Storage kernel:         res 50/00:00:17:d7:dd/00:00:38:00:00/e0 Emask 0x10 (ATA bus error)
May 16 04:08:54 Storage kernel: ata6.00: status: { DRDY }
May 16 04:08:54 Storage kernel: ata6: hard resetting link
May 16 04:08:54 Storage kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
May 16 04:08:54 Storage kernel: ata6.00: configured for UDMA/133
May 16 04:08:54 Storage kernel: ata6: EH complete

 

Reboot did it, thanks. I'll get new cables and swap them out.