I was running balance for the first time in ages last night and I think it crashed/went wrong as my dockers stopped working and I woke up to a FCP 'Unable to write to cache' error.  I've tried rebooting but the error is still there and I'm not sure how to fix.  Help please

 

Diagnostics attached from before the reboot

 

highlander-diagnostics-20191102-0740.zip

Syslog is flooded with unrelated errors and because of that there several hours are missing, but there are hardware issues with one of the cache devices:

 

Nov  2 05:40:14 Highlander kernel: BTRFS error (device sde1): bdev /dev/sdc1 errs: wr 238, rd 20533, flush 1, corrupt 0, gen 0

 

Post clean diags after a reboot to see current state, also see here for better pool monitoring.

Thanks for having a look - new diags attached.

 

 

highlander-diagnostics-20191102-0948.zip

Cache2 is dropping offline:

 

Nov  2 08:14:41 Highlander kernel: ata1: hard resetting link
Nov  2 08:14:51 Highlander kernel: ata1: softreset failed (device not ready)
Nov  2 08:14:51 Highlander kernel: ata1: hard resetting link
Nov  2 08:15:01 Highlander kernel: ata1: softreset failed (device not ready)
Nov  2 08:15:01 Highlander kernel: ata1: hard resetting link
Nov  2 08:15:12 Highlander kernel: ata1: link is slow to respond, please be patient (ready=0)
Nov  2 08:15:36 Highlander kernel: ata1: softreset failed (device not ready)
Nov  2 08:15:36 Highlander kernel: ata1: limiting SATA link speed to 3.0 Gbps
Nov  2 08:15:36 Highlander kernel: ata1: hard resetting link
Nov  2 08:15:41 Highlander kernel: ata1: softreset failed (device not ready)
Nov  2 08:15:41 Highlander kernel: ata1: reset failed, giving up
Nov  2 08:15:41 Highlander kernel: ata1.00: disabled

 

Replace cables and see link above for fixing pool.

1 hour ago, johnnie.black said:

 

Replace cables and see link above for fixing pool.

Replacing cables did the trick.  Maybe one of my old cables came lose, not sure how but thanks anyway.

 

I ran btrfs dev stats /mnt/cache and it kicked up no errors this time.  I'm going to add the script to monitor going forwards.  Are there any other checks I should run?

Like mentioned on the FAQ entry you should run a scrub and confirm there are no uncorrectable errors.

42 minutes ago, DZMM said:

Maybe one of my old cables came lose, not sure

It happens. They seem to be designed to come loose.

hmm same problem occurred again.  I've changed cables again and run scrub with no errors coming up, so no idea what the problem is