Problem:

The cache pool crashes resulting in unresponsive dockers.

There is little indication of errors in the unRAID Web GUI, the cache drive temps disappear on the main page (and the dockers aren't responsive).

The unRAID itself remains responsive throughout, although without cache drive features.

 

The problem has been consistent for awhile (3-4 months), but sporadic.  Often it'll crash overnight or when not being actively used.

It crashed last night and I rebooted this morning, but it crashed again after ~3 hours of uptime.

Diagnostic logs are attached of the recent crash, I have others available if they help.

 

Solution Attempts:

I had assumed that the previous cache drive (single Samsung 850) was failing, or a bad cable (after only 1 year of use).

I've removed the 850 and tossed that cable in favor of the current RAID1 2xP550 setup (with new cables, disks sdf and sdg).

 

Fix Common Problems Plugin

The only typical error shown is a Call Traces are found error (I haven't investigated this yet).

Running the test again while the cache pool is down displays a "Unable to write to cache" and "Unable to write to Docker Image" error.

 

A system reboot solves the problem, until it happens again.

I'm grasping at straws for other solutions, but coming up short.  Any suggestions or advice?

 

System Info:

Unraid Version: 6.3.2

M/B: ASUSTeK Computer INC. - P6X58D PREMIUM

CPU: Intel® Core™ i7 CPU 930 @ 2.80GHz

HVM: Enabled

IOMMU: Enabled

Cache: 256 kB, 1024 kB, 8192 kB

Memory: 18 GB (max. installable capacity 24 GB)

Network: eth0: 100 Mb/s, full duplex, mtu 1500  eth1: not connected

Kernel: Linux 4.9.10-unRAID x86_64

OpenSSL: 1.0.2k

 

Data Disks 1-8 are on a AOC-SAS2LP-MV8 card and do not seem to have trouble.

Parity and Data Disks 9-11 are on the onboard SATA2 controller and do not seem to have trouble.

Cache Disks are on the onboard SATA3 controller.

 

Previous cache disk was also on the SATA3 controller.  As I've changed the rest of the hardware, it appears to be this controller or a software/config issue I believe.

No VMs running, 8 dockers running.  Docker image, appdata, etc are all run off the cache pool as "Prefer".

 

Full diagnostic logs attached.

Below is an excerpt of the standard log of what appears to be the repeating error when this happens.

 

Mar 26 13:44:47 Vilhelm kernel: btrfs_dev_stat_print_on_error: 215 callbacks suppressed
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdf1 errs: wr 23, rd 11270, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdg1 errs: wr 18254, rd 11797, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: 65 callbacks suppressed
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 1090656
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 6, rd 9501, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdf1 errs: wr 23, rd 11271, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdg1 errs: wr 18254, rd 11798, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 3187808
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 6, rd 9502, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdf1 errs: wr 23, rd 11272, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdg1 errs: wr 18254, rd 11799, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 1090656
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 6, rd 9503, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: BTRFS error (device sdf1): bdev /dev/sdf1 errs: wr 23, rd 11273, flush 1, corrupt 0, gen 0
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 3187808
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 1090656
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 3187808
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 1090688
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 3187840
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 1090688
Mar 26 13:44:47 Vilhelm kernel: blk_update_request: I/O error, dev loop0, sector 3187840

diagnostics-20170326-1347.zip

You're using a Marvell controller for both SSDs, they can be problematic, first thing to try would be to move them to the onboard Intel SATA ports, even if they are sata2, it should fix your issues.

 

You should also run a scrub on the cache pool after that.

Edited March 26, 20179 yr by johnnie.black

Just to add some more info, both are having timeout errors and end up getting disabled, pretty sure it's a controller problem.
 

First one goes here:

Mar 26 13:27:49 Vilhelm kernel: ata8: softreset failed (1st FIS failed)
Mar 26 13:27:49 Vilhelm kernel: ata8: reset failed, giving up
Mar 26 13:27:49 Vilhelm kernel: ata8.00: disabled

And a couple of minutes later the other one:
 

Mar 26 13:29:20 Vilhelm kernel: ata7: softreset failed (1st FIS failed)
Mar 26 13:29:20 Vilhelm kernel: ata7: reset failed, giving up
Mar 26 13:29:20 Vilhelm kernel: ata7.00: disabled

 

Thank you.

I've switched the Cache drives over to the SATA2 onboard, and it's been running smooth for 24 hours now.

Don't forget to run a scrub, it's likely that there are a few errors.