I added a second cache drive to my array.  The VMs froze and the system became unresponsive.  I rebooted the server to restore access.  The VMs are no longer accessible upon reboot.  I tried removing the new cache drive but that resulted in the original drive showing unmountable.  I reattached the new cache drive and rebooted and this is what shows.  I seem to have lost all my VM images.

 

 

btrfs filesystem show:

Label: none  uuid: bf7fd42c-678e-4e5f-9d70-ee7d17ec4738

Total devices 2 FS bytes used 243.77GiB

devid    1 size 465.76GiB used 464.76GiB path /dev/sdk1

devid    2 size 894.25GiB used 246.74GiB path /dev/sde1

 

btrfs filesystem df:

Data, RAID1: total=246.74GiB, used=85.80GiB

Data, single: total=217.01GiB, used=157.74GiB

System, single: total=4.00MiB, used=112.00KiB

Metadata, single: total=1.01GiB, used=231.14MiB

GlobalReserve, single: total=80.00MiB, used=0.00B

 

edit -

 

I removed the cache drives from the configuration and mounted the original 500G drive using unassigned drives plugin.  I was able to copy off one vdisk image file but others are failing

 

root@BriansTower:/mnt/disks/Samsung_SSD_850_EVO_500GB_S21HNXAG484067F/VMs/mc2# cp vdisk1.img  /mnt/user/Storage/VMBackups1/mc2

cp: error reading 'vdisk1.img': Input/output error

 

syslog has the following

 

Jan  2 02:37:39 BriansTower kernel: scsi 5:0:0:0: Direct-Access    ATA      Samsung SSD 850  1B6Q PQ: 0 ANSI: 5

Jan  2 02:37:39 BriansTower kernel: sd 5:0:0:0: [sdk] 976773168 512-byte logical blocks: (500 GB/466 GiB)

Jan  2 02:37:39 BriansTower kernel: sd 5:0:0:0: [sdk] Write Protect is off

Jan  2 02:37:39 BriansTower kernel: sd 5:0:0:0: [sdk] Mode Sense: 00 3a 00 00

Jan  2 02:37:39 BriansTower kernel: sd 5:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Jan  2 02:37:39 BriansTower kernel: sd 5:0:0:0: Attached scsi generic sg4 type 0

Jan  2 02:37:39 BriansTower kernel: sdk: sdk1

Jan  2 02:37:39 BriansTower kernel: sd 5:0:0:0: [sdk] Attached SCSI disk

Jan  2 02:42:08 BriansTower kernel: btrfs_dev_stat_print_on_error: 67 callbacks suppressed

Jan  2 02:42:08 BriansTower kernel: BTRFS error (device sde1): bdev /dev/sdk1 errs: wr 53, rd 25, flush 0, corrupt 0, gen 0

 

 

I thought all that would happen when I added the disk to the cache pool is it would mirror the existing disk image.  I did notice that my available storage on the original drive went up to 700G with the addition of the new 960g drive.  The only image that I really would like to restore is the mc2 one above.  Am I out of luck?

 

-edit

 

I rebooted the system and added the two cache disks back into the unraid config.  This seems to have gotten past the unmounted state and kicks off the balance process.  It gets about 30% in and

 

btrfs balance status:

No balance found on '/mnt/cache'

 

I've attached the latest syslog

 

brianstower-diagnostics-20170102-0043.zip

brianstower-diagnostics-20170102-0313.zip

You are getting constant ATA errors on the Samsung SSD, replace the SATA cable (power also if possible just to rule it out), then run a balance and a scrub.

That was it.  I switched the sata cable and re-seated the power connectors.  Balance completed and my VMs start without hanging up.  Thank you!