The first thing I did when I had problems with Disk 1 (md1) was to run xfs_repair on the disk, ultimately requiring the -L switch.  It came back online and I took the opportunity to use unBalance to move everything from that disk to the other disks of the array.  It took about 50 hours and is complete.

 

After that I shut down the system, replaced the drive, fired up the system, allocated it, formatted it, and parity is rebuilding data.

 

What I'm not clear on is why both the Docker Service failed to start and none of the Virtual Machines are there anymore.

 

I looked at the libvirt.log and found this:

 

`2023-11-09 22:48:00.248+0000: 6006: error : catchXMLError:1015 : /etc/libvirt/qemu/Home Assistant.xml:1: Document is empty
(null)`

 

So I checked that path and found that there's nothing in /etc/libvirt (I get "/bin/ls: reading directory '.': Input/output error") if I list files.  There is however, a path to /etc/libvirt- which contains the qemu folder, etc.

 

Should I rename the hyphen folder to /etc/libvirt after removing /etc/libvirt?

 

What about the Docker Service?  Very confused at this point.

tower-diagnostics-20231109-1806.zip

Update, I just noticed that the 1TB cache pool is full (which isn't typically the case) and the /system/docker and /system/libvirt folders are located there as /system is marked as "prefer" on the cache settings.  I imagine the cache being full is causing this problem, but I'm not certain.

 

The bigger problem is that I'm unable to figure that out until the Parity check is done?  That would mean my services are dead in the water for another 9 hours.  Perhaps I'm not managing this system well, but 50+ hours to move critical data to healthy disks and then another 12 hours for parity check is a long time to not have critical services.

Nov  9 15:50:49 Tower kernel: loop: Write error at byte offset 6213271552, length 4096.
Nov  9 15:50:49 Tower kernel: I/O error, dev loop3, sector 12135296 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 0
Nov  9 15:50:49 Tower kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Nov  9 15:50:49 Tower kernel: loop: Write error at byte offset 5944836096, length 4096.

 

Docker image can't be used because the cache is completely full, you need to free up some space.