So recently, I had to power down the server because something was hung to the point where the GUI and my ssh session were very slow to respond. After it came back up, I'm seeing a disk read error in syslog, most of my docker containers can't start (two of them claim to be started for some reason ¯\_(ツ)_/¯) because of a 403 not found error (because it can't see the disk apparently), and VM's page is empty.  Also array disk 1 reports to be fine, but is also showing up under unassigned devices.  If I  cd to /mnt/disk1 and run ls -lharts I get this: "/bin/ls: cannot open directory '.': Input/output error".  I immediately get worried because array disk2 was already being emulated and I'm waiting for a replacement to show up.

 

 

At that point, I restarted in safe mode and started array in maintenance mode to run xfs_repair on suspected problem disks and all apparently went fine.  I could manually mount disk1 to a created mount point and was able to pull the data off just to be safe. I restarted the server and it's all still borked.  Would it be best to do a new config at this point?  I don't want to lose data, but I do have all mission critical things backed up.  I still don't want to have to recreate all my media if there's any way around it.  My diag file is attached.  Thanks in advance for the help!

tower-diagnostics-20230825-1119.zip

Edited August 25, 20232 yr by Mobius71
typos

All your devices dropped offline, cannot see the VM XML, are you passing though any PCIe device to the VM?

 

Reboot and post new diags after array start, don't start any VMs.

I'm passing a graphics card to the vm and maybe a usb controller I think. VM is set to start automatically.  If I just reboot into headless Unraid, the boot freezes after "loading bzroot....ok" and will not progress further. If I reboot into Unraid GUI mode it will actually boot.  New Diag file attached.

tower-diagnostics-20230825-1436.zip

Also seeing high mem and cpu usage:

 

 

I have those 4 cores locked to the OS so I can use the rest elsewhere.  Also the log is at 100%...so looks like syslog is going crazy and eating up resources.

 

 

Gonna shut things down for now.

Edited August 25, 20232 yr by Mobius71

I couldn't access the vm config to turn off auto start until I killed rsyslogd.  Then the GUI began behaving much more normally.  I turned off auto start for VM and rebooted the server.  New diagnostics file attached.   I'm still having most or all of the same issues as before.  Log immediately gets filled with "Tower kernel: md: disk0 read error"

tower-diagnostics-20230825-1801.zip

Edited August 25, 20232 yr by Mobius71

There's still no XML for the VM in the diags, copy/paste here from the VMs page.

Had to reboot several times to get the VMs page to show up but here it is

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>6fe4dea8-2e66-4620-f656-86e038845777</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>25165824</memory>
  <currentMemory unit='KiB'>25165824</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='9'/>
    <vcpupin vcpu='1' cpuset='21'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='22'/>
    <vcpupin vcpu='4' cpuset='11'/>
    <vcpupin vcpu='5' cpuset='23'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/6fe4dea8-2e66-4620-f656-86e038845777_VARS-pure-efi.fd</nvram>
    <smbios mode='host'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='3' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/S1SMNSAFC04456N/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-Samsung_SSD_870_EVO_1TB_S625NJ0R283000Z'/>
      <target dev='hdd' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:8a:e4:30'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/MSI_2070_Super_Ventus_OC_vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

Also wanted to note, that initially, when restarted, my shares show up, but after a couple of minutes they all disappear and Unraid says I have zero shares.

19 minutes ago, Mobius71 said:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/MSI_2070_Super_Ventus_OC_vbios.rom'/>

You were passing through device 08:00.0 (08:00.1, 08:00.2 and 08:00.3) to the VM, before this was the GPU but now it's a SATA controller, remove these entries if you are no longer using the GPU or edit them all to 09:00.X if you still are.

Oh man, I don't know how I didn't catch that.  I'll sort that out when I can get back in front of all that later this afternoon and let you know how it goes.  Thanks JorgeB!

Yep, that fixed it!  Thanks again JorgeB!