Hello, I've spent all DAY on this! 

 

So my Cache pool is filling up slowly. The only thing that I have on it are appdata and a Windows 10 VM. The VM itself is no larger than 322GB and appdata is 1GB.

 

My Cache Pool is 465GB, and over the last day.... my cache pool is filling up slowly but surely and there is nothing new written to it. 

 

When I run several commands from PUTTY it even tells me my VM is 322GB and appdata is 1GB. I have searched high and low for whatever files are causing my cache drive to fill up. 

 

Mover doesn't help, nothing helps!

 

AHHHH Help! (Lol) Any help would be greatly appreciated. As of right now my Cache drive is 100GB over capacity and I cannot for the life of me figure out why.

You should post diagnostics (Tools->Diagnostics) if you want anything other than wild guesses.

4 hours ago, Rikumo1978 said:

My Cache Pool is 465GB

 

Looks like a pool of 2 different size devices, if yes, available space reported will be incorrect, max usable capacity is the size of the smaller device.

4 hours ago, johnnie.black said:

 

Looks like a pool of 2 different size devices, if yes, available space reported will be incorrect, max usable capacity is the size of the smaller device.

Nope, two 500GB SSD's in RAID1, Same make and model. Same size and all. Which diagnostics from Tools should I post?

3 minutes ago, Rikumo1978 said:

Which diagnostics from Tools should I post?

 

Tools -> Diagnostics

 

Upload the zip

Edited May 29, 20179 yr by johnnie.black

Just now, johnnie.black said:

 

Tools -> Diagnostics

I've done this. I'm left with a TON of log files basically...am I to post every single one of these? Which ones would people like to see to help me diagnose this issue?

1 minute ago, Rikumo1978 said:

am I to post every single one of these?

 

Upload the zip, single file.

Ok, so before I uploaded this, I double checked ALL the files on my Cache drive, as of right now I only have only 1 directory and 1 file inside it. My VM Image. I will even upload a screenshot of what the command line is telling me about it. 

 

Literally only directory being shown is VM directory and that is being shown as 322GB, which is the size this VM has been since the 22nd. Nothing has changed

linus-diagnostics-20170528-2209.zip

Appdata is configured as cache yes, and the mover is moving it to the array, is this on purpose? Usually should be configured as cache "prefer".

 

You're also getting OOM errors, but I'm not seeing a reason for the cache to report out of space, unless you deleted some files since, what was happening when the server was reporting it?

 

Quote

May 28 21:29:48 LINUS shfs/user: cache disk full

 

Yes this is correct, I just moved Appdata so I would have less culprits for this issue. Were these OOM errors from UNRAID itself or the VM?

 

 

Believe I may have spotted an error, I created another VM just out of curiosity, and this new VM isn't owned by ROOT, it's owned by Nobody, and its in the Users group. Could this be the reason my main VM is giving me so much trouble? It's owned by ROOT...

 

Here is the screenshot of the new VM I created for comparison. Thanks for the patience everyone I consider myself a moderate Linux user, definitely better than I was last year lol. I'm doing the best with the knowledge I have. Usually I can figure this stuff out without reaching out to the experts but this time it's just stumping me!

19 minutes ago, johnnie.black said:

Appdata is configured as cache yes, and the mover is moving it to the array, is this on purpose? Usually should be configured as cache "prefer".

 

You're also getting OOM errors, but I'm not seeing a reason for the cache to report out of space, unless you deleted some files since, what was happening when the server was reporting it?

 

 

Thats just it, the server was never reporting a FULL cache (unless I started a docker) the only real issue I was having was there only being appr. 300GB of files ON that cache drive, so theres no reason for it to be reading at 415GB currently. As of this morning it's still telling me its almost 100GB over known capacity.

Size of a sparse vdisk with btrfs can be off when reported by ls/du, if you're not already see here to make the vdisk release all unused space and not keep constantly growing:

 

 

This sounds EXACTLY like what the issue is. Johnnie.Black I am going to go and do all this now. Wish me luck! This forum is a LIFE SAVER. It didnt matter what order and what keywords I used I couldn't find this solution haha. Will post back with results.

This instantly freed up 50GB on my cache drive. It's still reporting 50GB larger than normal but this is a definite start!! I love this forum. Will continue to look into what else is causing it to report higher than normal. 

 

Thanks Johnny.Black

I did read this topics posts, but didn't get anything to happen.

 

My cache pool of two different sizes ssd's is filling up. I should have over 100Gb more free space when looking at MC or cache share on windows. 

 

Here's my diagnostics. Hope someone could help me of this, please...

unraid-diagnostics-20190226-2008.zip

One disk is 250G the other is 120G the default for a cache pool is raid 1 (mirror) which means that you only have max 120g available. It's a known limitation of btrfs to misreporting free space when using mismatched drives

Sent via telekinesis

32 minutes ago, JQNE said:

I did read this topics posts, but didn't get anything to happen.

According to your diags you didn't change the controller to scsi and added unmap, just follow the instructions in the link above:

https://forums.unraid.net/topic/51703-vm-faq/?do=findComment&comment=557606

 

2 hours ago, Squid said:

One disk is 250G the other is 120G the default for a cache pool is raid 1 (mirror) which means that you only have max 120g available. It's a known limitation of btrfs to misreporting free space when using mismatched drives

Sent via telekinesis
 

I know what the default is, I don't  have that situation. I have them in somekind raid0 setup...

2 hours ago, johnnie.black said:

According to your diags you didn't change the controller to scsi and added unmap, just follow the instructions in the link above:

https://forums.unraid.net/topic/51703-vm-faq/?do=findComment&comment=557606

 

Thanks I'll look that through!

On 2/26/2019 at 8:46 PM, johnnie.black said:

According to your diags you didn't change the controller to scsi and added unmap, just follow the instructions in the link above:

https://forums.unraid.net/topic/51703-vm-faq/?do=findComment&comment=557606

 

I posted a problem on that forum:

 

15 minutes ago, JQNE said:

I posted a problem on that forum:

You shouldn't have, that thread is supposed to be for FAQ entries only.

23 minutes ago, JQNE said:

a problem

As for the problem, post the VM XML, easier to analyze that from the diags.

56 minutes ago, johnnie.black said:

You shouldn't have, that thread is supposed to be for FAQ entries only.

Oops. I deleted that...

48 minutes ago, johnnie.black said:

As for the problem, post the VM XML, easier to analyze that from the diags.

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='4'>
  <name>Joonas Windows 10</name>
  <uuid>x</uuid>
  <description>Gaming</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/eaf7bebc-48c7-bb18-431a-8b07bd737733_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <source file='/mnt/cache/vdisk_ssd/Joonas Windows 10/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='scsi'/>
      <boot order='1'/>
      <alias name='scsi0-0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='x'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-Joonas Windows 10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/Setup_files/GTX_file/asus1070.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>
 

That looks fine, please post the diags again here since I though I saw an extra character there.