A couple of my machines are paused and btrfs rebalance does nothing.  I ran btrfs balance start -dusage=75 /mnt/cache but showed 0 to relocate.  I have attached my diag, please help.  Also tried dusage=50, 25, ect.

tower-diagnostics-20190128-0615.zip

Edited January 28, 20197 yr by jordanmw

Any ideas? anyone?  Really don't want to have to rebuild those machines- especially if I don't know why they went into a paused state and couldn't resume.  It does look like it had something to do with the backup that was scheduled last night but not seeing issues with the space on that drive either.

After looking into things further- it appears that it was my data disk that needed the re-balance.  Have that running currently- will mark solved if it works.  Thought this was a non-issue on versions past 6.4?

It's not a balance problem, there's plenty on unallocated space.

So, the re balance that is running won't fix the issue? What else should I be looking at?

Cache has plenty of space, are you sure the pausing VMs are only using cache? Some of the VMs have a vdisk on other devices, e.g.:

 

file=/mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0KA58219L/Windows 10-Lance/vdisk2.img

And these are full:

 

Filesystem      Size  Used Avail Use% Mounted on

/dev/sdg1       932G  932G     0 100% /mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0K914540W
/dev/sdc1       932G  932G     0 100% /mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0KA58219L

 

That's really messed up since I deleted the entire partition and formatted before attaching them to the other vms.

What I guess I am not understanding is why file=/mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0KA58219L/Windows 10-Lance/vdisk2.img exists at all?  I deleted the partition on that SSD and formatted it with NTFS but apparently it still contains that file.  I don't understand how I can delete the partition- format the drive which shows completely empty but still contain files?!?!  Somebody help me understand this- I have done this process 3 times with thte same result.

Edit the VM and delete the second vdisk.

I have done that several times- but it doesn't seem to respect it.  I am trying to get some more info and clear them with fdisk now.  Is there some reliable way to pre-clear those disks?

If they are on a controller with trim support use:

 

blkdiscard /dev/sdX

 

Tried that on both drive but it's still doing the same thing.  I format the drive with ntfs- and then pass it through to the VM- initialize the disk in windows as GPT and format- then start copying data.  Within a few minutes, the VM pauses.  

tower-diagnostics-20190129-1327.zip

This is really getting frustrating and I am not sure how I am having such major issues with a seemingly simple change.  Any other ideas?  

21 hours ago, johnnie.black said:

Cache has plenty of space, are you sure the pausing VMs are only using cache? Some of the VMs have a vdisk on other devices, e.g.:

file=/mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0KA58219L/Windows 10-Lance/vdisk2.img

 

 

How can I find where this is?  It is not setup to be configured to any VM since I removed it.  It seems like this is still the issue- a partition that I can't remove that contains that img file but is invisible to fdisk.... never dealt with this before...

1 hour ago, jordanmw said:

I format the drive with ntfs- and then pass it through to the VM- initialize the disk in windows as GPT and format- then start copying data.  Within a few minutes, the VM pauses.  

Do you unmount the disks in UD first? You can't pass through a disk and mount it with UD at the same time, or it will cause trouble.

yes, I only mounted them to show the free space- they are showing in UD as unmounted.

When mounted- they show almost empty:

/dev/sdc1       932G   94M  932G   1% /mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0KA58219L
/dev/sdg1       932G   94M  932G   1% /mnt/disks/Samsung_SSD_860_EVO_1TB_S3Z8NB0K914540W

 

but I am assuming that is because it isn't seeing the partition that exists with that .img file or something?!?!? this is maddening- it's like those drives are cursed ever since I removed them from the other 2 VMs... Can someone at unraid come to my rescue?  Nobody is chiming in from support?!?

shown as block device:

/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/sda1: LABEL="UNRAID" UUID="2732-64F5" TYPE="vfat"
/dev/sdb1: UUID="3716ef75-40d8-40b2-a482-028468185c25" UUID_SUB="4bc1197f-d6c5-45f6-a2b8-ccc2df3c4cb5" TYPE="btrfs" PARTUUID="c68bc3f5-3682-428f-98e9-048346eab6d3"
/dev/sdd1: UUID="02cdd474-7618-423b-bbb4-953a286e592a" UUID_SUB="a7508196-d8f0-46bf-8434-7d78bbf13e40" TYPE="btrfs" PARTUUID="1de014c2-01"
/dev/sde1: UUID="02cdd474-7618-423b-bbb4-953a286e592a" UUID_SUB="31c92a40-ce11-48a2-a00d-0bfa5edb9ab6" TYPE="btrfs"
/dev/md1: UUID="3716ef75-40d8-40b2-a482-028468185c25" UUID_SUB="4bc1197f-d6c5-45f6-a2b8-ccc2df3c4cb5" TYPE="btrfs"
/dev/md2: UUID="f1301ac6-d1a6-4302-a658-de3c15084523" TYPE="xfs"
/dev/loop2: UUID="ad236372-a6dc-4b35-bc0c-17f5af3bad0d" UUID_SUB="155ff16d-8680-473d-9909-f57cdb69ccb2" TYPE="btrfs"
/dev/loop3: UUID="ed7e0976-c1b9-49de-a97c-695687f6f774" UUID_SUB="5aeb2b60-9bb9-49f1-b37e-1f24f82c7a93" TYPE="btrfs"
/dev/sdc1: UUID="2B9D0823066E6D1A" TYPE="ntfs" PTTYPE="dos" PARTUUID="a5787e3d-01"
/dev/sdg1: UUID="01D6F711686C8163" TYPE="ntfs" PTTYPE="dos" PARTUUID="6826c3da-01"

 

So the last 2 are my drives- showing that they are formatted...

Still stuck in the same loop- no matter how many times I kill the partition and reformat, it runs for a few minutes of copying files after formatting the disk and then pauses the machine.  Can someone please look at this latest diagnostic?

tower-diagnostics-20190129-1931.zip

I have even re-created the VM and still get the same result- I can initialize and format the drive but the machine pauses as I copy data to it- every time.

It might have something to do with he way you're doing it, but no idea what, since I don't pass-trough disks to VMs, maybe try posting on the VM forum.

Actually- after looking at some other posts it occured to me that I am running out of memory during the file copy.  Is there some way to prevent it from running the system out of memory when doing large file copies?

2 minutes ago, jordanmw said:

it occured to me that I am running out of memory during the file copy.

Never heard of this before.

apparently you have- being quite useful in this thread- turbo writes?