Hi all,

 

Looking for some expertise here. I have VMs running on some SSD cache drives in BTRFS RAID 1.  Yes, I am backing up, but If I lose a drive should I be able to expect it to run ok until I get the drive replaced? Or are there any limitations in BTFRS' file system that would cause problems until rebuild?

 

Just read a few conflicting things that made me want to clarify. Thanks!

It should run OK with a single drive, make sure shares are not set to nocow.

2 hours ago, JorgeB said:

It should run OK with a single drive, make sure shares are not set to nocow.

Thanks, @JorgeB. It all runs very fast and I am using the built-in domains share set to prefer cache with nocow. But if one of the cache drives fail, it'll continue to run with no data loss until I can get the drive replaced (assuming the second drive doesn't fail of course)? I'm not familiar with the underlying tech of btrfs and curious to learn more. Like does the nocow setting actually pause and write changes in chunks? How does it maintain a redundant copy of a file that's constantly changing like a vm disk?

Thanks!

35 minutes ago, Alyred said:

it'll continue to run with no data loss until I can get the drive replaced

Yes, as long as you are using a redundant pool, and the default is redundant (raid1).

 

36 minutes ago, Alyred said:

with nocow

Nocow should be set to auto, if it's set to no, there won't be checksums and if a device drops and then reconnects it might result in data corruption, since btrfs has no way of knowing which device has the correct data, more info here.

I'll check to make sure it's set to auto. With it set that way, will it just avoid cow on certain types of files, such as vm disks? How are vm disks handled to ensure they're mirrored in case of failure? Should I switch to a different cache filesystem for vm disks?

Read your other post, thanks for the information on checking for errors. I do have it set up to scrub and rebalance monthly and have seen no errors there (though I am seeing an interesting rebalance note that I'll post about later), but good to know how I can confirm the states.

10 minutes ago, Alyred said:

With it set that way, will it just avoid cow on certain types of files, such as vm disks?

With nocow it will disable cow for any data written to that share.

 

12 minutes ago, Alyred said:

How are vm disks handled to ensure they're mirrored in case of failure?

Like any other file, any blocks are written to both devices.

 

12 minutes ago, Alyred said:

Should I switch to a different cache filesystem for vm disks?

xfs or btrs nocow might perform a little better, but xfs is single device only and btrfs nocow as mentioned should be avoided, note that a mirror is for redundancy, you still need backups of anything important.

51 minutes ago, JorgeB said:

With nocow it will disable cow for any data written to that share.

Yes, so with nocow set to "auto", it will determine if nocow should be set for specific files, like the VM disks I'm asking about? And other files on the same share would work as normal using cow? Does data integrity suffer in a block write operation if a file is set to nocow? Maybe I'm confused by the terminology, where if copy-on-write is disabled via nocow it doesn't seem that things would be written in a timely enough fashion to keep both copies in sync? Or does nocow just make writes happen in a different method?

 

And yes, as I mentioned, I am backing up, I'm more concerned about downtime if I have failures. They're not critical business VMs but would take annoying amounts of time to restore, so I'd prefer to keep them running with redundancy and get new drives in for replacement rather than doing restores if there is a hardware failure. I understand RAID is about uptime, not backup.

Auto in this case means yes, it's enabled for everything.

4 hours ago, JorgeB said:

Auto in this case means yes, it's enabled for everything.

Ohh, I think I see part of my confusion. Your original reply said to make sure the share was NOT set to nocow. Somehow I missed the "not" in your post, so I wasn't following. It was early for me, sorry for sounding dense and thank you for the help! I'll make sure nocow is set to auto on that share.

So I took a look at the "domains" default share on my Unraid server and the setting was "no" on COW.  I don't believe I updated that myself, so should I change it? The recommended settings are to have it "off":

 

If I do change that to "Auto", do I need to adjust the VM disk files already in the share? UnRaid's info there indicates that the nocow attribute is set per file and would have been set to "no" on the vdisk images themselves. If they are set to "Auto" is there a problem, or just a performance impact (and how much of an impact)?

 

Trying to get my head wrapped around this stuff better.

You'd need to move the data, re-create the share with cow=auto and move the data back, VMs must be off during that.

Yup, was afraid of that. Read another post discussing it over the last hour as well from back in May or so, about switching that default off and possible improvements in 6.11. Well... guess I'll get started.  Thanks for the help.

Ok, so I just created a new share and made sure COW was set to "Auto" using the same cache disks (My NVMe array). From the console I did a cp --reflink=never from the one share under /mnt/user/domains to the new share, /mnt/user/vdisk/<vm name>. Reading what I could find this should re-enable copy-on-write, does anyone know of a way I can manually verify?

 

The machines do seem to be running normally, though it's annoying that the one that I'm passing two physical network cards to through IOMMU has to have the cards re-detected each and every time, and then re-configured as if I had removed the old card and installed a new one...

You can check with:

lsattr /path/to/file

If the output is

---------------C------

nocow is set, if there's no C it's not set.

6 hours ago, JorgeB said:

You can check with:

lsattr /path/to/file

If the output is

---------------C------

nocow is set, if there's no C it's not set.

 

Excellent, thank you! I'm showing all dashes so I think I'm good! As a note for anyone else, I had to read it directly from the directory it is currently in, trying to read it from the /mnt/user share gave me an "Operation not supported While reading flags on <filename>" error.

 

Appreciate your assistance and knowledge to get this worked out, and hope that it helps some others!

 

Was there any news with 6.11 on getting a better solution for this set up (I think it was being discussed as an LVM mode)? I did go ahead and upgrade to 6.11 last night as well since I had the array and VMs all down anyway...

Edited September 29, 20223 yr by Alyred

3 minutes ago, Alyred said:

trying to read it from the /mnt/user share gave me an "Operation not supported While reading flags on <filename>" error.

Yeah, forgot to mention, you'd need to use the disk share path, e.g. /mnt/cache/file

 

4 minutes ago, Alyred said:

I think it was being discussed as an LVM mode

I believe there are plans for that but it' won't be for 6.11, maybe 6.12 or 6.13

1 minute ago, JorgeB said:

I believe there are plans for that but it' won't be for 6.11, maybe 6.12 or 6.13

 

Hopefully to use it, it won't require too much infrastructure rebuild... Definitely going to hold off on upgrading those NVMes until I find out if I have to rebuild the entire cache pool...