Veeam NFS backup causes .fuse_hidden files and permanently open files (mover fails)

[BlueBull]

Hello

I use Veeam backup to backup my (vSphere) VM's and workstations and subsequently copy those backups from my Veeam server's local storage to an unraid share through NFS. Every time such a backup copy process runs, some .fuse_hidden files are being created in the backup root directory. The backup copies complete succesfully. After that, even if I shutdown my Veeam server, the SHFS process keeps these .fuse_hidden files open on the array, along with all the backup files that were written to the array. I have cache enabled for this share and all the files are written to cache. The mover process then refuses to move these files to my spinning disks, neither the fuse nor the backup files, it skips them

A reboot solves it, the fuse_hidden files disappear and then the mover works again too but this is not a solution of course. I have already completely disabled docker and I'm not running any VM's. The Veeam server has been offline for about 4 hours now but the files remain open, at least according to the open files plugin

You can find a screenshot of what the files look like, what the open files look like, what the mover process output looks like and an anonymous diagnostics file attached. I have blacked out the filenames wherever they contained personal information, but all screenshots display both the .fuse_hidden files and the actual Veeam backup files

Any ideas what could be causing this and/or how I might resolve this? I find it very strange that these files remain open even when I shut down the Veeam server. This must be something on the array itself causing this, since I have also shut down every other device with access to my array and the files remain open

Cheers

Diagnostics.zip

Edited February 27, 2023 by BlueBull

[dlandon]

Are you running Unraid in VMWare?

[BlueBull]

10 hours ago, dlandon said:

Are you running Unraid in VMWare?

Correct. ESXi 8.0 with a passed-through PCIe SAS HBA and coupled to a vCenter server and an NVMe (well, a VMDK on an NVMe) cache drive. No linux boot manager or anything like that, these days you can boot directly off of the unraid USB drive without any workarounds, so it should be a fairly standard unraid install. I've thought about just disabling cache for that share but the problem is that if those files remain open, that will keep my drives spun up permanently. I've tried looking for a cause myself but can't make heads or tails of it since the files remain open even after there is no more connection over NFS or SMB for that matter

Edited March 3, 2023 by BlueBull

[dlandon]

Unraid is not supported when running on a Hypervisor.

 

You should look into the backup software and see why the files are not being closed.  Depending on the server to time out and close the files is not proper file management.