silverkin

Logfile_virtiofs.CSV I’ve dug deeper and tried absolutely everything related to virtiofsd and the virtio parameter configuration in the XML from xattr, through cache mode=‘always’ or ‘none’, all the way to thread-pool-size, etc. and I still haven’t managed to solve this problem. I even added a special parameter to virtiofs on Windows so that it recognizes the partition type as NTFS rather than virtiofs. I’m bypassing FUSE, for testing I’ve started using the share mounted directly at /mnt/cache. Here’s the issue: I have a file called database.pak, and if I duplicate it on the share, Backblaze sees it but then skips it, so I never see it uploaded on their website. However, if the same file resides locally on the VM’s vdisk and I do the exact same copy, the problem doesn’t occur. I also tried copying the file from the VM straight onto the share, but in that case it’s still ignored. For deeper diagnostics I ran Procmon, and in the log you can spot differences: for one, reading the file on the share ends with an “END OF FILE” result, which doesn’t happen with the local version; and there’s also a discrepancy in the fileAttributes flag (ANCI). I’m attaching a fragment of the Procmon log here hopefully someone will be able to pinpoint what’s going wrong…

Managed to fix that issue by: replacing args="-noleaf" with args="-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name backups-prune -o -name .sync -prune -o -print" and to show the difference on filecounting, changed local thisDirCount=$(find $dir_to_scan -type f $depth_arg | wc -l) to local thisDirCount=$(find $dir_to_scan $args $depth_arg | wc -l) So in my opinion, those options are not working correctly without directly changing the script, as tried to add those parameters to option variable or args variable, but in my case those options were never included in the final find command. I know this is a temporary solution and kind of a hard-link one, @dlandon could you take a look at it? :)

Hello, I've tried to use Backblaze Personal Backup along with virtiofs shares, but I do have an issue with some files that are not being uploaded to Backblaze at all. Seeing in the logs (IRC bztransmit one): DID NOT find ChunkId: 00000000002ba2fa in hashTable, keeping oldValue=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_20080101020304, fileName=Z:\backups\personal\project_X\database.pak and in file: bz_done_internal_consistency.dat: 20250508144254 - NotEnoughChunkRecordsSeen - AfterBzdoneLargeFileAnalysis: totNumChunk=409, highestChunkSeqSeen=407, XYXBXXX_FILE_NAME: Z:\backups\personal\project_X\database.pakXYXBXXX_FULL_LINE: unknown This example weight 4GB, but there are plenty of other files that are 256MB's each, and they are also not getting uploaded. It doesnt matter when I'll try to rescan once again the shares, or wait a week to let it run bz_tofix action in order to try to "fix them" and upload them again. Tested same files directly under barebone Windows machine and those were successfully uploaded. Disabled writeback cache on VM, set cache to none on all of the shares, but it didn't help. My backup is like 4TB's, I am not trying to backup linux ISO's, as I don't want to waste space for Backblaze, but I do need to backup my Unraid along with other important files to me, as currently it is able to backup like 99% of the files (speaking of number of files not the size), and it's not enough to me, as for the backup I would like to have it done right. Did anyone had similar issues with that?

Well, thank you @Alex R. Berg for your contribution, but I have done it already, few days ago just copied your previously posted .cfg, replaced the shares with applicable ones, and replaced the exclusions, and still no result - number of files are not getting changed, and the find command is the same like with default one.

No, it is not helpful in my situation. Like I said, it doesn't matter what kind of folder (not share) I am gonna exclude, the number on the filecount is not going to get lowered, also the time between building the 'cache' with each depth is the same. I would love to verify this with .csv log file, but that one doesn't appear on my system

I am on Unraid 7.1.2 and having a problem when trying to exclude some directories. options="-p 1 -i "backups" -W 90 -X 1200 -M 60 -U -U 0 -l on -a '-noleaf -name backups_new -prune -o -print'" service="1" cachePressure="1" suspend="" shares="" include="backups" adaptive="1" minDepth="" depth="" diskIdleTimer="90" scanTimeoutIdle="1200" scanTimeoutBusy="" scanTimeoutStable="" minimum="" maximum="60" mulithreaded="" ulimit="-U 0" log="on" other="-a '-noleaf -name backups_new -prune -o -print'" I have 2 million of files in backups share, which like 1.9 million of them are under backups_new directory. Tried with exclude and without this, still job is taking the same time to do it's work. Also, when checking for find command in process list, all what I can find is: root 2613525 0.0 0.0 2704 1556 ? S 11:56 0:00 /bin/timeout 30 find /mnt/disk1/backups -noleaf root 2613526 0.0 0.0 2704 1332 ? S 11:56 0:00 /bin/timeout 30 find /mnt/disk2/backups -noleaf root 2613527 0.0 0.0 2704 1332 ? S 11:56 0:00 /bin/timeout 30 find /mnt/disk3/backups -noleaf root 2613528 107 0.0 5200 3492 ? R 11:56 0:00 find /mnt/disk1/backups -noleaf root 2613529 107 0.0 8992 7476 ? R 11:56 0:00 find /mnt/disk2/backups -noleaf root 2613530 103 0.0 4332 2796 ? R 11:56 0:00 find /mnt/disk3/backups -noleaf Also, even when I have an option log="on" it is not creating that cache_dirs.csv file, just the typical .log.

It is not only for smb, the smb I can omit, the problematic thing are the 'arrs

I have one issue with the synchronize option. The problem is that if I have files synchronized and I delete a file via SMB from /mnt/user, the file is only deleted from the cache, but it remains in user0. Same stuff with sonarr, e.g. 1080p ISO got downloaded, synchronized to array, then a 4K one appeared - sonarr would remove 1080p one, but only on the cache, and the one on the array still exist. Is there a way to change this behavior? //Edit: Regarding SMB, first delete is removing it from cache, then I can refresh and remove it again from array. But still, the situation for arrs exists.

Tested this, and it works perfectly, it just checks if files are present on the array, comparing mtime, and then just deleting the stuff from cache, so there is no double write!

When it is enabled, I am talking about scenario, when until mover threshold (let's say 90%) the files are going to be just copied to array (synchronized) and when threshold is going to be met, then what's going to happen, is mover going to overwrite or compare them? //Edit, when tested that, it said in the logs of course the move command, but still have no idea how it's gonna behave, what's the exact mover command? Is the compare flag present in the rsync? //Edit2: It should work by default, the mtime is the same on both of files, the synchronized one on array and the one on the cache, so "mover" should ignore them, and just simply remove from cache

Synchronize yes, bur what about the mover if the files are already present on array due to synchronization. Is it going to overwrite them (double write of single file, because of the previous synchronization) or compare and just delete from cache?

Yes, sorry for not correcting it here. @BJZwart https://github.com/masterwishx/ca.mover.tuning/issues/50 Regarding the hardlink issue - @jv6478 just tested, and that synchronize option works perfectly for my needs - it's leaving stuff on cache, copying it to array and placing the hardlinks there. So e.g. have a Linux ISO on /media and /download on cache with a hardlink and also on array with different inode, but with a hardlink too. @Masterwishx How the script is going to behave after it's going to meet the mover threshold after synchronizing data to array? Is it going to replace the files, or is it hash checking and simply removing files from cache? Wanted to know if there is no double writing with that method :).

Imo, that could be a good solution to always have array "synchronized", so all of the files would be laying on array and cache simultaneously, but if cache is going to hit a mover threshold, then this data is just going to evaporate from cache, as it's already on array. That's how I planned to use this functionality, to let me fully utilize the cache to prevent disks from spinning as long as I can and also have a cache data protected by parity

The problem is that currently this option doesn't seem to be working (for me at least) and I have raised an issue on this in this topic and also on GitHub.

Hmm, IIRC it always worked like that, and I was specifically using a jdupes in a crontab every night to create new hardlinks for files moved to array