10 minutes ago, rogman said:

Since it's unlimited storage I left the share backup alone and added per drives. We'll see how long it takes, then I can always uncheck the share drive letter

I thought about that too - but I would have run out of drive letters :D

So it appears that backblaze is smart enough to know that the files have already been uploaded previously even under a different drive letter and it's only scanning them. This should go pretty quick

22 minutes ago, rogman said:

So it appears that backblaze is smart enough to know that the files have already been uploaded previously even under a different drive letter and it's only scanning them. This should go pretty quick

So you are telling me that it would have been smarter to remount the drives maybe one by one - oh well :) Still on the tiny shit - might take until tomorrow. Around 700 000 files to go before something that is big comes up

2 hours ago, gandalf15 said:

So you are telling me that it would have been smarter to remount the drives maybe one by one - oh well :) Still on the tiny shit - might take until tomorrow. Around 700 000 files to go before something that is big comes up

The only downside I see is that it will not back up a file until Mover moves it from the cache to the array disk. Mine is weekly so not that big of an issue. It's chomping through these files and marking each one as "already backed up" but oddly it's using 35GB RAM for the process; I've never seen it over 7GB.

7 hours ago, rogman said:

The only downside I see is that it will not back up a file until Mover moves it from the cache to the array disk. Mine is weekly so not that big of an issue. It's chomping through these files and marking each one as "already backed up" but oddly it's using 35GB RAM for the process; I've never seen it over 7GB.

Simply add the cache too?

If it is moved then, the client will pick it up and move it too towards the disc ;)

@foz, not an issue but curious how many levels the scanner looks through for the read speed check?

I'm trying to find in in github code just haven't tracked it down yet.

[ ok ] M: /drive_m readable, 2 top-level entries sampled
         M: read speed not measured: no file over 64 MB within three levels of /drive_m

Last question: I always get a warning for having no swap. I have a swap drive set in the client, but I guess it never has to roll over to use it, so it generates a warning? I don't think it's an issue, but it always makes me think something is wrong when the scan outputs a warning,

Host resources
  [ ok ] 62 GB host RAM
  [warn] no swap - a memory peak becomes an out-of-memory kill, which is what leaves the stale lock behind
  [ ok ] process table healthy (0 zombies)

Edited September 24Sep 24 by rogman
swap question

Hi @rogman, I happened to be doing some bug fixing (there seem to be lots of edge cases where the backup can hang to squash) and the drive check was hard-coded to 3 levels, which will easily miss files if you've got things laid out with deeper categorisation.

I've left the 3-level setting as the default, as this should do for most and is a good balance for speed, but in Settings in the next release (beta 247), you will be able to alter the depth yourself or select unlimited search with a 60-second timeout per drive.

Edit: The swap message is for OS swap, not the temporary drive setting in Backblaze. If you've got plenty of RAM, then this shouldn't be an issue. The problem is that Backblaze can peak at some extremely high usage, so it's hard to say what a reasonable cutoff is. I'll have a think and maybe make this some sort of setting too. If there's a reasonable calculation I can make with regard to historic usage, then it might be something I can automatically mark as OK.

Edited September 24Sep 24 by foz

24 minutes ago, foz said:

Hi @rogman, I happened to be doing some bug fixing (there seem to be lots of edge cases where the backup can hang to squash) and the drive check was hard-coded to 3 levels, which will easily miss files if you've got things laid out with deeper categorisation.

I've left the 3-level setting as the default, as this should do for most and is a good balance for speed, but in Settings in the next release (beta 247), you will be able to alter the depth yourself or select unlimited search with a 60-second timeout per drive.

Edit: The swap message is for OS swap, not the temporary drive setting in Backblaze. If you've got plenty of RAM, then this shouldn't be an issue. The problem is that Backblaze can peak at some extremely high usage, so it's hard to say what a reasonable cutoff is. I'll have a think and maybe make this some sort of setting too. If there's a reasonable calculation I can make with regard to historic usage, then it might be something I can automatically mark as OK.

What's puzzling is I've never seen such high RAM usage until I changed to the single drives and it's checking each file for deduplication. The container says it's using 35GB, but my Unraid dashboard says I'm only using (17.5GB total on system).

Unraid dash

vs container stats

Interesting little rabbit hole to go down, but relatively straightforward to correct:

The two numbers are both right; they just count different things. The container figure is the kernel's memory charge for the container, and that includes the page cache for every file the client has read. Your dedup scan reads every file on those disks, so the cache grows into the tens of gigabytes and gets booked against the container. Unraid's dashboard counts cache as free, which is why it shows 14 GiB used and 43 GiB free at the same moment. The client's processes themselves are holding a few gigabytes, and the cache gives way to anything else that needs the memory. So nothing to worry about, and it will shrink once the scan is through. The next beta shows this on the Monitor: the processes' own memory on the gauge, with the cache figure beside it, so the figures are clearer.

It also has a cleaner bb-doctor check on Swap - if the host has a reasonable amount of RAM (I went with 16GB), the line will report as info only if swap is still not configured. However, if repeated OOM events occur, this will turn back into a warning to hopefully keep the diagnostics relevant and useful.

Folder depth is set to 4 and now getting measurements on all drives.

Nice update with the memory reporting

Little update from my side: First 2 TB are backuped again

Speed increased a lot: I am only on 35 MB files yet and it goes to 80-90 MB/s anduses 19 Threads according to the Monitor.

So it was indeed the direct diskx mount that increased the speed (including config directly not over mnt/user/).

Lets see if it gets up even more when the files get bigger

Edit: Fun fact comming in: I was just able to see where the client starts to split the files. Once jumped from 95 MB files to 102 MB Files a short break happend. Then it starts uploading at around 9 to 10 threats. This results in a short break between uploads. Once it is split it seems like that it instantly uses the max available threats to upload. I assume it will get faster again once the files get bigger to 1GB+.

Overall its still much faster than before the change

Edit 2:

Well. that speaks for itself... To compare: When I deleted my backup, I was around 13% and it showed 320 days to complete - and I was exactly 101 days in.
I guess the new one will be complete much faster - but so far only on 370 MB files. Lets see when the big ones come in :)

Edited September 25Sep 25 by gandalf15

Update on switching to direct drives instead of array pool. It got to 85% and when I woke up today it was back down to 20%. No idea what happened but we are back a few days of deduplication checks now 😮‍💨

Heya.

Everything seems to be backing up correctly,

However, I've this message the last few releases:

Inheriting backup state. The client reports after files swap, 60.0% (part 4 of 4), at 56 Mbit/s. Nothing is backed up until it finishes, and the counters read zero because the client has not rebuilt them yet. The files already in the datacentre are matched rather than sent again.

and this in bb-doctor

Backup passes
         an inherit of the backup state is in progress: tbs_after_files_swap, 60%. No pass runs until it finishes.
  [ ok ] no pass has lost the four-hour lock in the current log

The numbers haven't changed in that time. Always 4/4 and 60%.

But like I said, files are being backed up and no errors otherwise.

Here's my bundle:

Thank you!

backblaze64-diag-202609271456.zip

Thanks for the bundle, it helped me narrow down the bug quickly. Your backup is fine: the latest pass went all the way through and recorded its files, and nothing in the transfer logs mentions an inherit. What happened is that your client never wrote the final "done" stage into its progress file; it stopped at "after files swap, 60%", most likely because the container was restarted during the last step, and my code treated anything without a "done" message as still running. That is why the notice, the terminal monitor and bb-doctor all kept reporting the apparently still in-progress state.

The next beta now reads it the way the client behaves: a progress file that hasn't changed for an hour while passes have run since counts as finished. The notice goes away, and bb-doctor notes that the inherit finished earlier and where its file stopped. Once you update to the latest release, these messages should clear automatically.

My own backup is only about a quarter of the way through its first backup (it’s quite large!) and I couldn’t inherit as my prior client was a Mac, so this scenario hadn’t come up for me; I’m glad for reports like yours - the software should work well for everyone!

The individual disc backup is now complete! I added the cache drive so it backs up daily before mover relocates them to the array discs. Can't stress often enough how thankful I am for what you've created for us Foz!

I do feel like it will be slightly easier to manage a bad disk replacement with this approach instead of figuring out what was on the disc.

So the current beta Watchdog does not seem to work.

Had it now several times that it simply froze and status said that nothing changed since hours:

1. 20:01watchdog: detected: DOWN the Backblaze service (bzserv) is not running, so no backup pass can start - the container restarts the service on its own; see the startapp log

2. 19:46Uploading for 25h 00m: 888 files (1.6 TB) so far, 158 already backed up: average 154.6 Mbit/s, 4.9 threads, mem 4.7 GB

3. 19:31watchdog: detected: DOWN the Backblaze service (bzserv) is not running, so no backup pass can start - the container restarts the service on its own; see the startapp log

4. 19:01watchdog: detected: DOWN the Backblaze service (bzserv) is not running, so no backup pass can start - the container restarts the service on its own; see the startapp log

That was the case for 6 hours.

Diagnostics attached:

backblaze64-diag-202610021824.zip

Yes, this is an issue I had discovered during bug fixing. The beta build of 28 September, build 257, the one your bundle shows, has a bug in the part of the container that restarts the Backblaze service when it dies. The first time the service stopped after that build, the restart routine called itself in a loop instead of restarting anything. The watchdog saw the service was down every half hour and handed the job to a routine that was no longer working, which is the pattern in your timeline.

The fix has been in :beta since 30 September, and today's beta adds a second line of defence: if the service is still down half an hour after the watchdog first reports it, the watchdog starts it itself instead of waiting on the service watch. Pull the latest beta and recreate the container from the Unraid template:

docker pull iamfoz/backblaze-64-personal-wine-container:beta

The broken routine in your running container is also putting load on a CPU core, and has been since the service died. Recreating the container ends that as well.

Your pass had been waiting on its upload children for 37 minutes before everything stopped, the same as on my own host the day before. I am looking at what ends those waits. A diagnostic bundle from the new build after it has run for a day would help, since from this build the bundle includes the recovery log and the service's own log.

Hopefully the changes have squashed this bug!

First: wonderful work here, thanks so much for maintaining this container. It fills a huge need for me!

It looks like I independently came to similar conclusions to what is suggested here: using /mnt/disk* mapped as drives in the container rather than user shares. As another datapoint this took my initial file scan from ~10 hours to ~15 minutes, and now my backup is proceeding at a very reasonable pace instead of looking like it's going to take 6 months to complete based on the pace it was moving through my small files.

I additionally used Backblaze's XML configurable exclusions file because I didn't want to add all the directories I want to exclude (mostly media shares) over and over again while testing. This conveniently excludes the path across all disks: https://www.backblaze.com/computer-backup/docs/configure-custom-exclusions-using-xml-windows

---

I was working with Claude to diagnose slow backups when the container's drives are mapped to Unraid user shares (/mnt/user/<share>) before switching to mapping disk and pool paths. Along the way it had me take a series of strace measurements, and they point to one specific Wine behavior that looks responsible for much of the slowdown on user shares. Its findings and suggested change are below, in case you think it's worth patching in your locally maintained WINE fork or for contributing upstream.

Claude wrote up the following summary which may or may not be useful/interesting:

Setup

Backblaze64 :beta on Unraid 7.3.2, client 10.0.3.1075, about 2.8M files. Each user share was mapped to its own drive letter (/mnt/user/<share> -> /drive_x). For comparison, a second container had the same data mapped through disk and pool paths (/mnt/diskN, /mnt/<pool>).

Finding

On user-share drives, a large share of wall time goes to fstatfs() calls that each take about 21 ms. Each of those calls is followed by a failed stat() of .ciopfs. This looks like Wine's directory case-sensitivity check in ntdll: when statfs reports a FUSE filesystem, Wine looks for a ciopfs marker. The result doesn't appear to be cached, so the check repeats for every lookup, roughly 50 times per file during upload preparation. shfs is FUSE, and its statfs is slow, presumably because it aggregates across every disk the share spans. On XFS and ZFS the same call takes about 20 µs, and the .ciopfs probe never runs.

Evidence

All figures come from strace -c -w -f -p <pid>, which reports wall-clock time per syscall, unless noted otherwise.

1. bztransmit -prepare_bzcombs on user-share drives, 30 seconds:

   • fstatfs: 2,418 calls at 5,450 µs average, 13.2 s total, 48% of wall time

   • read (mostly wineserver replies): 11,555 calls at 938 µs average, 39% of wall time

   • newfstatat: 1,305 failed calls; ioctl: 2,439 failed calls; getdents64: 1,026 calls

   • Throughput: about 1,740 files per hour, 0.3 Mbit/s, monitor showing 0.0 threads, with the CPU and network mostly idle

2. The same process, strace -f -T -y -e trace=fstatfs for 20 seconds, grouped by path:

   • About 520 calls on shfs-backed drives, totaling about 10.8 s, about 21 ms per call

   • About 2,050 calls on ZFS paths (/config, exclusive shares), totaling 0.04 s, about 20 µs per call

   • The failed newfstatat calls were all ".ciopfs", apart from the client's own .tmp status files.

3. bzfilelist scan on user-share drives:

   • About 185 entries per second

   • fstatfs at about 17.9 ms average

   • About 63 .ciopfs probes per second

   • read at 0.7-1.5 ms average

4. Native statfs on the host, measured as 500 calls in a single stat -f process to remove process-start overhead:

   • /mnt/disk18: 45 µs

   • /mnt/user/Nick (an exclusive share, which resolves to ZFS): 202 µs

   • /mnt/user/OldD and /mnt/user/Backups (spread across array disks): 21,586 µs and 21,416 µs

   For comparison, a per-file stat on the same files: 40 µs through the disk path, 549 µs through /mnt/user.

5. winedevice's periodic fstatfs sweep of the drive roots: 20-28 ms on each shfs drive, about 20 µs on each ZFS drive.

6. The same data mapped through disk and pool paths:

   • Scan: 1,042 entries per second (5.6x), with a full scan finishing in under 15 minutes. fstatfs 18 µs, read 19 µs, 0 .ciopfs probes.

   • prepare_bzcombs: per-entry work rate 4.8x higher. fstatfs 18 µs (0.7% of wall time), read 83 µs (16%), 0 .ciopfs probes.

Interpretation

The .ciopfs probe accounts for about half of the upload-preparation time on user shares. read latency (wineserver round trips) is also 10-50x higher on user shares. My guess is that wineserver itself sometimes blocks on shfs calls and stalls every client, but that's an inference: I didn't trace wineserver directly.

Suggested change

Cache the case-sensitivity result per filesystem (keyed on st_dev), or at least per directory, so the fstatfs and .ciopfs check runs once instead of on every lookup.

Edited October 5Oct 5 by nick5429

@foz I'm not seeing any errors but when I switched to per disc backup, one of my drives N: doesn't show up in the status tab and says nothing is marked for backup but it's already backed it up. Doctor scan reports nothing selected for backup.

Update: Installed beta 269, and the issue appears resolved now.

Edited October 6Oct 6 by rogman
working now

As asked I upload the log for the update - so far it runs without a problem. I noticed that I actually had also the warning mentioned in the post above but never realized it - it was fixed after I updated but the log might still help:backblaze64-diag-202610081715.zip

@foz I have a feature request if possible.

I get these semi-frequently: "watchdog: recovered: stopped orphaned bztransmit.exe [394996], running 24m after the pass that started it exited"

Is it possible to have a setting to input a time in minutes for bztransmit to stop it (say 3, 5, 10 min - user input)?

• 03:31watchdog: recovered: stopped orphaned bztransmit.exe [642494], running 24m after the pass that started it exited

• 02:31watchdog: recovered: stopped orphaned bztransmit.exe [604880], running 24m after the pass that started it exited

• 01:31watchdog: recovered: stopped orphaned bztransmit.exe [563048], running 24m after the pass that started it exited

• 00:31watchdog: recovered: stopped orphaned bztransmit.exe [517843], running 24m after the pass that started it exited

• 23:30watchdog: recovered: stopped orphaned bztransmit.exe [465831], running 24m after the pass that started it exited

• 22:20watchdog: recovered: stopped orphaned bztransmit.exe [421677], running 24m after the pass that started it exited

2nd issue

I've also been getting a lot of "Backblaze reports a failed file check" and when I dismiss it or reset it, they just come right back.

What I found is in the "bzdc_filecheck.xml" file it reports

---

<?xml version="1.0" encoding="UTF-8" ?>

<filecheckresponse>

<local_computer_inserted gmt_date_time_written="20261008160637" />

<metainfo bypass="false" hguid="" file_check_is_clean="false" safety_frozen="not_frozen" />

<verify_failed filename="bz_done_20261007_1.bzff" reason="wrong_size_bz_comb_3641242" />

</filecheckresponse>

---

The backup finishes and has no errors, no files marked as couldn't upload and the only way to clear out the status is to edit that xml file and remove the filename line and change the check_is_clean to "true"

This is probably not the correct way to fix it but I hate seeing those red error letters when it is working ultimately

Edited October 9Oct 9 by rogman

It’s been a busy few days, mostly on the beta. I started work on the issues raised here, then found more bugs to squash, and a filed issue on GitHub also needed attention.

The file-handle leak

Client 10.0.3.1075’s service opens its Backup.sql database every few seconds. Under Wine, none of those handles ever closed, so they built up until Wine hit its 40,960 open-file limit. On my server that took about six hours, and on the reporter’s about nineteen. After that, nothing in the container could open a file. Backup passes hung, and eventually the service died. If your bzserv has been stopping about once a day since 1075, this is why.

The cause was in Wine. SQLite keeps its database locked, and Wine was holding every closed handle on a locked file open because of how Linux file locks work. The beta (build 299 onwards) is now on Wine 11.19, with a patch that uses a different kind of Linux lock, so it releases closed handles. On my server, the count for that file now holds at 4. On the old build, it passed 400 within twelve minutes. 11.19 also fixes the 11.18 start-up fault I found when it was briefly packaged with the beta image.

Both images also have a fallback:

• With Automatic recovery on, the watchdog restarts the Backblaze service when Wine’s open files reach 85% of the limit (FD_RESTART_PCT).

• bb-doctor now reports the count against the limit.

The stable images stay on standard Wine, so they rely on that restart.

If you’re using /config on /mnt/user/appdata/..., consider moving it to /mnt/cache/appdata/.... Every file Wine holds open is also held open by shfs, which shares one limit with everything else on /mnt/user. In the issue, the leak ran shfs out, and a PostgreSQL container started crashing.

A bug in autorecovery

Between 6th and 8th October, the watchdog had a rule meant to stop stray Backblaze processes. It decided a process was stray when its parent was the container’s init process. Wine starts every Windows process that way, so the rule stopped scans and, once, a live upload process, and the pass hung waiting for it. Nothing was lost, because the client redoes the work, but if you pulled an image in that window and have Automatic recovery on, please update; the code is gone.

Other changes in the beta

• An Exclusions panel on the Settings tab (more on this below).

• bb-doctor names any directory the scanner couldn’t open, with the reason. Windows rejects names containing ", and names ending in a dot, and the container user can’t read some ownerships. When the scanner hits one of these, it skips the folder silently, and the only sign was “producing file lists” for hours. On my server, every scan for three days had been stopping at one folder with quotes in its name. The Monitor shows a warning while such a folder exists, and it can be turned off in Settings.

• bb-doctor warns when a mapped drive’s folders are separate filesystems. The client doesn’t cross mount points. On Unraid, each share on a ZFS pool is its own dataset, so mapping /mnt/cache as one drive backs up nothing. Map /mnt/cache/<share> instead.

• bb-doctor’s drive selection check now reads the client’s own volume list (see the reply on drive N: below).

• The watchdog now starts the Backblaze service itself if it has stayed down for more than one check. Two people had it sit down for hours.

• Diagnostic bundles now include the recovery log and the end of the service’s own log.

• The Status tab’s drive bars are in drive-letter order and line up. They no longer jump around when the ETA changes width.

The stable images moved to the latest base image (v4.14.0) and picked up the watchdog and bb-doctor changes. They’re still on standard Wine 11.0.

Re: mapping disks, the XML exclusions file and the strace results

@nick5429, thanks for taking the measurements, which found something I’d missed. Every time Wine fails to find a file under the exact name it was given, it checks whether the folder is case sensitive. On a FUSE filesystem such as shfs, part of that check is a statfs call. shfs answers it by adding up free space across every disk the share spans, so each call takes about 21 ms. Your trace had it at nearly half the wall time of a pass.

The beta (build 299 onwards) now carries a patch that makes the decision once per device and caches it. I’ve moved my own server to disk paths, so I haven’t been able to measure it on user shares yet - I'll be adding them back after my backup completes. If you still have the user-share container, I’d be glad to see a before and after. Mapping /mnt/diskN and the pool shares is still the faster setup, since shfs is slower at everything else too. I’ll be raising the Wine side upstream: there’s an existing Wine bug for slow FUSE filesystems that this adds weight to.

On the exclusions file: you were right that it’s the sensible way to exclude a folder across twenty disks. The Settings tab now has an Exclusions panel that manages the rules in bzexcluderules_editable.xml. It only touches its own rules between two marker comments, so your hand-written rules stay exactly as they are. It keeps the previous file alongside, and its rules are included in the settings export.

One tip for anyone switching to per-disk drives: the client abandons a drive selection saved while it’s uploading (there’s just a line in its log). Pause the backup, untick the drive and press OK, tick it and press OK again, then resume. Before mapping /mnt/user alongside the disks, check that /mnt/user/.bzvol doesn’t show one of the disks’ stamps, because the client recognises drives by that stamp.

Re: drive N: not showing on Status, and bb-doctor saying nothing is selected

@rogman, glad it sorted itself out. I suspect two things lined up:

The Status tab only shows a drive once the client has written a file list for it, and after a switch to per-disk drives the client rescans each disk, which can take a while.

bb-doctor’s selection check was looking in the wrong place. It looked for a per-drive entry the client only writes when you change a drive from its default, so a drive could be selected and still read as “nothing selected”. From build 279, it reads the client’s volume list, the list the client works from.

If it comes back on a current beta, a bb-report bundle would help me pin it down.

Ah, shows how long I've been working on the latest round of changes with this page open - I hadn't refreshed and seen these new messages!

Re: the diagnostics bundle

@gandalf15 thanks, it helps. The bundle is from build 261, so it shows how things were before the fixes:

• The service died once a day, about 19 hours apart. That’s the file-handle leak from GitHub issue iamfoz/backblaze-64-personal-wine-container#13.

• After each death the service couldn’t start again for 30 to 60 minutes, until the watchdog cleared the stuck pass.

• The “no selection entry” warnings for E: to V: were the false warning fixed in build 279. That’s the one you saw disappear.

The current beta fixes the leak itself, so you shouldn’t see those daily stops any more.

It also found a file under appdata/Notifiarr the container can’t read. If you don’t need appdata backed up, an exclusion for it is the simplest fix.

Re: orphaned bztransmit and the failed file check

@rogman, sorry, both of these are down to my bugs. Between 6th and 8th October the watchdog had a rule that stopped Backblaze processes it thought had been left behind. Under Wine every process looks that way, so it was stopping the live upload process about 24 minutes into each pass. That’s the hourly pattern you’re seeing. The rule was removed on 8th October, so rather than a timeout setting, please update to the latest beta (build 301) and the messages will stop.

I think the failed file check came from the same thing. The file it names, bz_done_20261007_1.bzff, is the client’s record of what it uploaded on 7th October. Backblaze’s servers check the size of each one, and wrong_size_bz_comb means the copy they hold doesn’t match. That’s what you’d expect if the upload of that record was cut off part way. The client keeps a log of these and sends the file again on later passes. I’m seeing the same failure on my own server since the 7th October kills, so I’m watching to see how it clears and will report back.

I’d leave bzdc_filecheck.xml alone. It’s the server’s verdict written down locally, so editing it only hides the message, and the next check rewrites it. In the meantime, you can switch that warning off under Settings > Warnings and events.

Dismissing it should keep it hidden until you press Reset warnings. If it comes back after a plain dismiss, can you tell me which build you’re on?

The file you pasted also includes your hguid, Backblaze’s ID for your computer. It’s best kept out of public posts, so you may want to edit it out.

Tested build 301 on a user-share scan of TV-CurrentShows, which spans 3 array disks and where native statfs through shfs still takes ~20 ms: 442 entries/s vs ~185 before, .ciopfs probes 0 vs ~63/s, and fstatfs calls per entry dropped about 17x (~0.34 to ~0.02). So the caching works!

Collected raw data:

===== after  Backblaze64_usershares  bzfilelist.exe  2026-10-09 10:55:09 =====

===== image =====
repository ghcr.io/iamfoz/backblaze-personal-wine:beta
image built 2026-10-08T18:56:30.87503774Z
Container
  Image:     beta
  build:     version=beta
  build:     revision=9005a85627f5b44d0bed133ca1a50ba61d877433
  build:     build=301
  build:     variant=beta-ubuntu26
  build:     built=2026-10-08T18:56:30Z
  Wine:      wine-11.19

Backblaze client
  Installed: 10.0.3.1075
  Available: 10.0.3.1075 (via ca000)
  Status:    up to date

===== drive mappings and backing filesystem =====
/drive_y   fuse       /mnt/user/TV-CurrentShows

===== wall-clock syscall profile, 60s (pid 1025052) =====
process age at start: 9 s
strace: Process 1025052 attached
strace: Process 1025052 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.43   33.578809         509     65860           read
 12.14    6.231815       11455       544           fstatfs
  5.13    2.633308          52     49736      2601 newfstatat
  4.54    2.329150          17    132235           rt_sigprocmask
  4.17    2.142396          80     26528     26196 getxattr
  3.06    1.572704         930      1691           getdents64
  1.00    0.513205          18     27956           write
  0.88    0.453982          18     24251           writev
  0.69    0.352019          17     19688           lseek
  0.56    0.286198          18     15406           close
  0.55    0.281865          20     13732           fchdir
  0.41    0.208192          24      8446           fstat
  0.39    0.201743          25      7897           openat
  0.32    0.165721          22      7508           recvmsg
  0.28    0.142422          17      8226           fcntl
  0.24    0.120629          19      6347      6347 ioctl
  0.13    0.066737          18      3607           rt_sigreturn
  0.05    0.025775          44       574           pwrite64
  0.02    0.009839          19       505       505 pread64
  0.00    0.000258          23        11           mprotect
  0.00    0.000176          43         4           mmap
------ ----------- ----------- --------- --------- ----------------
100.00   51.316942         121    420752     35649 total

summary
  entries/s (getxattr)     442.1
  fstatfs       9.1/s  avg   11455 us   10.4% of wall
  read       1097.7/s  avg     509 us   56.0% of wall

===== fstatfs by drive, 20s =====
     31 calls     0.517 s total    16.662 ms avg  /drive_y

===== .ciopfs probes, 10s =====
0 probes in 10 s

===== CPU, 3 x 5s (scanner/prep process and wineserver) =====
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1025052 nobody    20   0 2045612  26744   9936 S   0.0   0.1   0:06.58 bzfilel+
1018387 nobody    20   0   18424  14968   1424 S   0.0   0.0   0:08.86 wineser+
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1018387 nobody    20   0   18424  14972   1424 S   5.4   0.0   0:09.13 wineser+
1025052 nobody    20   0 2045612  26744   9936 S   2.8   0.1   0:06.72 bzfilel+
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1018387 nobody    20   0   18424  14972   1424 S   4.6   0.0   0:09.36 wineser+
1025052 nobody    20   0 2045612  26744   9936 S   2.4   0.1   0:06.84 bzfilel+

Digging a little deeper monitoring the trace:

--- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_TKILL, si_pid=1852, si_uid=99} ---
--- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_TKILL, si_pid=1852, si_uid=99} ---
newfstatat(AT_FDCWD</config/wine/drive_c/windows/system32>, "/config/wine/dosdevices/y:/myDir/ectory", {st_mode=S_IFDIR|0777, st_size=42,       ...}, 0) = 0
fstatfs(16</drive_y/myDir/ectory>, {f_type=FUSE_SUPER_MAGIC, f_bsize=4096, f_blocks=14264479561, f_bfree=4780425483, f_bavail=4780425483,       f_files=32173436487, f_ffree=32171879627, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOATIME}) = 0
newfstatat(AT_FDCWD</config/wine/drive_c/windows/system32>, "/config/wine/dosdevices/mountpointmanager", 0x7ffffdfff530, 0) = -1 ENOENT (No such file or directo      ry)

Claude notes:

Follow-up on the remaining fstatfs: each one happens once per directory, in a fixed sequence. A failed stat of dosdevices/mountpointmanager (Wine resolving \\.\MountPointManager), one or two SIGUSR1 from wineserver, a stat of the directory, then fstatfs on the directory's handle (descriptor passed from wineserver, no local openat). On shfs that fstatfs costs ~11-17 ms and returns identical results for every directory on the share. It looks like the volume-information path (device type, characteristics or filesystem attributes) rather than the case-sensitivity check, so caching that result per device, like the case-sensitivity patch does, would likely remove most of the remaining per-directory cost

(but this is likely minor compared to the fix you already made)

Nice work!

This share I'm testing with is full of large files instead of a ton of tiny ones, but once it got to the point of actually uploading files it's uploading at a significant fraction of my gigabit upstream link (200-500 mbit/sec)

In the upload-prep phase (bztransmit -prepare_bzcombs) on the same shfs share, build 301 made no fstatfs calls on the shfs drive at all (vs ~80/s at 5.45 ms each before, 48% of wall time) and zero .ciopfs probes. The process is no longer waiting on shfs: wineserver now runs at ~75% of a core and bztransmit at 35-50%, with upload threads reading file content

syscall profile while uploading files:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 64.11  381.670886         316   1207587           read
 15.79   94.009386          45   2070102           rt_sigprocmask
  3.88   23.102137          45    504686           write
  3.67   21.824808          46    472780           writev
  2.72   16.201333      300024        54           pselect6
  2.28   13.548210          46    293070           clock_gettime
  1.88   11.210074          46    242804           getsockopt
  1.04    6.213267          63     97293        26 sendmsg
  0.88    5.257933          47    111506     11711 newfstatat
  0.49    2.936435          46     63332     48527 recvmsg
  0.48    2.853184          66     42824       850 mmap
  0.37    2.214767          46     47223        75 close
  0.31    1.819285          56     32162     32162 getxattr
  0.29    1.747434          40     43600           mprotect
  0.26    1.529106          43     34756      2870 openat
  0.23    1.382583          41     33154           lseek
  0.22    1.314960          42     31257           fcntl
  0.21    1.240101          42     29370        50 fstat
  0.16    0.930960          42     22100           fchdir
  0.15    0.902320          66     13572           getdents64
  0.11    0.634950        5079       125           clone
  0.10    0.606234       12124        50           wait4
  0.09    0.523412          42     12410     12360 ioctl
  0.06    0.345682          40      8550       250 access
  0.05    0.327091          45      7225           munmap
  0.05    0.281705        1126       250        50 pread64
  0.05    0.268928        5378        50           execve
  0.03    0.199884          39      5000      3150 readlink
  0.01    0.037420          41       900           fstatfs
  0.00    0.028524          76       374           pwrite64
  0.00    0.024236          35       675           rt_sigaction
  0.00    0.013196          38       346           brk
  0.00    0.012618          48       262           getrusage
  0.00    0.012359          35       350           prlimit64
  0.00    0.011034         147        75           poll
  0.00    0.008531          37       225        50 prctl
  0.00    0.008124          40       200           getcwd
  0.00    0.007560          43       175       150 connect
  0.00    0.007461          42       175           socket
  0.00    0.007341          41       175           set_robust_list
  0.00    0.007166          40       175           pipe2
  0.00    0.006787          38       175           arch_prctl
  0.00    0.006459          49       131           sched_yield
  0.00    0.006357          42       150           futex
  0.00    0.005787          38       150           setsockopt
  0.00    0.004301          43       100           uname
  0.00    0.004257          42       100           sysinfo
  0.00    0.003831          38       100           open
  0.00    0.003665          35       102           getrandom
  0.00    0.003619          36       100           dup2
  0.00    0.003605          36       100           umask
  0.00    0.002958          39        75           gettid
  0.00    0.002931          39        75           sigaltstack
  0.00    0.002839          37        75           rseq
  0.00    0.002814          56        50           socketpair
  0.00    0.002484          49        50           fadvise64
  0.00    0.002332          93        25           sendmmsg
  0.00    0.002290          45        50           recvfrom
  0.00    0.002138          42        50        50 fgetxattr
  0.00    0.001975          39        50        50 userfaultfd
  0.00    0.001890          37        50           getuid
  0.00    0.001819          34        52           dup
  0.00    0.001783          35        50           getpid
  0.00    0.001745          34        50           set_tid_address
  0.00    0.001724          34        50           setsid
  0.00    0.001107          44        25        25 clone3
  0.00    0.001098          43        25           rt_sigreturn
------ ----------- ----------- --------- --------- ----------------
100.00  595.365221         109   5433009    112406 total

summary
  entries/s (getxattr)     536.0
  fstatfs      15.0/s  avg      41 us    
  read      20126.5/s  avg     316 us  

My own drive-based backup is mostly complete so I'm personally going to shut down this test usershare container and stick with that I think, but this change may have taken things to the point where backing up based on usershares is actually viable