10 minutes ago, rogman said:

Since it's unlimited storage I left the share backup alone and added per drives. We'll see how long it takes, then I can always uncheck the share drive letter

I thought about that too - but I would have run out of drive letters :D

So it appears that backblaze is smart enough to know that the files have already been uploaded previously even under a different drive letter and it's only scanning them. This should go pretty quick

22 minutes ago, rogman said:

So it appears that backblaze is smart enough to know that the files have already been uploaded previously even under a different drive letter and it's only scanning them. This should go pretty quick

So you are telling me that it would have been smarter to remount the drives maybe one by one - oh well :) Still on the tiny shit - might take until tomorrow. Around 700 000 files to go before something that is big comes up

2 hours ago, gandalf15 said:

So you are telling me that it would have been smarter to remount the drives maybe one by one - oh well :) Still on the tiny shit - might take until tomorrow. Around 700 000 files to go before something that is big comes up

The only downside I see is that it will not back up a file until Mover moves it from the cache to the array disk. Mine is weekly so not that big of an issue. It's chomping through these files and marking each one as "already backed up" but oddly it's using 35GB RAM for the process; I've never seen it over 7GB.

7 hours ago, rogman said:

The only downside I see is that it will not back up a file until Mover moves it from the cache to the array disk. Mine is weekly so not that big of an issue. It's chomping through these files and marking each one as "already backed up" but oddly it's using 35GB RAM for the process; I've never seen it over 7GB.

Simply add the cache too?

If it is moved then, the client will pick it up and move it too towards the disc ;)

@foz, not an issue but curious how many levels the scanner looks through for the read speed check?

I'm trying to find in in github code just haven't tracked it down yet.

[ ok ] M: /drive_m readable, 2 top-level entries sampled
         M: read speed not measured: no file over 64 MB within three levels of /drive_m

Last question: I always get a warning for having no swap. I have a swap drive set in the client, but I guess it never has to roll over to use it, so it generates a warning? I don't think it's an issue, but it always makes me think something is wrong when the scan outputs a warning,

Host resources
  [ ok ] 62 GB host RAM
  [warn] no swap - a memory peak becomes an out-of-memory kill, which is what leaves the stale lock behind
  [ ok ] process table healthy (0 zombies)

Edited September 24Sep 24 by rogman
swap question

Hi @rogman, I happened to be doing some bug fixing (there seem to be lots of edge cases where the backup can hang to squash) and the drive check was hard-coded to 3 levels, which will easily miss files if you've got things laid out with deeper categorisation.

I've left the 3-level setting as the default, as this should do for most and is a good balance for speed, but in Settings in the next release (beta 247), you will be able to alter the depth yourself or select unlimited search with a 60-second timeout per drive.

Edit: The swap message is for OS swap, not the temporary drive setting in Backblaze. If you've got plenty of RAM, then this shouldn't be an issue. The problem is that Backblaze can peak at some extremely high usage, so it's hard to say what a reasonable cutoff is. I'll have a think and maybe make this some sort of setting too. If there's a reasonable calculation I can make with regard to historic usage, then it might be something I can automatically mark as OK.

Edited September 24Sep 24 by foz

24 minutes ago, foz said:

Hi @rogman, I happened to be doing some bug fixing (there seem to be lots of edge cases where the backup can hang to squash) and the drive check was hard-coded to 3 levels, which will easily miss files if you've got things laid out with deeper categorisation.

I've left the 3-level setting as the default, as this should do for most and is a good balance for speed, but in Settings in the next release (beta 247), you will be able to alter the depth yourself or select unlimited search with a 60-second timeout per drive.

Edit: The swap message is for OS swap, not the temporary drive setting in Backblaze. If you've got plenty of RAM, then this shouldn't be an issue. The problem is that Backblaze can peak at some extremely high usage, so it's hard to say what a reasonable cutoff is. I'll have a think and maybe make this some sort of setting too. If there's a reasonable calculation I can make with regard to historic usage, then it might be something I can automatically mark as OK.

What's puzzling is I've never seen such high RAM usage until I changed to the single drives and it's checking each file for deduplication. The container says it's using 35GB, but my Unraid dashboard says I'm only using (17.5GB total on system).

Unraid dash

vs container stats

Interesting little rabbit hole to go down, but relatively straightforward to correct:

The two numbers are both right; they just count different things. The container figure is the kernel's memory charge for the container, and that includes the page cache for every file the client has read. Your dedup scan reads every file on those disks, so the cache grows into the tens of gigabytes and gets booked against the container. Unraid's dashboard counts cache as free, which is why it shows 14 GiB used and 43 GiB free at the same moment. The client's processes themselves are holding a few gigabytes, and the cache gives way to anything else that needs the memory. So nothing to worry about, and it will shrink once the scan is through. The next beta shows this on the Monitor: the processes' own memory on the gauge, with the cache figure beside it, so the figures are clearer.

It also has a cleaner bb-doctor check on Swap - if the host has a reasonable amount of RAM (I went with 16GB), the line will report as info only if swap is still not configured. However, if repeated OOM events occur, this will turn back into a warning to hopefully keep the diagnostics relevant and useful.

Folder depth is set to 4 and now getting measurements on all drives.

Nice update with the memory reporting

Little update from my side: First 2 TB are backuped again

Speed increased a lot: I am only on 35 MB files yet and it goes to 80-90 MB/s anduses 19 Threads according to the Monitor.

So it was indeed the direct diskx mount that increased the speed (including config directly not over mnt/user/).

Lets see if it gets up even more when the files get bigger

Edit: Fun fact comming in: I was just able to see where the client starts to split the files. Once jumped from 95 MB files to 102 MB Files a short break happend. Then it starts uploading at around 9 to 10 threats. This results in a short break between uploads. Once it is split it seems like that it instantly uses the max available threats to upload. I assume it will get faster again once the files get bigger to 1GB+.

Overall its still much faster than before the change

Edit 2:

Well. that speaks for itself... To compare: When I deleted my backup, I was around 13% and it showed 320 days to complete - and I was exactly 101 days in.
I guess the new one will be complete much faster - but so far only on 370 MB files. Lets see when the big ones come in :)

Edited September 25Sep 25 by gandalf15

Update on switching to direct drives instead of array pool. It got to 85% and when I woke up today it was back down to 20%. No idea what happened but we are back a few days of deduplication checks now 😮‍💨

Heya.

Everything seems to be backing up correctly,

However, I've this message the last few releases:

Inheriting backup state. The client reports after files swap, 60.0% (part 4 of 4), at 56 Mbit/s. Nothing is backed up until it finishes, and the counters read zero because the client has not rebuilt them yet. The files already in the datacentre are matched rather than sent again.

and this in bb-doctor

Backup passes
         an inherit of the backup state is in progress: tbs_after_files_swap, 60%. No pass runs until it finishes.
  [ ok ] no pass has lost the four-hour lock in the current log

The numbers haven't changed in that time. Always 4/4 and 60%.

But like I said, files are being backed up and no errors otherwise.

Here's my bundle:

Thank you!

backblaze64-diag-202609271456.zip

Thanks for the bundle, it helped me narrow down the bug quickly. Your backup is fine: the latest pass went all the way through and recorded its files, and nothing in the transfer logs mentions an inherit. What happened is that your client never wrote the final "done" stage into its progress file; it stopped at "after files swap, 60%", most likely because the container was restarted during the last step, and my code treated anything without a "done" message as still running. That is why the notice, the terminal monitor and bb-doctor all kept reporting the apparently still in-progress state.

The next beta now reads it the way the client behaves: a progress file that hasn't changed for an hour while passes have run since counts as finished. The notice goes away, and bb-doctor notes that the inherit finished earlier and where its file stopped. Once you update to the latest release, these messages should clear automatically.

My own backup is only about a quarter of the way through its first backup (it’s quite large!) and I couldn’t inherit as my prior client was a Mac, so this scenario hadn’t come up for me; I’m glad for reports like yours - the software should work well for everyone!

The individual disc backup is now complete! I added the cache drive so it backs up daily before mover relocates them to the array discs. Can't stress often enough how thankful I am for what you've created for us Foz!

I do feel like it will be slightly easier to manage a bad disk replacement with this approach instead of figuring out what was on the disc.

So the current beta Watchdog does not seem to work.

Had it now several times that it simply froze and status said that nothing changed since hours:

1. 20:01watchdog: detected: DOWN the Backblaze service (bzserv) is not running, so no backup pass can start - the container restarts the service on its own; see the startapp log

2. 19:46Uploading for 25h 00m: 888 files (1.6 TB) so far, 158 already backed up: average 154.6 Mbit/s, 4.9 threads, mem 4.7 GB

3. 19:31watchdog: detected: DOWN the Backblaze service (bzserv) is not running, so no backup pass can start - the container restarts the service on its own; see the startapp log

4. 19:01watchdog: detected: DOWN the Backblaze service (bzserv) is not running, so no backup pass can start - the container restarts the service on its own; see the startapp log

That was the case for 6 hours.

Diagnostics attached:

backblaze64-diag-202610021824.zip

Yes, this is an issue I had discovered during bug fixing. The beta build of 28 September, build 257, the one your bundle shows, has a bug in the part of the container that restarts the Backblaze service when it dies. The first time the service stopped after that build, the restart routine called itself in a loop instead of restarting anything. The watchdog saw the service was down every half hour and handed the job to a routine that was no longer working, which is the pattern in your timeline.

The fix has been in :beta since 30 September, and today's beta adds a second line of defence: if the service is still down half an hour after the watchdog first reports it, the watchdog starts it itself instead of waiting on the service watch. Pull the latest beta and recreate the container from the Unraid template:

docker pull iamfoz/backblaze-64-personal-wine-container:beta

The broken routine in your running container is also putting load on a CPU core, and has been since the service died. Recreating the container ends that as well.

Your pass had been waiting on its upload children for 37 minutes before everything stopped, the same as on my own host the day before. I am looking at what ends those waits. A diagnostic bundle from the new build after it has run for a day would help, since from this build the bundle includes the recovery log and the service's own log.

Hopefully the changes have squashed this bug!

First: wonderful work here, thanks so much for maintaining this container. It fills a huge need for me!

It looks like I independently came to similar conclusions to what is suggested here: using /mnt/disk* mapped as drives in the container rather than user shares. As another datapoint this took my initial file scan from ~10 hours to ~15 minutes, and now my backup is proceeding at a very reasonable pace instead of looking like it's going to take 6 months to complete based on the pace it was moving through my small files.

I additionally used Backblaze's XML configurable exclusions file because I didn't want to add all the directories I want to exclude (mostly media shares) over and over again while testing. This conveniently excludes the path across all disks: https://www.backblaze.com/computer-backup/docs/configure-custom-exclusions-using-xml-windows

---

I was working with Claude to diagnose slow backups when the container's drives are mapped to Unraid user shares (/mnt/user/<share>) before switching to mapping disk and pool paths. Along the way it had me take a series of strace measurements, and they point to one specific Wine behavior that looks responsible for much of the slowdown on user shares. Its findings and suggested change are below, in case you think it's worth patching in your locally maintained WINE fork or for contributing upstream.

Claude wrote up the following summary which may or may not be useful/interesting:

Setup

Backblaze64 :beta on Unraid 7.3.2, client 10.0.3.1075, about 2.8M files. Each user share was mapped to its own drive letter (/mnt/user/<share> -> /drive_x). For comparison, a second container had the same data mapped through disk and pool paths (/mnt/diskN, /mnt/<pool>).

Finding

On user-share drives, a large share of wall time goes to fstatfs() calls that each take about 21 ms. Each of those calls is followed by a failed stat() of .ciopfs. This looks like Wine's directory case-sensitivity check in ntdll: when statfs reports a FUSE filesystem, Wine looks for a ciopfs marker. The result doesn't appear to be cached, so the check repeats for every lookup, roughly 50 times per file during upload preparation. shfs is FUSE, and its statfs is slow, presumably because it aggregates across every disk the share spans. On XFS and ZFS the same call takes about 20 µs, and the .ciopfs probe never runs.

Evidence

All figures come from strace -c -w -f -p <pid>, which reports wall-clock time per syscall, unless noted otherwise.

1. bztransmit -prepare_bzcombs on user-share drives, 30 seconds:

   • fstatfs: 2,418 calls at 5,450 µs average, 13.2 s total, 48% of wall time

   • read (mostly wineserver replies): 11,555 calls at 938 µs average, 39% of wall time

   • newfstatat: 1,305 failed calls; ioctl: 2,439 failed calls; getdents64: 1,026 calls

   • Throughput: about 1,740 files per hour, 0.3 Mbit/s, monitor showing 0.0 threads, with the CPU and network mostly idle

2. The same process, strace -f -T -y -e trace=fstatfs for 20 seconds, grouped by path:

   • About 520 calls on shfs-backed drives, totaling about 10.8 s, about 21 ms per call

   • About 2,050 calls on ZFS paths (/config, exclusive shares), totaling 0.04 s, about 20 µs per call

   • The failed newfstatat calls were all ".ciopfs", apart from the client's own .tmp status files.

3. bzfilelist scan on user-share drives:

   • About 185 entries per second

   • fstatfs at about 17.9 ms average

   • About 63 .ciopfs probes per second

   • read at 0.7-1.5 ms average

4. Native statfs on the host, measured as 500 calls in a single stat -f process to remove process-start overhead:

   • /mnt/disk18: 45 µs

   • /mnt/user/Nick (an exclusive share, which resolves to ZFS): 202 µs

   • /mnt/user/OldD and /mnt/user/Backups (spread across array disks): 21,586 µs and 21,416 µs

   For comparison, a per-file stat on the same files: 40 µs through the disk path, 549 µs through /mnt/user.

5. winedevice's periodic fstatfs sweep of the drive roots: 20-28 ms on each shfs drive, about 20 µs on each ZFS drive.

6. The same data mapped through disk and pool paths:

   • Scan: 1,042 entries per second (5.6x), with a full scan finishing in under 15 minutes. fstatfs 18 µs, read 19 µs, 0 .ciopfs probes.

   • prepare_bzcombs: per-entry work rate 4.8x higher. fstatfs 18 µs (0.7% of wall time), read 83 µs (16%), 0 .ciopfs probes.

Interpretation

The .ciopfs probe accounts for about half of the upload-preparation time on user shares. read latency (wineserver round trips) is also 10-50x higher on user shares. My guess is that wineserver itself sometimes blocks on shfs calls and stalls every client, but that's an inference: I didn't trace wineserver directly.

Suggested change

Cache the case-sensitivity result per filesystem (keyed on st_dev), or at least per directory, so the fstatfs and .ciopfs check runs once instead of on every lookup.

Edited Monday at 03:25 PM2 days by nick5429

@foz I'm not seeing any errors but when I switched to per disc backup, one of my drives N: doesn't show up in the status tab and says nothing is marked for backup but it's already backed it up. Doctor scan reports nothing selected for backup.

Update: Installed beta 269, and the issue appears resolved now.

Edited Tuesday at 10:22 PM1 day by rogman
working now