foz

Ah, shows how long I've been working on the latest round of changes with this page open - I hadn't refreshed and seen these new messages! Re: the diagnostics bundle @gandalf15 thanks, it helps. The bundle is from build 261, so it shows how things were before the fixes: The service died once a day, about 19 hours apart. That’s the file-handle leak from GitHub issue iamfoz/backblaze-64-personal-wine-container#13. After each death the service couldn’t start again for 30 to 60 minutes, until the watchdog cleared the stuck pass. The “no selection entry” warnings for E: to V: were the false warning fixed in build 279. That’s the one you saw disappear. The current beta fixes the leak itself, so you shouldn’t see those daily stops any more. It also found a file under appdata/Notifiarr the container can’t read. If you don’t need appdata backed up, an exclusion for it is the simplest fix. Re: orphaned bztransmit and the failed file check @rogman, sorry, both of these are down to my bugs. Between 6th and 8th October the watchdog had a rule that stopped Backblaze processes it thought had been left behind. Under Wine every process looks that way, so it was stopping the live upload process about 24 minutes into each pass. That’s the hourly pattern you’re seeing. The rule was removed on 8th October, so rather than a timeout setting, please update to the latest beta (build 301) and the messages will stop. I think the failed file check came from the same thing. The file it names, bz_done_20261007_1.bzff, is the client’s record of what it uploaded on 7th October. Backblaze’s servers check the size of each one, and wrong_size_bz_comb means the copy they hold doesn’t match. That’s what you’d expect if the upload of that record was cut off part way. The client keeps a log of these and sends the file again on later passes. I’m seeing the same failure on my own server since the 7th October kills, so I’m watching to see how it clears and will report back. I’d leave bzdc_filecheck.xml alone. It’s the server’s verdict written down locally, so editing it only hides the message, and the next check rewrites it. In the meantime, you can switch that warning off under Settings > Warnings and events. Dismissing it should keep it hidden until you press Reset warnings. If it comes back after a plain dismiss, can you tell me which build you’re on? The file you pasted also includes your hguid, Backblaze’s ID for your computer. It’s best kept out of public posts, so you may want to edit it out.

It’s been a busy few days, mostly on the beta. I started work on the issues raised here, then found more bugs to squash, and a filed issue on GitHub also needed attention. The file-handle leak Client 10.0.3.1075’s service opens its Backup.sql database every few seconds. Under Wine, none of those handles ever closed, so they built up until Wine hit its 40,960 open-file limit. On my server that took about six hours, and on the reporter’s about nineteen. After that, nothing in the container could open a file. Backup passes hung, and eventually the service died. If your bzserv has been stopping about once a day since 1075, this is why. The cause was in Wine. SQLite keeps its database locked, and Wine was holding every closed handle on a locked file open because of how Linux file locks work. The beta (build 299 onwards) is now on Wine 11.19, with a patch that uses a different kind of Linux lock, so it releases closed handles. On my server, the count for that file now holds at 4. On the old build, it passed 400 within twelve minutes. 11.19 also fixes the 11.18 start-up fault I found when it was briefly packaged with the beta image. Both images also have a fallback: With Automatic recovery on, the watchdog restarts the Backblaze service when Wine’s open files reach 85% of the limit (FD_RESTART_PCT). bb-doctor now reports the count against the limit. The stable images stay on standard Wine, so they rely on that restart. If you’re using /config on /mnt/user/appdata/..., consider moving it to /mnt/cache/appdata/.... Every file Wine holds open is also held open by shfs, which shares one limit with everything else on /mnt/user. In the issue, the leak ran shfs out, and a PostgreSQL container started crashing. A bug in autorecovery Between 6th and 8th October, the watchdog had a rule meant to stop stray Backblaze processes. It decided a process was stray when its parent was the container’s init process. Wine starts every Windows process that way, so the rule stopped scans and, once, a live upload process, and the pass hung waiting for it. Nothing was lost, because the client redoes the work, but if you pulled an image in that window and have Automatic recovery on, please update; the code is gone. Other changes in the beta An Exclusions panel on the Settings tab (more on this below). bb-doctor names any directory the scanner couldn’t open, with the reason. Windows rejects names containing ", and names ending in a dot, and the container user can’t read some ownerships. When the scanner hits one of these, it skips the folder silently, and the only sign was “producing file lists” for hours. On my server, every scan for three days had been stopping at one folder with quotes in its name. The Monitor shows a warning while such a folder exists, and it can be turned off in Settings. bb-doctor warns when a mapped drive’s folders are separate filesystems. The client doesn’t cross mount points. On Unraid, each share on a ZFS pool is its own dataset, so mapping /mnt/cache as one drive backs up nothing. Map /mnt/cache/<share> instead. bb-doctor’s drive selection check now reads the client’s own volume list (see the reply on drive N: below). The watchdog now starts the Backblaze service itself if it has stayed down for more than one check. Two people had it sit down for hours. Diagnostic bundles now include the recovery log and the end of the service’s own log. The Status tab’s drive bars are in drive-letter order and line up. They no longer jump around when the ETA changes width. The stable images moved to the latest base image (v4.14.0) and picked up the watchdog and bb-doctor changes. They’re still on standard Wine 11.0. Re: mapping disks, the XML exclusions file and the strace results @nick5429, thanks for taking the measurements, which found something I’d missed. Every time Wine fails to find a file under the exact name it was given, it checks whether the folder is case sensitive. On a FUSE filesystem such as shfs, part of that check is a statfs call. shfs answers it by adding up free space across every disk the share spans, so each call takes about 21 ms. Your trace had it at nearly half the wall time of a pass. The beta (build 299 onwards) now carries a patch that makes the decision once per device and caches it. I’ve moved my own server to disk paths, so I haven’t been able to measure it on user shares yet - I'll be adding them back after my backup completes. If you still have the user-share container, I’d be glad to see a before and after. Mapping /mnt/diskN and the pool shares is still the faster setup, since shfs is slower at everything else too. I’ll be raising the Wine side upstream: there’s an existing Wine bug for slow FUSE filesystems that this adds weight to. On the exclusions file: you were right that it’s the sensible way to exclude a folder across twenty disks. The Settings tab now has an Exclusions panel that manages the rules in bzexcluderules_editable.xml. It only touches its own rules between two marker comments, so your hand-written rules stay exactly as they are. It keeps the previous file alongside, and its rules are included in the settings export. One tip for anyone switching to per-disk drives: the client abandons a drive selection saved while it’s uploading (there’s just a line in its log). Pause the backup, untick the drive and press OK, tick it and press OK again, then resume. Before mapping /mnt/user alongside the disks, check that /mnt/user/.bzvol doesn’t show one of the disks’ stamps, because the client recognises drives by that stamp. Re: drive N: not showing on Status, and bb-doctor saying nothing is selected @rogman, glad it sorted itself out. I suspect two things lined up: The Status tab only shows a drive once the client has written a file list for it, and after a switch to per-disk drives the client rescans each disk, which can take a while. bb-doctor’s selection check was looking in the wrong place. It looked for a per-drive entry the client only writes when you change a drive from its default, so a drive could be selected and still read as “nothing selected”. From build 279, it reads the client’s volume list, the list the client works from. If it comes back on a current beta, a bb-report bundle would help me pin it down.

Yes, this is an issue I had discovered during bug fixing. The beta build of 28 September, build 257, the one your bundle shows, has a bug in the part of the container that restarts the Backblaze service when it dies. The first time the service stopped after that build, the restart routine called itself in a loop instead of restarting anything. The watchdog saw the service was down every half hour and handed the job to a routine that was no longer working, which is the pattern in your timeline. The fix has been in :beta since 30 September, and today's beta adds a second line of defence: if the service is still down half an hour after the watchdog first reports it, the watchdog starts it itself instead of waiting on the service watch. Pull the latest beta and recreate the container from the Unraid template: docker pull iamfoz/backblaze-64-personal-wine-container:betaThe broken routine in your running container is also putting load on a CPU core, and has been since the service died. Recreating the container ends that as well. Your pass had been waiting on its upload children for 37 minutes before everything stopped, the same as on my own host the day before. I am looking at what ends those waits. A diagnostic bundle from the new build after it has run for a day would help, since from this build the bundle includes the recovery log and the service's own log. Hopefully the changes have squashed this bug!

Thanks for the bundle, it helped me narrow down the bug quickly. Your backup is fine: the latest pass went all the way through and recorded its files, and nothing in the transfer logs mentions an inherit. What happened is that your client never wrote the final "done" stage into its progress file; it stopped at "after files swap, 60%", most likely because the container was restarted during the last step, and my code treated anything without a "done" message as still running. That is why the notice, the terminal monitor and bb-doctor all kept reporting the apparently still in-progress state. The next beta now reads it the way the client behaves: a progress file that hasn't changed for an hour while passes have run since counts as finished. The notice goes away, and bb-doctor notes that the inherit finished earlier and where its file stopped. Once you update to the latest release, these messages should clear automatically. My own backup is only about a quarter of the way through its first backup (it’s quite large!) and I couldn’t inherit as my prior client was a Mac, so this scenario hadn’t come up for me; I’m glad for reports like yours - the software should work well for everyone!

Interesting little rabbit hole to go down, but relatively straightforward to correct: The two numbers are both right; they just count different things. The container figure is the kernel's memory charge for the container, and that includes the page cache for every file the client has read. Your dedup scan reads every file on those disks, so the cache grows into the tens of gigabytes and gets booked against the container. Unraid's dashboard counts cache as free, which is why it shows 14 GiB used and 43 GiB free at the same moment. The client's processes themselves are holding a few gigabytes, and the cache gives way to anything else that needs the memory. So nothing to worry about, and it will shrink once the scan is through. The next beta shows this on the Monitor: the processes' own memory on the gauge, with the cache figure beside it, so the figures are clearer. It also has a cleaner bb-doctor check on Swap - if the host has a reasonable amount of RAM (I went with 16GB), the line will report as info only if swap is still not configured. However, if repeated OOM events occur, this will turn back into a warning to hopefully keep the diagnostics relevant and useful.

Hi @rogman, I happened to be doing some bug fixing (there seem to be lots of edge cases where the backup can hang to squash) and the drive check was hard-coded to 3 levels, which will easily miss files if you've got things laid out with deeper categorisation. I've left the 3-level setting as the default, as this should do for most and is a good balance for speed, but in Settings in the next release (beta 247), you will be able to alter the depth yourself or select unlimited search with a 60-second timeout per drive. Edit: The swap message is for OS swap, not the temporary drive setting in Backblaze. If you've got plenty of RAM, then this shouldn't be an issue. The problem is that Backblaze can peak at some extremely high usage, so it's hard to say what a reasonable cutoff is. I'll have a think and maybe make this some sort of setting too. If there's a reasonable calculation I can make with regard to historic usage, then it might be something I can automatically mark as OK.

Thanks for confirming. The ten-second idle is the parent process reading the next chunk. Your log from before the change shows the client had sixteen upload slots open, and each child finished its 10 MB chunk in about three seconds, but the parent only handed out a new chunk every two seconds. Before each launch it reads the next 10 MB from the source drive, hashes it and stages it, and two seconds for 10 MB is a 5 MB/s read. However many threads you allow, the parent cannot feed them faster than it can read. Your drives are mapped through /mnt/user, so every read goes through Unraid's shfs layer, and long sequential reads like these are what it is slowest at. Change the container's drive mappings to the disk or pool path underneath, /mnt/cache/share or /mnt/diskN/share rather than /mnt/user/share, and do the same for the temporary drive. That takes shfs out of the read path. Then watch the chunks per minute on the Monitor; the parent's cadence is what should change. Before changing anything, pull :beta again to update to build 236 and run bb-doctor. The Source drives section now reads 64 MB from a large file on each drive and prints the speed, with how many chunks a minute the pass can stage from it. Under 20 MB/s on the drive your big files live on and the mapping should fix it. If it comes out fast, the bottleneck is somewhere else. In this case, if you could give me another diagnostic bundle, I'll investigate further.

Turns out it’s not a problem with your drives; it's a bug in the check I wrote, sorry. Your client keeps one volume record for every drive that has ever been assigned a particular letter, and the check compared your stamp against the first record instead of all of them, then found it under D: anyway and printed nonsense. The ID in that warning is the one your own client log reports as attached, so D: is fine. The fix is building now; pull :beta again in half an hour and bb-doctor should read every drive properly. Good to know D: is cache-only. Then the write failures at 03:11 weren't the mover on that share, and the thing to look at next is where the client's own data folder lives, which is C: inside appdata, and whether anything moved or locked files under /config at that time.

@rogman A settings export and import function has been added to the system. It allows you to specify an encryption password to export and import API tokens and notification service strings as well. It also stores Backblaze's own configuration and will restore that to a fresh container too, but this will not store or restore the backup state itself - that remains in appdata and/or the drive .bzvol directories and must be backed up separately or you need to use the inherit state function. @gandalf15 Thanks for the bundle. It has two issues, and the second is the one that needs to be fixed first. Your backup has been stopped since 03:11 this morning, not just slow. At that moment, the client failed to write to its own data folder and to the scratch folder on D:, lost its four-hour lock, and no pass has run since. Those errors are between 03:01 and 03:11, which looks like the Unraid mover moving files out from under the client. Check the mover schedule, and make sure the appdata share and whatever share holds Backblaze's scratch folder are set to cache only. Then restart the container, and update to the current :beta while you're at it: it now notices a dead service or a stuck pass and restarts them, whereas build 199 sat there saying all was OK. The slowness is the scratch folder. Backblaze's large-file path deletes and rewrites every chunk through that folder on D:, an shfs user share, before each upload child starts, and that is why only two or three threads are ever busy and "Preparing" takes so long. Point the client's temporary data drive at a cache-only share, or better a disk path, and the parent stops being the bottleneck. On the D: warning: that one was my check's fault, and it's fixed in the current :beta. bb-doctor looked for a volume ID incorrectly, so it warned about D: on a stray token and said nothing about the drives with a real problem. Your own client log lists F: and G: as not attached, with no volume ID, which means those two are not being backed up whatever the settings window shows. Run bb-doctor on the new build and it will say, per drive, which of four states it is in: the client knows the ID under another letter, the client has a different ID for that letter, the drive is stamped for a different computer identity, or the client has no record of the drive at all. For the middle two there is a value the client itself wrote elsewhere, and bb-doctor --fix restores it into the drive's .bzvol stamp, keeping the old stamp beside it. Restart the container afterwards so the client re-reads it. For the last one there is nothing to restore: untick and re-tick the drive in the client's settings, and expect that drive's upload to start over. Whatever it says, never delete the .bzvol folder on a drive. Backblaze's own README in there says the drive's backed-up files go with it. While you have it running, --fix also now starts the Backblaze service if it has died, stops a pass that is stuck waiting on a lost upload child so a fresh one starts, and rewrites the supportedOS manifest if it is missing. Plain bb-doctor reports all of those without changing anything.

New beta build with the things reported over the last day or so, and some other odds, ends and bugfixes. @rogman, the Status tab went blank in build 207 due to a stray apostrophe in one of the notices that broke the page's script, that's been fixed. As per your request, client update notifications are in. Warnings on the Monitor now have an X to dismiss them, one per warning, and warnings dismissed on the Status tab no longer show on the Monitor band either. Two can't be hidden, because both mean nothing is being backed up: a safety freeze, and passes losing their four-hour lock. Dismissed warnings stay listed in Settings until you press Reset warnings. Settings now has one "Warnings and events" list in place of the Events list under Notifications and the separate Warnings box. Each warning has a Show box and, where there's a matching notification, a Notify box. Reset warnings puts dismissed warnings back without touching those choices. On a phone, the "Preparing" and "Finishing" line in Uploading now wraps instead of pushing the panel off the screen. The Status panel title no longer sits on the top bar, and the warnings list is one card per warning rather than a stack of loose lines. The Backup client section in Settings fills in once the first client reading arrives. It used to ask once when the page opened and then give up, so a page opened in the first minute after a restart stayed empty until you reloaded it. @anna, on the timestamps being two hours behind: Backblaze writes its own log files in UTC, whatever TZ is set to. The dashboard converts to local time where it shows log events, and the container's own logs follow TZ, so it matters where you saw it. If it was in the raw Backblaze logs, through the Tools tab, bb-report or a shell, that's expected. If it was on a dashboard screen, in the terminal monitor or in one of the bb-* tools, tell me which one and what time it showed against the time it should have been, and I'll fix it. All the 'pretty' user-facing UX should show you everything in your local timezone (or whatever you've set your TZ to). Pull :beta and it's all there. As usual, any issues or requests, let me know. As I mentioned previously, I've pushed a fix for the client issue to stable, but if all of this work in the beta is holding up, I'll look to do a proper release soon, as there's now quite a gap between the two now.

Follow-up on the "producing file lists" problem: I've found why client 10.0.3.1075 fails under Wine, fixed it in the beta, and the beta now runs 1075 with passes completing. Stable has a fix of its own. Details and what to do below. What 1075 does differently. Its service, bzserv, checks on each backup pass every ten seconds by opening the bztransmit process. The pass locks its own process down so that nobody can open it. On Windows, the service gets in anyway, because it runs as the system account, which holds a privilege that overrides that lock. Wine didn't honour that privilege, so the check failed, bzserv decided the pass had died, deleted the four-hour lock and started a new pass, and the old one, still running, gave up when it found its lock gone. 1069's service never does the check, which is why it always worked. So one correction to my last post: the lock file was not on disk the whole time. bzserv was deleting it, and tracing which process did that is what led to the cause. The beta's Wine is built from source, so I've added the missing behaviour there. On my server, a 1075 pass now runs from STARTBACKUP through to TransmitFilesInOneTodoFile with no lock error, and the pass-check command from my last post shows the healthy sequence. The current :beta has that fix, Wine 11.18, a newer base image, and no client pin, so it follows Backblaze's newest client again. Stable uses Wine from WineHQ's packages, which I want to receive the patches via inclusion in the source, so stable v10.2.2 (out today) pins the client to 10.0.1.1069 and stops the updater. If you're on stable and restarted since mid-September, you have 1075 too, and updating the container puts 1069 back on its own. No template changes needed. If you're on the beta: If you did the manual rollback and set Disable Auto-Update to true: pull :beta, then either leave that setting to stay on 1069, or set it back to false and restart to move to 1075. Both work on the new beta. If you'd rather hold one exact version, the BACKBLAZE_VERSION variable does that and wins over the update settings. The template you installed from doesn't have it, so add it by hand with "Add another Path, Port, Variable, Label or Device" on the container's edit page. Leave it empty to let the image decide. @anna, the lock error after an unclean shutdown is different from this one. A pass that dies without releasing its lock leaves the next passes failing to grab it until the client clears it, which it does by itself after a while. The repeating one, every pass, every minute, was 1075. If passes still don't complete after updating, run the pass check command from my last post and paste the result, along with a diag bundle.

Hey all, busy as usual so apologies for the delay, but I've got to the bottom of the "producing file lists" problem, with thanks to everyone who posted logs here and on GitHub. The cause is the Backblaze client itself, version 10.0.3.1075, installed by the auto-update function. Backblaze released 1075 in the same week as beta 181, and the container's updater installs the newest client on every start, so anyone who restarted since about 14 September got it. Under Wine, 1075 loses its own four-hour lock on every backup pass. The lock file is on disk; the client's check seems to miss it. The pass aborts at the transmit step, bzserv starts another a few minutes later, and that one does the same. Meanwhile, large files keep uploading through the chunk path, so the rate looks fine and bb-health says OK. The GUI stays on "Producing file lists" as that's the stage every pass restarts from. To @rogman's question about the coincidence: it was both at once, and I've separated them. I ran 1075 on a container with the monitor unable to touch Wine at all, and it failed the same way. 10.0.1.1069 on the same image completes passes. The GitHub reporter's bundle shows the same thing from the other side: every push line before their switch carries 10.0.1.1069, and their startapp.log shows the container installing 1075 at the moment they pulled the new image. @Shadowrunner was close. The file stays on disk throughout, and the client fails its own check for it. Creating it by hand doesn't help for the same reason, and the "Failed to grab fourHourLock" lines that follow are the next passes finding the lock the dead one left. @anna, I'd expect yours to come back; a lucky pass doesn't necessarily indicate the client is working. The fix is in the beta that's building now: The container pins the client to 10.0.1.1069 by default and installs it over whatever is there, in place. Pull :beta, restart, and the next pass should complete. There's a new variable, BACKBLAZE_VERSION; set it empty when you want the updater back. The Status tab, the terminal monitor, bb-doctor, the metrics and notifications now warn when passes are losing the lock, so this can't hide behind a healthy-looking upload rate again. If you don't want to wait, this does the same thing on the image you have. It copies the 1069 files over the install and runs Backblaze's installer. Backblaze64 is the template's default container name; substitute yours if you changed it: U="$(docker exec Backblaze64 stat -c '%u:%g' /config)" docker exec -u "$U" -e HOME=/config -e WINEPREFIX=/config/wine/ Backblaze64 sh -c 'cd /config/wine/dosdevices/c:/ && curl -fsSL "https://secure.backblaze.com/api/install_backblaze?file=bzinstall-win32-10.0.1.1069.exe" -o install_1069.exe && rm -rf bzextract1069 && mkdir bzextract1069 && 7z x -y -obzextract1069 install_1069.exe >/dev/null && d="$(dirname "$(find bzextract1069 -iname bzdoinstall.exe | head -1)")" && b="/config/wine/drive_c/Program Files/Backblaze" && cp "$d"/bz*.exe "$d"/*.dll "$d"/*.xml "$d"/*.gif "$d"/*.ico "$d"/*.txt "$b/" 2>/dev/null; cd "$d" && wine bzdoinstall.exe -doinstall "$(winepath -w "$d")"; echo "exit=$?"'Then set Disable Auto-Update to true in the container template before it restarts, or the updater puts 1075 straight back. Please don't delete Program Files\Backblaze to force a reinstall. The machine identity lives in bzinstall.xml in that directory. Without it the client treats your server as a new computer and you have to inherit your backup state, which I learned the slow way tonight. Copying over the top keeps it. To check a pass is completing, this shows the last few pass events from the transmit log: docker exec Backblaze64 sh -c 'cd "/config/wine/dosdevices/c:/ProgramData/Backblaze/bzdata/bzlogs/bztransmit" && grep -h -E "STARTBACKUP|Finished B reading|TransmitFilesInOneTodoFile|Lost four hour lock" "$(ls -t *.log | head -1)" | tail -8 | cut -c1-120'Healthy is STARTBACKUP, then Finished B reading, then TransmitFilesInOneTodoFile, and no Lost four hour lock after it. This will keep everything working alongside the most current beta, I will of course be investigating what is causing trouble with the new client and seeing if its something we can ameliorate. If you're still having problems, let me know, and I'll investigate further.

Thank you, and it is my pleasure. It's gratifying to see people getting value from my work. I think you've already noticed the update, but I added some specific templates for the various popular notification tools as well as the option of building your own output - my fault for half-assing it a bit in the first place. A few other bug fixes and cleanups as well.

@rogman, not selfish at all. A very useful request, and one that broke me out of my (slightly nostalgic) CLI reverie. Beta 156 has a Tools tab. It runs bb-doctor, bb-health and bb-version from the page and shows what they print and allows you to interact with options and outputs as you'd expect. Notes are on the page, but let me know if there are questions. A bug. docker exec enters the container as root, and root passes every permission test. bb-doctor decides whether a skipped file is readable with a permission test, and whether /config is writable with another, so from the console both always said yes. If your share had the wrong owner, which is the most common fault this container has, running bb-doctor the way the README told you to said your files could be read. The Tools tab runs as the container user, which is the user whose permissions matter, so it gives the true answer. And bb-doctor itself now re-runs as that user when it is started as root, so the console gives the true answer as well. If you ran bb-doctor from the console before 150 and it said your files were fine, it may have been wrong. Status tab. Skipped Files is now Status, and it leads with everything the client says needs attention: warnings, a pause with the reason, a first backup running or finished, and then the skipped list. The pause part is the one I care about. The client writes a reason code with every pause and we were showing a flag. Now it says "Paused from here" when you or the API paused it, and "Paused by the client" with the cause when the client chose to, for example when Backblaze's cluster authority is not answering, which is usually their maintenance and resumes on its own. One is a button, the other is a wait, and the page says which. A Copy status summary button gives five lines of plain text for a post like this one: build, state, progress, health, today's uploads. Nothing in it names a file. Notifications. The API tab has a Notifications section. Point it at an ntfy topic or a webhook that takes JSON, tick the events, and the container tells your phone when the backup needs you: a safety freeze, files skipped past a threshold, no completed backup within the client's own limit, a stall, a pause the client chose, first backup complete, a milestone. Once when it starts, once when it clears, never on every poll, and nothing that names a file is ever sent. A Test button checks the endpoint. Quiet hours. Also in the API tab. Windows by day of the week with a start and end; the container asks the client to pause at the start and starts the backup at the end. The client's own pause lasts about two hours, so inside a window the container renews it. If you start the backup yourself during a window it stays running until the window ends. bb-doctor. Two new checks. It samples each mapped drive as the container user and names anything it cannot read, with the owner and the chown line, before the client has to give up on files. And it compares the volume ID the client wrote under .bzvol with the volumes the client knows about; a drive it no longer recognises is the "No files are selected" state after an inherit, which the last thread hit and nothing named. bb-health reports FROZEN when Backblaze has frozen the backup, so the Docker tab goes unhealthy for that state. API. A new switch allows you to turn it off without revoking keys; off, everything returns 404 and the keys are kept. Revoked and expired keys have a Delete button. New endpoints: /summary, /metrics in Prometheus format, /events as a server-sent stream on state changes, /tools behind new diagnose and diagnose:repair permissions, and /openapi.json. The OpenAPI document is also in the repository. Smaller things. All thirteen themes now apply to every tab; before, the Status, Tools and API pages used the default. Text uses the width of the window instead of wrapping at about 375 pixels. Keys: 1 to 5 switch tabs, s opens settings, p pauses or resumes. The browser tab title carries a mark for uploading, paused, or a warning. The desktop pane shows a Reconnect button when noVNC drops its connection, which after a container restart used to be a blank pane. A Today line under the seven-day chart, and a progress-over-time line in About.

Hi @YourNekoWaifu, welcome - this is normal when the client restarts and is most noticeable on large backup sets. The client will scan the set, which involves putting a file list into memory. My own set is 90TB, and my memory usage will hit 10GB easily during this phase. As @rogman said, the best way to manage this is to either have a large amount of RAM on hand or configure Swap. I'm currently running off of an N100 stick computer with 12GB of RAM and Proxmox, so the Backblaze process can easily overwhelm the host, which I've resolved with a Swap partition. After the file processing and the backup begins, memory use will fall back. Thread numbers will also contribute to this pressure. Backblaze will prioritise files by size, so each run will always start with new and changed small files, then go on to larger files. After about 100MB, they are chunked by 10MB pieces. Each piece will use a thread to upload, and this is when users with decent upload bandwidth will find the most effective use of that bandwidth comes into play. Hi @DevanteWeary, welcome. Good news first: nothing is deleted. A Safety Freeze is Backblaze protecting a backup when something looks wrong with the drive underneath it. Your data is still on their side, and the freeze stops any sudden catastrophic changes. Backblaze have a page on clearing one yourself: https://www.backblaze.com/computer-backup/docs/resolve-a-safety-freeze Read it before you touch anything. The last step is re-licensing with Inherit Backup State, and that's where it's possible to pick the wrong computer record and lose the link to the backup you're trying to save. You've been through that screen once already, so you know it. Just be careful which entry you choose. The uninstall and reinstall step is easier here than on Windows. Delete /config/wine/drive_c/Program Files/Backblaze/ and restart the container. It downloads the client again and reinstalls it on the next start, which is the same thing it does when a version update comes through. Your appdata and the client's own data are in a different place and aren't touched by that, so the licence and the inherit history survive. What I can't tell you is whether that's enough. Backblaze's uninstall step may be meant to clear the client's local state as well, and the container's reinstall keeps it. If the freeze is still there afterwards, say so before you go further, because the next thing anyone would reach for is wiping that state, and that's the part with real risk attached. Before any of that, I'd fix the thing that probably caused it. The first two symptoms were a red herring. "No files are selected" and "Initial Backup Paused" is what the window says while the client is still building its file list, which on a large share takes hours. You saw that resolve itself: correct size, correct count, then "preparing files". Nothing was wrong there. The permissions skips are the part that matters. New container, new appdata, so User ID and Group ID are back at their defaults of 99 and 100. If your shares are owned by something else, the container can't read them, and a large slice of an existing backup suddenly going unreadable is the kind of thing I'd expect to look like an altering event from Backblaze's side. Hypothetical rather than diagnostic, but it fits both things you've reported. On the latest beta, run this: docker exec backblaze-personal-wine bb-doctorIt reads the client's own skipped-file list and works out, for each file, whether it's actually gone or just unreadable from here. Where it can't read something, it prints the directory with its owner and mode, next to the UID the container runs as. If it comes back with a list of directories, that's the answer. Fix ownership on the host with the command it prints. Do it on the host rather than inside the container, and don't blanket-chown the whole share. Get that clean first, then work through Backblaze's page. On .bzvol: today's date on those files is normal, the client updates them. The 2023 date on bzscratch and the empty bzcurrentlargefile are normal too. That's the staging area for files over 100 MB, and it's empty because nothing large is in flight. @gandalf15 I think yours is a different thing. "Initial Backup Paused" with a Continue button is the client's own pause rather than a freeze, and clicking Continue is the right answer. If it starts happening regularly, I can take a look, but it is normal for the client to cycle through these states by itself from time to time. And as noted, I've been working on some updates and bugfixes on the beta channel, here's what's changed: The memory gauge now names the program using the memory. A container partway through its first backup announced that the backup was complete, and latched that to disk for a week. The completion check shared a helper with the staleness warning, and that helper answers "yes" when there are no totals to judge against. Right for a warning, where a missing figure should not raise a false alarm. Wrong for declaring something finished, and during a file-list scan the totals are absent. It now needs positive evidence. ETA sanity took a pass. A backup starts on its small files, and a handful of those gives a per-byte rate you can extrapolate into geological time. One run read "4259966d", about 11,600 years, and the trend feature then compared today's nonsense against yesterday's. Estimates beyond a century are withheld now, and the trend ignores anything resting on fewer than three completed transfers. A grim estimate still shows if it is real: 100 TB on a 1 Mbit/s uplink is about 25 years, and that is accurate. With nothing usable, it reads "not yet". The failure counter no longer says "failed" as this was misleading. The number it showed is attempts a storage vault turned away for being busy or full. The client retries against another vault and the file goes up. It reads "retried" now, without the alarm colour, and the API carries the breakdown by reason. The count that means data is not backed up is the skipped-file count, which is reported separately and loudly. Pause reporting got two corrections, and one of them was me being wrong. The monitor said "Paused" the moment the request landed, while the client was still finishing the transfers already in flight. Backblaze's own window looked like it had not noticed, because it was finishing up. A pause reads "Pausing" until it takes. The monitors also said "Preparing <file>" next to a state of Paused, which reads as stalled; the client parks on the file it was about to take, so while a pause is set that is the next file rather than the current one. This may be what you were experiencing with the API @rogman - let me know if this is still a problem. Backblaze keeps a list of files it has given up on, with a reason against each one. It does not queue them, it does not retry them, and nothing in the desktop window tells you they exist. A backup can look finished while a directory's worth of files was never sent. The beta surfaces that list in three places now: a warning in both monitors, a Skipped Files tab in the web interface with a filter and a breakdown by reason, and bb-doctor, which goes further and works out why. bb-doctor reads a sample of the list and sorts it into three groups. Files that no longer exist, which need nothing done. Files that are readable again and will clear at the next scan. And files this container still cannot read, where it names the directory, prints its owner and mode next to the UID the container runs as, and gives you the command to run on the host. It repairs nothing, even with --fix. These are your files on a mounted share, often thousands of them, and other software on your host may depend on who owns them. Two bugs found and fixed in that check: the list is written by a Windows program, so every line ends in CRLF, and the shell left the carriage return on the end of each path. The same check counted the report's own header as a skipped file, so a clean list reported two. The monitor was never affected, because Python strips line endings where the shell does not. Smaller things. Buttons and chips in the dark themes had near-black text hardcoded on them, around 1.5:1 against dark grey, so they were effectively invisible. bb-monitor run without a terminal gracefully exits and indicates the issue instead of dying in a curses traceback, which is what docker exec without -it gets you. There is a seven-day upload chart, a breakdown of what the backup is made of by category, and a "backing up since" date.