-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Sorry I've been away for a bit, life likes to keep me busy! The Issues tab won't show them, which is why the monitor reads them itself. The client keeps its own list that nothing in the GUI surfaces: docker exec Backblaze64 sh -c 'F="/config/wine/dosdevices/c:/ProgramData/Backblaze/bzdata/bzreports/bzlist_skipped_files.txt"; echo "=== reasons ==="; cut -f2 "$F" | sort | uniq -c; echo "=== files ==="; cut -f2,3 "$F"' Substitute your container name if it isn't Backblaze64. With five files it's worth reading the actual reasons rather than assuming, but under this container the common one is BZ_FILE_PERMANENT_BAD_PERMISSIONS, which means the container user couldn't read the file. That points at ownership or permissions on the mounted share rather than anything Backblaze has done. Mine was 770 files in a single directory that had ended up owned by root with 0700; a chown fixed it, and they cleared on the next scan. The thing worth knowing either way is that skipped files aren't queued and aren't retried. They're just not backed up, and until the monitor started reading that file there was nothing telling you so. Post the reasons column and I'll tell you what yours are.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Beta 132 is up and the big thing in it is an HTTP API. @rogman Your dashboard plugin looks awesome! I have had a dashboard plugin of my own on the roadmap, but it has sat well down the list behind the Wine work and the monitor, and I also knew that it would hit a wall: there is no sensible way for a plugin to get data out of this container. The monitor's own feed is behind the web login, its shape is whatever the page needs that week, and I would be changing it under anyone building against it. An API was always the part that mattered, and the plugin was the easy bit afterwards. Then you posted yours, and finishing the API properly became a thing worth doing. That is what this build is: it exists so your plugin has something solid to talk to. I will still finish mine at some point, but it is the smaller job of the two and there is no hurry, after all the API is designed for others to be able to make cool stuff with. That goes for anyone, not only rogman. If you want to build something that talks to this container, the API is there for it, and I would rather you used it than scraped a page - it should be considerably more stable this way. Your tile polls /monitor/api/status, which sits behind whatever Web Authentication the container is set to, so it only works with that turned off. That is not a limitation of your plugin, it is that there was nothing else to point at. There is now: /api/v1/status authenticates with a key instead of the browser login, so it works whether authentication is on or off, and it is stable rather than a page's internals. Same port, no extra ports to open. It also carries a lot more than the page feed does, in raw numbers rather than the strings the dashboard draws, so you can format for yourself: bytes and seconds rather than "19.5 TB" and "3 days". Turning it onIt is off until you make a key, and answers 404 until then, so nothing changes for anyone who does not want it. There is a new API tab in the web interface, or from a terminal: docker exec backblaze-personal-wine bb-apikey create --label "dashboard" --scope readThe key is shown once. Then: curl -H "Authorization: Bearer <key>" https://<host>:<port>/api/v1/statusPermissions are granted per operation. A status display gets read and nothing else: not the names of your files, and no ability to touch the backup. If you want something that can start or pause a backup, that is a separate permission again, and starting is separate from pausing. Full reference, every field and what it means, is in the repository at docs/api-v1.md. What anyone might do with itNot just dashboards. Some things it would support today: A Home Assistant sensor, with an automation that tells you when the backup has not completed in a week. A phone notification through ntfy or Gotify when files start getting skipped, which is the one that would have actually saved me something. A Grafana panel, if you already run one, since everything comes out as numbers. A cron job that starts a backup after your array finishes a parity check. A Stream Deck button that pauses the backup while you are gaming, and starts it again after. Another Unraid plugin, if someone wants a different take on the tile. The point is that the container should not care what is on the other end. If you build something with it and hit a field that is missing or wrong, say so and I will add it. FixesMulti-part totals were wrong again, in a new way: a 221 MB file with 10 MB parts showing as "22/236". The part size is fixed for a file, but one bad reading could set a row's total for good. It cannot now. Also in: the monitor says when a backup is paused rather than reading "Uploading" with nothing moving, a key created on the command line is visible to the service rather than silently invisible, and the dashboard reuses its connection instead of opening a new one every two seconds. @gandalf15 on threads - setting 100 and seeing 7 or 8 is expected: that setting is a ceiling, not a target, and the client only opens what it has work for. Chunks of one file go out in parallel, but the client is not queueing a hundred at once. Where you are setting threads, this must mean you have turned the auto throttle off. Try moving the slider to faster backups if you haven't already, and maybe using the loading trick to see if Backblaze is causing odd things to happen due to lack of system pressure. My own system is currently uploading ~200MB files, which get about 20 threads at 90 Mbit/s. I've set Backblaze to manual throttle, faster backups and max 30 threads. For your 16 MB/s at 17 ms, the arithmetic that matters is per connection rather than overall. Each connection is limited to roughly its send buffer divided by the round trip, and under Wine that buffer is 64 KB, which at 17 ms is about 3.8 MB/s per thread. Eight threads gives about 30 MB/s as a ceiling, so 16 is short of it but the same order. More threads would help you more than anything else, which is the opposite of what you would expect, and it is the send buffer rather than your line. Some of this may change in the future with the work I'm doing with Wine. As always this is beta only. Stable is unchanged at 10.2.1. Quote the build number if you report anything.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Beta 127 is up. There has been a few changes since 110, most of it from going through the files the Backblaze client keeps in bzdata to see what it already knows and does not show you. The monitor reports what the client is doing rather than guessing from which processes are running. The client writes its own status out, and it turns out the field I was reading first (cur_state) says "transmitting" no matter what, including while it is building file lists. The useful one names the file in hand, and for a large file it says which phase it is in: preparing, part 12 of 21, then finishing. A file-list scan now shows a real percentage, directories indexed out of the total, with a running count of files found, rather than just the word "Scanning". Chunked files can now be shown in a compact view. Every chunk of the large file being split has an index, a byte offset and a SHA-1 on disk, and that same SHA-1 turns up in each transfer's own record, so a thread can be matched to the exact chunk it is carrying. Chunks fill in where they actually sit in the file, out of order, with the ones in flight marked differently. Chunks that finished before you opened the monitor stay unmarked, because there is no way to tell those from ones not yet started. It is off by default, next to the theme picker. Files Backblaze has given up on are now listed. The client keeps a list of files it has skipped with a reason, and it neither queues nor retries them, so nothing else tells you they are unprotected. When I turned this on against my own machine it found 770 skipped files. All of them were in one directory that had ended up owned by root with 0700 permissions, so the container user could not read it. About 28 GB I believed was backed up and was not. Under this container, the cause is nearly always ownership of the mounted source rather than anything Backblaze has done, and it is invisible until something asks for the list. If you run nothing else from this build, look at that number. A first backup is still working through the set now says so, with how long it has been going and how far it has got, instead of looking like something is stuck. Mine reads day 67 and 12.9%. There is no "initial backup finished" flag anywhere, so this is worked out from how much of the set has never been sent. Also in: the count of uploads for the day with failures broken out, the bytes compression has saved, and Backblaze's own measured throughput figure. It reports about 3.5 Mbit/s per connection for files over a megabyte here, and 244 kbit/s for small ones, which is a much worse case dominated by a round trip per file rather than by the socket buffer. The web dashboard is mobile-friendly now, so it is usable from a phone. FixesThe "21/6594" part counts are fixed. The count was coming from a field that describes the part a thread happens to be carrying rather than the file's part size, and small readings turned filesize-divided-by-part into tens of thousands. It reads the configured part size instead, which is constant for a file. A total that wrong also meant the record never reached completion, so every later upload of the same file kept piling into it. The compact chunk view showed a film at 0/21 while the client was busy producing file lists, for the same reason: the directory describing the large file being split is not cleared when that file completes. It also drew one row per thread rather than per file, so a dozen threads on one film gave a dozen identical rows. File names with an apostrophe or an ampersand were coming out as raw XML, so "Mike Judge's" instead of an apostrophe. And the compact view toggle did nothing, because the strip it was meant to control was being drawn regardless of the setting. Small files were never appearing in Recently Completed. They are gone before a poll can catch a thread carrying one, and Backblaze pushes them in bundles rather than one at a time, so there is no per-file record of them in the log either. They are now taken from the client naming each one as it works through them, and listed without a thread, size or rate because none of those exist for a file handled that way. That tells you the client moved on from the file, not that a transfer was confirmed. @rogman Those Homarr layouts look cool, having some templates for various dashboards might be nice - something to go on the backlog. Glad the 8099 clash turned out to be it, and good call moving yours to 8098 and writing up the conflict for anyone else who had the earlier version installed. On the credit: it is staying where it is - the web dashboard is in the container because of your initial idea, I think its important to show credit where due! @gandalf15 Thanks for the restore test info, data is always useful! As always, everything above is beta only. Stable is unchanged at 10.2.1 and stays there until this has had a decent run. Quote the build number if you report anything, bottom right of both monitors and from bb-version.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Beta 110 is up with two fixes from this thread. @gandalf15 the clipped number was due to the size formatter stopping at gigabytes, so your 251 TB backup came out as "257524.9 GB" and ran off the end of the label. It goes up to petabytes now, so you should see "19.5 TB/251.5 TB" instead. Thanks for the screenshot. The ETA of 171 days is doing what it should, incidentally. It weights your last ten completed transfers rather than the live sample, so it will settle as more finish. A first pass over a quarter of a petabyte really does take that long. @rogman the dark theme is now your requested values, black rather than the dark grey I had. You were right that it was not really dark. On the monitor tab sticking on "Waiting for the monitor service", I have a suspicion and one thing I would like you to check. Your earlier setup ran your script on port 8099 through a cont-init.d entry in the container template. The built-in service uses that same port, and cont-init.d runs before services start, so if those entries are still in your template your script takes 8099 first, the built-in one cannot bind, and nginx has nothing to proxy to. That page is what you get when the service is not listening, and it will sit there forever because it keeps polling and never gets an answer. Run this and paste the output: docker exec Backblaze64 sh -c 'pgrep -af bb-monitor-web; ss -ltnp 2>/dev/null | grep 8099'If it shows something pointing at /config/bb-monitor-web.py, that is it, and the fix is removing the old Port and Path entries from your template. The container provides both now, so nothing is lost by dropping them. If that is not it, this runs the same code in the foreground so any startup error shows up instead of disappearing into the supervisor: docker exec Backblaze64 /usr/local/bin/bb-monitor-web --dump-rttI do not think SECURE_CONNECTION is involved. nginx talks to the service over plain HTTP inside the container regardless of what it serves to you, so TLS should not reach that far. @DayspringGaming looking good, there is clearly an issue with the data paths in Wine that need sorting out, hopefully it won't be too long before I get a shape of the code that needs changing and I can submit something that fixes it. My original patches were designed to be small incisions; the change that might be coming will be a much bigger piece of work. Still, while that plays out, everyone can get good network speeds from the beta.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Beta build 107 is up, and my implementation of the web dashboard is now part of the container. Click the WebUI button and you land on a tabbed page: the Wine desktop via VNC on one tab, the upload monitor on the other. Switching between them does not drop your VNC session, and the monitor tab does not start polling until you open it. It is on the port you already use, so there is no second port to map and nothing new exposed on your LAN. The terminal bb-monitor has caught up too. It now has the same overall progress and ETA, files remaining, uptime, the assigned upload server, the sparkline, and a settings dialog with the same thirteen themes. Press s to open it, Tab moves between Preferences and About. Both monitors read from one shared data layer now, so a feature will appear in both rather than one trailing the other. docker pull ghcr.io/iamfoz/backblaze-personal-wine:beta and you are on 107 (or if already running, you can just update in Unraid). It'll make its way to stable shortly, assuming we don't find any bugs. @rogman The HTML escaping you added was a great catch. File names went into the page unescaped, so a file whose name contained markup could inject it into the dashboard. Fixed in this beta, and credited in the changelog. On the ping work, where you said "Is this useful? Probably not." - it is. Per-connection throughput is the send buffer divided by the round-trip time, so RTT is the other half of the ceiling I have been going on about. With the 64 KB buffer, my 149 ms path works out at about 3.5 Mbit/s per connection, which is why thread count moves my numbers and line speed does not. The dashboard now shows RTT next to the transfer rate for that reason. I did rework how it is measured though; rather than opening a connection to time it, it reads the figure the kernel already keeps. Every established connection carries a smoothed round-trip time, and there is an interface that exposes it for sockets the process does not own. There is no traffic at all when nothing is uploading, and the number describes the actual upload connections rather than a separate probe alongside them. Your instinct to prefer the live upload connection over a hostname was the right one and I kept it. On the password and self-signed certificate: for a standalone script those were the correct call, and doing them properly with PBKDF2 and a constant-time comparison is good. The container does it differently only because it can; serving through the existing web interface means it inherits whatever WEB_AUTHENTICATION and SECURE_CONNECTION you already have, so there is one set of credentials rather than two. No change on the Wine front since Friday. The maintainers have my reply and I am waiting. The 64 KB lead is still a lead: I have now confirmed the arithmetic on my own hardware, but I still have not measured what a real Windows box reports, and until I do it stays a suspicion rather than a finding. The dashboard works on a phone, incidentally. I hadn't planned on working on it, but it turned out to be about an hour's work once the layout was already there. As ever: if something in this is confusing, wrong, or annoying, say so. Quote the build number from the corner of the monitor when you do.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
A quick update: the upstream work has moved on rather than stalled. I have been going back and forth with the Wine maintainers about how the proper fix should be built, and I have just sent them a long reply conceding most of their objections to my original approach and asking them to pick the design. That is the right order. The workaround in the beta stays until whatever they choose lands, and then it goes. While digging into that I found something else, and I want to be careful about it because I have not finished checking. There is a Windows call that asks the system how much data a connection can usefully hold in flight, and Windows works that out per connection. Wine answers it with a fixed 64 KB every time, and the Backblaze client appears to size its send buffer from that answer. If that holds up it would put a ceiling on each connection that gets worse the further you are from the datacentre, and it would be a separate fix from the one under discussion. I need to measure what real Windows returns before I say anything stronger than that, so treat it as a lead rather than a result for now. @gandalf15 thank you, that is exactly the kind of before-and-after I was hoping for. 8 to 10 MB/s up to 12 to 18 MB/s is a bigger jump than I expected to see reported. On the thread count: that number is Backblaze's decision, not the container's. Your setting is an upper bound, and the client picks how many to actually run based on what it has queued and how the transfers are going. I see it swing between 3 and 17 here across a single evening on the same settings, so 6 is not necessarily a problem. What your numbers do tell me is that you are close to Backblaze in network terms. Six threads at 12 to 18 MB/s is roughly 16 to 24 Mbit/s per connection, and per-connection throughput is governed by the send buffer divided by round-trip time. To get that per thread you must have a fairly short path to their datacentre. Mine is 144 ms and I see about a fifth of that per thread. So your 10 gigabit line is not the limiting factor and neither is file size at 2.2 GB. If the thread count climbs on its own, expect the total to climb with it. @rogman the version I have now is your final one. Compact mode was a good call, I'm looking to build this out further into an API so all sorts of consumers will be able to fetch status and other information on backups. 20 MB/s on 200 MB files sounds right for what the client does with larger files. It splits them further and runs more threads, which is the lever that actually moves the total. More when I have the measurements. Thanks both.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
OK, a few updates: two releases today. The stable one is a pair of fixes. The beta one roughly triples my upload speed. v10.2.1 fixes some little bugs: bb-monitor showed completion times in UTC while its own clock showed local time, so if you'd set TZ, the two disagreed by an hour in the same panel. Backblaze stamps its own logs in UTC no matter what the container's timezone is, and I'd been passing those straight through. Every image now also stamps a build number. The :beta tag is mutable, so I figured it would become useful to know which beta. bb-version prints the number, and the monitor shows it in the corner. If you report anything, quote it so I know what you're running. Beta build 103 carries a second Wine patch. On my connection, it takes aggregate upload from around 13 Mbit/s to 41. First, a correction. When I posted about the writability fix I said that once Wine reported writability correctly, the speed should all be there. I've now measured that properly, and I was wrong. Same host, same thread count, same files, ten-minute windows, alternating builds: my patch alone landed at 13.0, 13.9 and 13.9 Mbit/s across three runs, and with the second patch it went to 40.9. Wine tells an application a socket is writable exactly once and then latches that fact. While the latch is set, it stops asking the kernel to watch the socket for writability at all. If the application learns "not writable" from a poll rather than from a send that failed, it then waits for a notification that can never arrive, because nothing is watching any more. It sleeps until its own timeout expires, every single time around the loop. My first patch fixed the answer Wine gives to select(). It never touched the notification path, and it turns out Backblaze's client uses both: I counted 44 WSAEventSelect and 46 select calls in a single short capture. So I was fixing one of the two paths it depends on, and not the one that governs its throughput. This second patch is temporary. It clears that latch when a poll observes that the socket cannot accept a send. Real Windows does not do that. I tested it on Server 2022 and 2025 before writing any of this, and Windows only re-arms after a send actually fails. So the patch makes Wine diverge from Windows on purpose, which is why it is in the beta and not in the stable images, and why I am not submitting it upstream. CI now fails the build if a patched Wine ever turns up in a stable image, so that separation is enforced rather than something I have to remember. The other similar design from the upstream of this project is a reasonable piece of work which found a real defect that mine did not address. Where mine differs is the trigger. Theirs infers "cannot accept a send" from the absence of POLLOUT, and Linux withholds POLLOUT well before the buffer is actually full, so that fires in cases where a send would have gone through fine. Mine asks the kernel directly whether it would accept the data. It also drops a redundant reselect, because the function already reselects every socket before it returns. Narrower trigger, one fewer moving part. Both of us are papering over the same gap, though, and neither version belongs upstream in wine as written. The proper fix is a larger redesign that I'm working through over at WineHQ now. My measurements turned up some things that are not yet handled, so that discussion has somewhere useful to go. When it lands, the container patches can disappear and the stable images will operate as expected. One other thing turned up along the way, and it applies to whichever build you run. The Backblaze client asks for a 64 KB socket send buffer itself. At my round-trip time that caps a single connection near 4 Mbit/s, no matter how clean your line is, and no patch changes it, because it is the client's own setting, and a Windows machine would hit exactly the same ceiling. What that means in practice: Backblaze scales by connection count, not by connection speed. If you are on a fast line and a long path, your thread count is the lever that matters. Mine is on 30. Be careful with this setting as wine can get funny with lots of threads. Big thanks to @rogman for his initial feature request for bb-monitor to have a web version, and then the python build that followed, because the console has a habit of dying and he wanted to watch a session overnight. He also added overall backup progress with an ETA, and a theme picker, neither of which had occurred to me. I have taken his concept and I am building it into the container proper: it will be served through the existing web UI on port 5800 rather than needing a second port, which means it will inherit whatever authentication and HTTPS you have already configured instead of exposing your file names to anyone on your LAN. The plan will be a tabbed view so the VNC desktop and the monitor sit side by side, with a settings dialog for the themes and whatever else accumulates. That is going into the 10.3 line, and it will appear in a beta before it goes stable. Thank you rogman, this is a better project for it. I've also noted the 99% 'stickiness' - I suspect this is where the client is waiting for confirmation of a successful upload, while I've been poking around, I've been making lots of notes and believe that there are numerous improvements and features that can be teased out of the client. As always, if you have an idea, an itch, or something that annoys you about this container, please say so. It does not need to be code. "This number is confusing" and "why does it do that" are both useful, and rogman's started as a question about a terminal window closing. Update or switch to the beta channel if you want to try the speed patch. Stable is unchanged in behaviour beyond the two fixes above.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
@gandalf15 No problem, if its the same account and an inherited state, it should be able to figure our what is already uploaded from hashing. @YourNekoWaifu That sounds like deduplication and or hash matching - if it still feels off, feel free to run bb-report and send me your logs, I can take a look. @rogman Looking good - I'm currently getting my older, slower speeds for some reason, but after a lot of bug hunting, I think there's some trouble between me and Backblaze, Telia's transatlantic link is showing congestion and loads of retransmission, so hopefully my own low numbers are not related to any of the code. There's movement on the Wine side of things, which may well be good in terms of overall future quality but probably not so great in terms of speedy deployment, I will update on this later.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Following up on my last post, where I'd said testing had turned up a problem. It turned into quite a bit more than one, and the short version: the beta image is now published and carries the reworked fix, it's faster here than anything I'd measured before, and the answer to combining the two fixes turned out to be more interesting than just yes or no. The problem my testing found was in my own patch. The way it measured send-buffer room could, right at the point where the buffer fills, report a socket as writable when a send would actually block. Backblaze never hits that path, which is why months of real backups never showed it. Wine's own socket test suite does hit it, and deadlocks on it, every time. Not shippable. Digging into why led somewhere better. Wine disagrees with itself about what "writable" means: the answer it gives applications comes from one place, and the decision about whether to even attempt a blocking send comes from another, using a stricter rule. The reworked fix uses the kernel's own definition (a socket is writable exactly when a send would be accepted) and applies it in both places. It heads back to the Wine reviewers next: I'll be updating my open submission once it has been through their test service across the Windows versions I can't spin up myself. On correctness, the numbers are already solid: Wine's full socket suite passes with fewer failures than an unpatched build, and the same tests pass on Windows Server 2022 and 2025 with zero failures. Once the updated submission is up I'll post the links here for anyone who wants the nuts and bolts of the actual changes. On combining it with the other fix, I tested that properly, including building the second half and running the test on real Windows. When a socket stops being writable without a send failing, Windows never re-signals it, so making Wine do so would be a divergence, and it's not going in. In principle, once writability is reported correctly the speed should all be there without it. In practice, that's exactly what I want your help confirming. Which brings me to the ask. On my connection, the new beta sustains around 79 Mbit/s, the best I've measured from any version. That's one line, one host, one backlog. It's also one workload shape: what's uploading here right now is 160 MB files, which Backblaze splits into 16 chunks of 10 MB, so per-chunk round-trip times may be what's capping my number rather than the socket path. Once the multi-GB files in my backlog start moving, I'd expect it to climb. The new :beta is published now: same tag, pull again and restart (on Unraid, the Repository field is ghcr.io/iamfoz/backblaze-personal-wine:beta). If you run it, I'd like numbers: rough speed before and after (bb-monitor in the container console shows it live), your thread count, your connection speed, and roughly what size files you're uploading. Especially if you'd compiled my earlier patched Wine yourself and swapped it in, or built an image with the other fix. If any setup is faster with the re-arm than with this, that's important data, and I want to see it rather than argue with it. The update also picks up a container fix: a stale lock left by a hard restart could wedge uploads in a way that looked healthy to every check. The container now detects that, and with the watchdog enabled, clears it itself. If you'd rather not run the beta, v10.2.0 is out today as a stable release and carries everything else from this round. The highlights: bb-monitor: a live upload dashboard in the container console showing speed, per-thread file progress, recently completed files, and memory gauges Backup-aware health: the Unraid Docker page now shows unhealthy when the backup is actually stalled, not just when a process dies Self-healing (opt-in): set ENABLE_WATCHDOG=true and the container clears stale locks and restarts deadlocked upload threads on its own bb-doctor: checks the install against every problem this project has hit, and --fix repairs the safe ones bb-report: builds a diagnostic bundle for bug reports with file names hashed, so nothing private leaves your box The beta only adds the Wine speed fix on top. The full list is in the changelog on the GitHub releases page.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Quick update while I'm mid-testing: the deeper test run has turned up a problem, so this is taking longer than planned. While putting both halves of the fix through Wine's full socket test suite, I found a case where my writability change can leave a blocking send waiting forever. Backblaze itself doesn't use sockets that way, which is why weeks of real backup testing never showed it; the client polls rather than blocks, and that path is fine. But it's a real defect, and I'm not shipping a combined beta until I understand it fully and have it fixed properly. Currently narrowing down exactly which conditions trigger it and whether it was present from the start or exposed by a newer Wine. I'll report back with the fix, and the Wine reviewers will get the full write-up either way. Short version: the speed fix works as well as ever for backups, I've caught something in testing that nobody else has hit yet, and I'd rather be slow and right than fast twice.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Yes, that is essentially what I am doing today, though it will end up as one change rather than two bolted together. The slow upload has two separate causes and you need both fixed to get full speed. The first is that Wine tells the application a socket is not writable when it is. Linux only reports a socket as writable once its send queue has drained well below the buffer size, whereas Windows says writable whenever there is any room at all. cURL (within Backblaze) believes Wine and stops sending. I tracked that down, filed it as Wine bug 59893, and submitted the fix as merge request !11272. It has been with the Wine developers for several weeks now. That fix is what is already in my beta image, and on my own hardware it took a real backup from about 1 Mbit/s to about 72. The second is a deadlock. Once Wine has told the application "you can write", it stops watching the socket for writability at all. If the application then hits a genuinely full queue and waits for the next writable notification, nothing is left watching, so the wait runs to its full timeout even though the socket drained milliseconds later. A Wine maintainer spotted this gap while reviewing my merge request and left it open as an unsolved problem. I had actually built a fix for that half back in the spring and dropped it, because when I tested it against a real Windows machine the behaviour did not match Windows and I was not prepared to submit something that made Wine less accurate, and it would have likely been rejected anyway. That is why it is not already in the MR. I have now come back to it with a version built on what I learned from the socket work. It acts only when the send queue is genuinely full, which is the state Windows treats as a failed send, rather than acting every time Linux declines to report writability, which, as above, happens routinely while there is still plenty of room. The result is that the recovery path only runs when it is actually warranted, and in the common case, the application never stops sending in the first place. Today's job is to build Wine with both halves, run the socket conformance suite before and after, stand up a Windows machine to confirm the second half matches real Windows, and check nothing else regresses. If Windows behaves the way I expect, both halves go to the Wine developers as a single change and into the beta image at the same time, so there'll be no more waiting for you guys if you run the beta image. I would rather this landed properly in Wine than lived on as a patch in one container. It is broken for everyone running a Windows networking application under Wine, not just us.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
A quick update, and firstly, apologies that this release has taken a little longer than planned. The usual life pressures got in the way, although I have been working on it on and off over the past few weeks. I have spent most of this weekend completing the remaining updates, bug fixes and security work, followed by smoke and CI testing and dogfooding the new image myself to make sure everything behaves as expected in a real installation. This is a fairly substantial release. Without spoiling the full announcement, it adds: Backup-aware container health reporting Optional recovery from known stall conditions A live terminal upload monitor Improved version reporting Installation diagnostics A safe way to produce sanitised support bundles on demand There has also been a considerable amount of hardening behind the scenes, particularly around recovery behaviour, diagnostic privacy and preventing broken images from being published. The final burn-in tests are running now after they caught a couple of minor issues that I wanted to correct rather than knowingly ship. Assuming nothing else appears, I expect the release to drop tomorrow. Wine upload-speed patch I am still not entirely comfortable shipping a patch before it has been properly reviewed and merged upstream into Wine. However, there is clearly a significant appetite both for the fix and for a straightforward way to try it without rebuilding Wine or injecting scripts into a running container. I will therefore provide an explicitly labelled beta image based on Ubuntu 26.04 with the patch already baked in. It will be easily selectable alongside the two stable release channels (Ubuntu 24, Ubuntu 26). Upcoming Backblaze client release I am also aware of an upcoming Backblaze client release that may introduce compatibility issues because it includes a newer version of cURL. As soon as Backblaze begins serving that build, I will test it against the Backblaze64 container and make any changes needed to maintain compatibility. More details, upgrade guidance and the full changelog tomorrow.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Thanks for the kind words, and no offence taken. It’s a fair concern. A bug that’s been open since 2010 says more about that particular bug than it does about WineHQ’s bug tracker as a whole. Old reports often stick around because they’re obscure, difficult to reproduce, or simply not affecting enough people for anyone to spend time on them. It doesn’t mean they’re being ignored. What usually gets a bug moving is someone putting in the work. In this case, that’s exactly what’s I'm doing - I didn’t just file a report, I wrote the fix for the bug, following Wine’s coding and submission guidelines, along with a conformance test that’s been validated against real Windows to make sure the behaviour matches Windows itself. I sent it to wine-devel as an RFC first, then submitted the merge request. It’s passed Wine’s automated test suite, and I’m currently discussing it with the maintainers responsible for that part of the code to address any feedback before it goes in. So it isn’t sitting unnoticed in a queue. It’s under review, and I’m actively working through the process. As for pushing an update myself, that’s effectively what this is part of. The aim is to get the fix into Wine proper so everyone benefits from it instead of maintaining a separate patch. In the meantime I keep the coainer up to date. I’ve already tested it against the latest Wine point release to make sure everything still works correctly on the versions the image is built from. If for whatever reason, WineHQ declines to include th fix in their release, that's when I will consider making the patch part of this software release and maintain it myself (e.g. continue to ensure it works with the current release of Wine). While we're waiting, here’s something I’ve been working on that I’m thinking about including in a future release: It’s a live upload monitor built into the container. There’s nothing to configure. It opens a full-screen terminal dashboard showing: Live estimated upload speed. Per-file progress, including filenames, sizes and progress bars for each active upload thread. A rolling list of recently completed files, including their size and upload speed. An overview of the current backup session, with active threads, chunks per minute, session totals, and memory and swap usage. Scrollable upload viewport when lots of files are uploading, and adaptation to almost any terminal size. If that’s something you’d find useful, let me know in the thread. If there’s enough interest, I’ll include it in an upcoming release. You don't need to write a reply if you don't want to - just liking this message is enough.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Hi @kurai, thanks; the screenshot confirms the client side is healthy (you're on a valid trial), so we can rule that out. That narrows "Horse" here to the one remaining cause: the client couldn't complete its handshake with the Backblaze datacenter when you hit Sign In on the inherit. With a healthy trial, that's almost always a connection timeout rather than a block and a large backup set makes it more likely, because the inherit has to pull your existing backup state from the datacenter and that can run long enough to time out. A few of things to check first, since timeouts are often transient: Just retry the inherit a few times. A fair number of Horse-on-inherit cases clear on the second or third attempt once the state pull gets through. Make sure nothing sits between the container and Backblaze - a VPN, proxy, or DNS filter on the Docker host or connecting network can break the specific datacenter call even while normal sign-in works. Confirm that you are transferring a like-for-like OS backup - my own original backup was on a Mac Mini, so I unfortunately had no choice but to start fresh. The one advantage of this is that it discards any garbage the backup has collected over time. The only potential issue here is how long your backup takes to complete - if it goes beyond 30 days, then you may want to pay for a second license until you complete the backup and then discard the original. If it keeps throwing Horse, let's look at what it actually logged so we're not guessing. Right after a failed attempt, grab the newest transmit log (from your Unraid terminal): docker exec <your-container> sh -c 'd="/config/wine/dosdevices/c:/ProgramData/Backblaze/bzdata/bzlogs/bztransmit"; f=$(ls -t "$d"/*.log | head -1); echo "$f"; tail -150 "$f"'Post the output (feel free to redact your email/any long ID strings); the line at the failure timestamp will show which datacenter it hit and whether it was a timeout or an actual error, and that tells us whether it's something we can fix on the container side.
-
[Support] Backblaze 64: Backblaze Personal Backup 10.x (64-bit) under Wine
Progress update: root cause and fix for the ~140 KB/s upload limit For everyone following the slow-upload limitation, I've tracked it down and have a working fix. Technical write-up below. Symptom: single-stream uploads under Wine top out around 140 KB/s regardless of bandwidth. It only affects app-limited senders — ones whose send() calls keep succeeding without ever filling the socket buffer. Backblaze's uploader bztransmit uses libcurl, which is exactly this pattern, so large checkpoint uploads never finished inside the client's 600-second upload timeout. Root cause: it's a difference in socket writability reporting between Wine and Windows. On Windows, select() reports a connected TCP socket as writable whenever send() it would still accept data — i.e. as long as the send buffer has any room. Wine derives its writability from the host poll(). On Linux, POLLOUT isn't set until the socket's send queue drains below ~2/3 of SO_SNDBUF. So while the buffer is between ~2/3 full and full, Windows says "writable", but Wine says "not writable" even though sends are still succeeding. libcurl's event loop does a non-blocking select() writability check before it waits on an FD_WRITE event. FD_WRITE is edge-triggered and isn't re-armed until a send actually fails with WSAEWOULDBLOCK. On Windows, the pre-check returns "writable" and curl sends immediately; under Wine it returns "not writable," so curl falls through and blocks on WSAWaitForMultipleEvents for its full ~1000 ms timeout - once per ~64 KB burst. That's where the 140 KB/s comes from. (curl documents the underlying Windows behaviour in curl issue #6146: https://github.com/curl/curl/issues/6146.) The fix: a small change in wineserver's poll_socket() server/sock.c) — report a connected stream socket writable when its send buffer still has room (TIOCOUTQ < SO_SNDBUF), matching Windows. That's it; one block. (For the curious: I also tested the obvious-looking second theory - that Wine should re-arm FD_WRITE on every send. Turns out real Windows does not do that; it only re-arms after WSAEWOULDBLOCK, per MSDN. So that change would have made Wine less like Windows, and I dropped it. The writability fix alone is sufficient.) Validation: ws2_32:sock conformance test (the invariant "select reports writable iff send would accept data"): passes on real Windows, fails on unmodified Wine (~230 of ~660 sends reported not-writable while still succeeding), passes with the fix. Real backup, end-to-end: a checkpoint that previously hit the 600 s timeout uploaded in ~80 s under the patched build. What's next: rather than ship a privately-patched Wine in the container straight away, I've filed it upstream (Wine bug 59893: https://bugs.winehq.org/show_bug.cgi?id=59893) and sent an RFC to wine-devel proposing the server/sock.c change plus the conformance test. The one open design question is that TIOCOUTQ is Linux-specific, so I want the maintainers' steer on the mechanism before opening the merge request. If it lands upstream, the container just picks it up from a future Wine release; no fork to maintain. If they decline, I'll bundle the patched build into the image. I'll update this thread when there's movement upstream. Unrelated tip: backup stuck / "Failed to grab fourHourLock" / needs a restart to run If your backup wedges (the log spamming Failed to grab fourHourLock, or the container only backing up after a restart) on lower-RAM hosts this is almost always an out-of-memory kill, not a Backblaze fault. What happens: bztransmit gets OOM-killed mid-pass while it holds the 4-hour backup lock. The lock file is left behind, so every following pass fails to acquire it and the engine spins. Check the host kernel log - on Unraid that's the Unraid terminal (or SSH), not inside the container: dmesg -T | grep -i 'killed process.*bztransmit'If you see kills there, that's it. The engine's memory scales with the number of files it tracks - the file-id index and the "build todo list" phase can climb to 3-4GB on a large set, which is enough to get reaped on an 8GB box. Fixes, roughly in order of effect: Restart the container once to clear the stale lock and resume. Cut the tracked file count: exclude large regenerable or duplicated trees - caches, package dirs, and especially backup-archive folders, which can add millions of files from a single migration. Drop your Backblaze version-history retention to 30 days if you don't need more - deleted files stay in the index until they age out, so long retention keeps memory high. Lower the backup thread count. Each thread is another bztransmit process; with the upload fix above, a single stream is fast, so you don't need many - 1–2 is plenty on a memory-constrained box, and Automatic can over-provision. If you can, give the host more RAM. Post your dmesg output and rough file count if you want a hand narrowing it down.
foz
Members
-
Joined
-
Last visited