KingArthur

Yeah I felt extremely stupid that I hadn't set that up. When I set up this server a few years ago, I don't believe that the Backup/Restore Appdata plugin yet existed (though perhaps one of its predecessors did). I obviously consciously knew appdata was important to maintain because I have some dockers with .yaml files I manually edit that live in appdata, but I was dumb. This also makes sense. I think I am just reflexively thinking I should preserve everything. But it's all config from the templates and then just working in appdata and whatever external locations. Thankfully I do backup the flash pretty frequently whenever I make docker changes! And now I have turned that setting on in the Backup/Restore Appdata so I no longer have to manually think about it. Do you have any advice on backing up VMs? Perhaps a recommended plugin? The only VM I run is a Home Assistant OS install, so it's relatively small. Is UrBackup a reasonable option as far as you know? (Sorry for the tangential question.)

Totally understood. I think that when I was looking up how to even turn this setting on I found an old forum response from you where you were cautioning somebody else of the dangers of accidentally doing a disk <-> share copy operation that just deletes the content. Definitely wasn't planning on leaving that enabled. Very much appreciate the recommendation! I'll add that right now. Dare I say, I think I kind of got away with this one. I had a small handful of files that refused to copy over, but that was just a single video file and then some Plex generated thumbnails from my Plex media folder. Currently though, every Docker container is operating as-expected. Time will tell if anything is unstable, and I'm definitely going to implement better management practices going forward (backing up my appdata and docker image, for example, duh). I obviously did not understand the structure of Docker and appdata stuff and could have really gotten bitten in the ass by this.

Ahhhh very good. I'm dumb. I was assuming it was some kind of explicit permissions issue or something. Am I correct to assume that deleting files this way doesn't somehow invalidate parity?

Yesterday my cache freaked out and started generating IO errors. I received help here to at least identify that my cache config was not going to be directly re-usable. I suspect that at least one of the drives I was using went kaput, so I re-configured to a pair of 1TB drives. (Previously had one big cache pool of 4 misc SSDs. Figure I'll be more organized and use one 1TB drive as cache for appdata and the other as cache for normal cache operations + my docker image.) During my debug and screwing around, trying to figure out what had happened when that drive failed, I started the array with Docker and VM services still enabled. At that time, since the cache was down and I had my array set to be the backup location, it generated my Docker image on a disk in my array. It also made a few appdata files in various places as Docker launched with some containers that didn't need to access appdata started. I've recovered my original Docker image file and original appdata folder contents from the failed array drives and am repopulating those to my new cache. However, if I go through file explorer on Windows and directly try to delete the version of the Docker image that got created when it re-generated it on an array disk (have the setting turned on to show individual drives over SMB), I get an error that says "You require permission from CASTLE/nobody to make changes to this file". Any advice on how to proceed? I'm sure what I'm doing is bad and sketchy in some way. Diag, for if that's useful here.castle-diagnostics-20250205-0156.zip

I'm looking for a road-map to help me fix this situation. I apologize in-advance for my lack of deep understanding of a lot of these systems. Current course of actions: The current course of action I am taking is that I've mounted my cache drives through the UD plugin so I can access them over local network through file explorer on my PC and I'm manually backup up everything. This is taking forever, as I would expect. My plan is then to format a new cache with 2x 1TB drives, copy everything back, and then hopefully that should get me mostly running? Would love to hear thoughts on if this is bad or inefficient or etc. Needless to say, I'll be setting up Appdata Backup when this is all done. I don't think that plugin existed when I built this thing and I'm paying the price for not being diligent about awesome new developments. The rest of my post below is before I started this backup process. So, following this morning's post (linked below), I need to clear out my cache pool and get back on my feet. I'm looking for some advice on the least disruptive way to go about this. I at least know just enough to be aware that my appdata folder and my Docker.img file live on my cache (lol). I thought I could be cheeky and just switch all my shares to go from cache -> array then run the mover, but that indeed yields about a million errors. Or, for example, attempting to copy my sonarr.db file yields this. Annoying! Is there any simple way to attempt this? Add another 1TB drive to the cache, somehow balance everything to shift to that drive, then remove the other drives? Presumably, these files are the ones that are borked and this error would persist. Or, alternatively, can I mount these drives in the UD interface and back up these files that throw errors? I know that risks some data corruption if the files themselves are borked, but is this a good idea to do? I think I'll be permitted to copy the files, and it's not like I have mission-critical info here if some of the individual files are corrupted. My setup is mostly the -Arr media management dockers + torrenting + Plex + tdarr for transcoding + Bazarr for subtitles. There are a couple other adjacent dockers, but that's the general ecosystem. My only VM is a basically unconfigured Home Assistant install that I started but then learned we were going to be moving soon, so put on hold until mid March after we move. I'm just confused as to what I should attempt to back up in order to have the least disruptive process. A few questions. 1) I do think I've been able to backup my Docker image (it at least copied successfully), does that save me a lot of effort? 2) For Sonarr and Radarr, I seem to be able to acquire the backup files from last week. Presumably that saves me a bunch of effort? Is the process here to just: 1) Salvage everything I can from AppData 2) Eliminate/format my current drives (don't know exactly how this process works, but I do think I'll do some hardware swapping and just go 2x 1TB drives this time) 3) Start the array with new cache 4) Get Dockers running one-by-one and see what's broken? Additionally Somebody pointed out in that other thread that some of my dockers had started to make appdata homes in my array. Do I need to do anything to sanitize that situation, or will the typical move after I fix my cache solve that? I think that I don't have a good understanding of what to expect here. I'm very much a follow-the-setup-guide-and-tinker-around type person with this setup, so I'm concerned the lift to get things back to a stable operating state will be monumental. )

I appreciate it, Jorge. I'm starting a new thread to ask for some more advice regarding backups and the least-disruptive way to do this.

Thanks, @JorgeB. So to be clear, that BTRFS error forces the pool into read-only? Is there a suggestion that this error is the result of a hardware failure with device sdb1, or is this a one-off pool error of some kind? Should I be excluding that device from my new config? As for the best way to approach this, is the process to copy what I want to attempt to save, shut down the array, configure my cache drives the way I want them, reformat, start the array, copy content back to the cache, then go from there?

EDIT: Whoops. Definitely not fixed. Getting Fix Common Problems Errors "unable to write to cache" and "unable to write to Docker Image" I was re-reading some other posts I had pulled up earlier where @JorgeB recommended to another person that the way to address the "All existing data on this device will be OVERWRITTEN when array is Started" warning if you sort of mess up a drive config is to un-assign everything, start the array, stop the array, re-assign. I did that. Then, I figured I'd try to start the array, and the cache pool started totally fine? I then went back and re-enabled Docker and VM Manager and the docker service started, as did the cache. But now I have to figure out what broke here. castle-diagnostics-20250204-1005.zip

Thanks for the advice @trurl. I am hoping that this won't be too awfully messy to resolve, but I'm not the most optimistic of that right now.

root@Castle:~# btrfs fi show Label: none uuid: bbe1c830-2781-400e-8a8e-6dda9da14c36 Total devices 4 FS bytes used 841.49GiB devid 1 size 238.47GiB used 154.00GiB path /dev/sdb1 devid 3 size 223.57GiB used 134.00GiB path /dev/sde1 devid 4 size 953.87GiB used 499.03GiB path /dev/sdc1 devid 5 size 238.47GiB used 150.00GiB path /dev/nvme0n1p1

Looks like 2nd diag file didn't post on my original thing. Here is that file, which is diag after starting the array.castle-diagnostics-20250204-0823.zip

Hey folks. Looking for some help here. Overnight, all my dockers crashed. I poked around a bit and figured I'd just reboot the server to start. After reboot, my cache wouldn't mount. Getting an error that says "Unmountable: Unsupported or no file system" I had one cache NVMe SSD that seemingly wouldn't generate a SMART report, so I removed it from the cache pool and tried to start the array, but got the same error. Now, when I try to add it back, I get the "All existing data on this device will be OVERWRITTEN when array is Started" warning. I do not remember how to resolve this, but figure I should before I tackle whatever caused my overnight issue. My presumption is that no data has been modified on these drives since they can't mount, so perhaps there is come config thing I can do to re-establish the 4 drive cache. No hardware or other changes in months. I've just been plugging along for a while now. As you can see, the drives here are a gross mishmash of all sorts of stuff. Posting both diag with the array stopped and after trying to start it with 3 of the 4 cache drives listed. __819 is without the array started. __823 is with it started but that 4th cache drive not selected in the pool so I don't accidentally overwrite something if I can avoid it. I would appreciate any help! castle-diagnostics-20250204-0819.zip

The cost would be roughly the same. ~$140 for either a 2696v3 or a 2699v3. Both 18 core, 36 threads. I think they're basically the same CPU but one has DDR3 support and one has DDR4 support. The decision would ultimately come down to if I feel like going super budget and getting this cheap board + the RAM I already have or buying a more reputable board and purchasing some DDR4. That difference is only ~$200 in cost, but when the entire upgrade is only going to be $350-$650 depending on parts, that $200 is a significant percentage of the overall. The same question I have about performance persists when looking at the mainstream E5s though. I quite literally have no clue if an E5 + a P400/P600 is more or less performant than an i7-10700. I mean, shoot, the 3rd upgrade option for me is to donate the Ryzen 1700X that's in my desktop and spend my upgrade money on a new CPU + mobo + RAM for my desktop. But, again, it is difficult for me to discern unRAID + Plex + docker performance differences between these CPUs of different generations.

You're correct on the results of your searching. There are a number of SKUs in the E5 lineup that support DDR3 and were only ever sold to other OEMs/system integrators. They're unlisted on Intel's site, but if you purchase these 2011-3 boards that support DDR3, they have a specific list of compatible processors that also support that DDR3. Do you run a GPU in the 2650 server, or do you just roll with software transcoding?

I'm considering an upgrade to my server. Currently it's just a deconstructed Optiplex (i5-6500) and a couple LSI cards, plus a bunch of various drives. Of course I started cheap, spent a bunch of money on drives, and now want to upgrade. Partly for fun, and partly to just smooth out operation so I can reasonably run some VMs while Plex and the -arrs are doing things. I recently discovered the Huananzhi X99 motherboards they sell on AliExpress. ~$120 for a seemingly competent X99 board. I could slot in a Xeon, and since I have some DDR3 RAM around I was considering an E5-2696v3. I'd probably want to drop a Quadro P600 or P620 into this in this config since I'd lose iGPU. Total cost = ~$450. Does this kind of approach still make sense these days? Most of the hardware related posts I see are either Ryzen 3000-series focused or push for using a 9th-ish gen i7 to retain iGPU stuff. I'm just curious as to what people consider to be the most effective budget config options when buying used parts these days. Edit: Adding links for the aforementioned Huananzhi board and the data page for the E5-2696v3. Huananzhi TF X99:https://www.aliexpress.com/item/4000120773162.html E5-2696v3: https://www.cpu-world.com/CPUs/Xeon/Intel-Xeon E5-2696 v3.html