EddieDS

I'm assuming the only work around would be moving the files to a new folder and then nuke the datasets, or is there any way to convert datasets into folders? Is there any other limitation or issue with child datasets I might not be aware of? For appdata I still think it can be useful so I can roll back to a previous snapshot in case something breaks during an update. Thanks for answering!

Hello everyone, I am trying to understand how Mover behaves in Unraid when a share configured with Primary/Secondary storage contains child ZFS datasets at the second level. My setup is the following: Unraid version: 7.2.4 ZFS pools: core vault scratch Main shares: appdata → Core documents → Core -> Vault media → Scratch -> Vault photos_immich → Core -> Vault system → Core temp → Scratch zfs_backup → Vault The issue is that, from the begginning, I started automatically converting all second-level folders into child ZFS datasets using the SpaceInvaderOne script that converts folders into datasets. Script So inside shares I no longer have normal folders, I now have many child datasets. I was doing these because (by using ZFS Master plugin) it gaves me a clear visual of how everything is organized; and being able to adjust dataset properties and snapshots individually foir each dataset seemed like a good thing to have. I've recently found that, if I'm not understanding incorrectly, mover might not affect these child datasets. Does Mover still work correctly for those child datasets when the parent share is configured as Primary -> Secondary, or does Mover only really support the top-level share itself? I am asking because until now I assumed that if, for example, documents = Core -> Vault, then anything inside documents, even if it was a separate ZFS dataset, would follow the same Mover behavior as the parent share. But now I am not sure whether that assumption is wrong, and whether nested datasets may actually fall outside the expected Mover behavior. Additional context: The child datasets are inside shares that do depend on Mover: documents, media, and photos_immich. I am less concerned about shares like appdata, system, temp, or zfs_backup, because they do not rely on moving data between two storage locations. What I would like to confirm is: Whether Mover supports nested datasets or not inside a share that uses Primary/Secondary storage. Whether a child dataset sitting under the parent share is actually moved in the same way as a normal subfolder. What the correct way is to test and verify this. Whether the recommended approach would be: To use normal subfolders inside the share if Mover behavior is required. To create separate top-level shares for data that must follow that logic. Is there any potential issue I might be overlooking? In case my current setup is wrong, how can I revert it? Thanks in advance!

Good evening everyone, I’ve set up my first unRAID server and have the basics working. Before getting deeper Docker and Apps, I want to put a solid backup strategy in place. I’ve got three ZFS pools: NVMe mirror for appdata NVMe for cache 4× HDD RAIDZ1 for bulk storage So far, I’m using SpaceInvaderOne’s script for ZFS snapshots and replication to automatically create snapshots of the mirror and HDD pools, and to replicate appdata from the mirror to the HDD pool. I’ve also seen the Appdata Backup/Restore plugin, which seems to do something similar but stops the Docker containers first. I’m assuming this is mainly meant for XFS, since ZFS (at least as far as I understand) can replicate without needing to stop anything. Is that correct, or would I still need the plugin anyway? For now, I’m backing up the USB flash drive directly through Unraid Connect. Is that enough? I’m fairly new to this, and I’ve read that databases should be dumped. Would the ZFS replication I’m using be sufficient, or should I handle DB backups differently? Lastly, I want to add an offsite backup, but this is where I get a bit lost. Since I think it is not possible to get a ZFS cloud storage, I’d like something with encryption and deduplication so I can keep versions. I've been checking out Duplicati and Duplicacy, but I honestly can’t tell which one would fit my needs better. Is there any better alternative? Any specific provider? I was checking Backblaze because of their unlimited plan, but I think is not allowed for servers. I’ve watched a bunch of videos, but most of them focus on one or the other, and the few comparisons I’ve found didn’t really clear things up for me. What do you guys think? What would be the best approach here? Anything I might be missing? Thanks in advance!

Good evening everyone, Following the advice I received here, I’ve finally set up my first server using a 12600K and 64 GB of RAM. So far, I’ve configured three ZFS pools: A 4×24 TB HDD RAIDZ1 pool for storage, called vault Two 2 TB NVMe drives in a mirror for appdata and important cached data, called core One NVMe ZFS pool as disposable cache, called scratch I’ve created some basic datasets and shares, and installed the plugins that are usually recommended on the forum: CA Auto Update Applications Community Applications Fix Common Problems Network UPS Tools (NUT) for Unraid, set up with my UPS policy and winNUT on my desktop PC Sanoid for Unraid 6/7 Tips and Tweaks User Scripts ZFS Master for Unraid I’ve also started using SpaceInvaderOne’s Unraid ZFS Dataset Snapshot and Replication user script to create snapshots of core and replicate them to vault, as well as to create snapshots of vault itself. I’m currently using the following snapshot retention policy: snapshot_hours="24" snapshot_days="30" snapshot_weeks="24" snapshot_months="12" snapshot_years="2" Before I start experimenting with Docker and moving my data over, I have a few questions. Is this snapshot policy overkill? I know I should run scrubs and SMART tests periodically, but I can’t find a way to do this directly from the GUI. Should I be using a plugin for that? Since I’m replicating core into vault, is that enough for backing up appdata, or should I still use the Appdata Backup plugin? In Fix Common Problems, I’m getting the following warnings: “Share documents set to use pool core, but files or folders exist on the vault pool” “Share media set to use pool scratch, but files or folders exist on the vault pool” “Share photos set to use pool core, but files or folders exist on the vault pool” Each of these shares is designed to use the other pools as cache for vault, so isn’t this warning just expected behavior? Should I simply ignore it? I’m also having issues with Unraid Connect. I tried to back up my flash drive there, but the website just stays on “Loading”, even though the plugin is installed. I’m planning to develop a proper 3-2-1 backup strategy once I’m more comfortable with the system. Is there anything else I might be missing, or something I’ve already messed up? What should I look forward next? Thanks in advance!

I’ve been looking into UniFi Teleport based on your previous answer. Since I already have a UniFi switch and gateway, it might be better for me to use it over Tailscale or a direct WireGuard connection to access the server on the go. Will think about it. Got it, I can skip the pre-clear process if I’m using ZFS, and I assume the same applies to XFS. Regarding disk wear, I’ve been researching whether spinning down is better for the drives and couldn't find a consensus. In the case of ZFS, it might be better to skip it altogether. As you mentioned, any access to a single disk in a ZFS pool would spin up the entire array. You mentioned that 24/7 spinning is better than frequent cycles, but what is considered "frequent"? I would save around 25€ a year with four disks, but that could rise to 75€ once I populate all available 12 bays. I didn't realize that "batch mixing" was more of an HDD concern. Since the FireCuda has a higher TBW, it’s likely more suitable for a scratch cache. I’ll keep the two SN850X drives in the mirror. I need to dig deeper into shares, but from what I gather, I should create a dataset for each type of data rather than using subfolders within a single dataset. Instead of a single "tank/media" dataset with "photos," "movies," and "music" subfolders, I should create three separate datasets: "tank/photos," "tank/movies," and "tank/music." This way, I can assign different options to each dataset and isolate which Docker services can access them. I plan to follow the 3-2-1 backup rule for my important data. For movies and music, I’ll likely rely on redundancy alone. For photos and non-replaceable data, my plan is to keep one copy on the array, one on an external NVMe drive, and a third backed up to a cloud service. In the future, I might add two Barracudas from my main PC (that are not being used much) as a secondary array for local backups to replace the external NVMe. Lastly, I’m a bit worried about future expansions with ZFS. If I understand correctly, increasing capacity would require me to add the exact same number of disks I already have. So, to expand my current 4-disk setup, would I need to add another 4x 24TB drives? If I want to max out the 12 bays, I would need to reado the array since I can't add another 8. Am I missing something here? Must the disks be the same model too? I know it’s possible to format disks in ZFS individually, but I assume I’d lose most of the advantages of ZFS that way. Seems like completely liking unRAIDs versatility going down this route. Thanks again!

I’ll definitely have to come back to the Docker tutorials once I have the arrays up and running. Some of the examples feature services I plan to use, so they will be very helpful. I have one question regarding the "how to handle a failed drive" video: he preclears the disk once it's added to Unraid. I ran HDDScan’s "Erase" on all my disks before shucking them, and they all came back without any bad sectors. Should I repeat the process in Unraid for any particular reason? Regarding disk formatting: I hadn’t considered using ZFS for the HDD array. I understand the benefits of checksums and snapshots, but would I lose the ability to spin them down altogether? At the moment, the electricity cost of a few extra HDDs isn't a concern, nor is the noise though less is always better. However, I’m worried about unnecessary disk wear. Since I’m starting with high capacity, would it be possible to keep at least two of them spun down most of the time? What would be the best approach here? I might be overestimating the benefit of spinning down disks. As for the increased performance, if I’m not mistaken, it would mainly affect read speeds since I’m using NVMe drives for cache. In what cases could this be a limitation for general use? I’m definitely skipping the Metadata VDEV; adding a single point of failure doesn't seem like a good idea. I think I’ll go for the mirrored ZFS pool. I noticed you recommended mixing both SN850X drives], isn’t it better practice to mix brands? Since both SN850X drives were manufactured on the same day, couldn’t they potentially share a manufacturing defect? I’ll decide whether to keep the third NVMe as a spare or use it as a scratch pool to reduce wear on the mirror. I always try to keep everything organized in folders. I assume datasets also allow for personalizing which services can access specific data? Thanks again for the detailed answer; it’s been very instructive!

I’ve reviewed everything you sent, and I’ve decided to go with XFS for the four HDDs (one for parity) and keep the NVMes in ZFS. However, after reading the documentation, I have a question regarding the pool configuration. I noticed that RAIDZ1 is described as being fast for large files but less ideal for small, random writes. If I’m keeping my Docker containers and VMs inside that pool, wouldn't that be an issue? I’m concerned about potential latency since parity must be calculated in real time. From what I’ve gathered, a Mirror might be more suitable for this kind of random I/O workload. Additionally, wouldn't a RAIDZ1 array be limited by the speed of the drive in the Gen4x2 port? I realize this is still far beyond the needs of a domestic network, but I’m curious if it could affect internal operations or system responsiveness. Because of this, I’m still considering the approach I mentioned before: a two-drive mirror for my primary pool and a second "scratch" pool without redundancy for replaceable data, such as downloads from the Arr stack. Does that approach make sense, or am I missing something? Any relevant issue with this approach? Some benefit I might be missing from RAIDZ1? I also noticed that compression is highly recommended; I would have completely overlooked it otherwise! I’ll also take note of the utilities and scripts you suggested. Thank you so much!

First of all, I want to thank you for such a detailed answer; I think things are much clearer now. I believe you're right, I was looking too much into things that don't offer any real utility at the moment, creating new issues along the way. I think I'll stick to bare-metal Unraid for now. I assume the best practice would be setting the HDD array to XFS; however, I noticed you mentioned using Btrfs in RAID1 for the cache. I thought ZFS was now supported and the recommended filesystem for the NVMe pool. What would be the reason for choosing Btrfs instead? In addition, since I’ll have three NVMe drives, I’m trying to determine the best way to distribute them. I was considering using the two Gen4x4 ports for the FireCuda and one SN850X so I can have redundancy through a mirror/RAID1 setup, while also mixing brands in case the WD batch has a defect. This pool would host my Docker containers and VMs, as well as the cache for important data like documents and Immich photos. The other SN850X would go into the Gen4x2 port. What would be the best use for it? Since it won't have redundancy, I was thinking of using it as a "disposable" cache for things like movies, music, or a download scratchpad. Is there a better alternative? Could I keep, for example, recent films and the music I usually access there so I can increase the spin down time of the array? I could leave it as a spare, but it might be better to use it to reduce wear on the primary mirrored pair. For now, I'll focus on getting everything running and understanding how services and Docker work. As you mentioned, I’ll need to design my disk and share layouts and decide how to configure everything. VLANs are probably something I need to take into account as well, so I’ll look into how Docker interacts with them. Again, thank you so much for your time!

Good evening everyone, I asked a few weeks ago whether Unraid was a good fit for my needs, and I’ve been reading up on it since then. I’m a beginner getting into homelabbing and self-hosting for the first time. I’ve already built the server and I'm leaning into using Unraid, mainly because I believe my use case would benefit from being able to spin down disks. The hardware consists of: CPU: Intel Core i5-12600K MOBO: ASUS PRIME B760-PLUS D4 RAM: 64GB DDR4 3200MHz CASE: Jonsbo N5 (12-bay hot-swap, currently 4 active via onboard SATA) HDD Array: 4x 24TB WD White Label (one for parity) NVMe: 2x 2TB WD Black SN850X + 1x 2TB Seagate FireCuda (if needed) It was recommended that I consider virtualizing Unraid in Proxmox, but I have a few concerns. I understand the main advantages are service isolation and the ability to restore via snapshots. My understanding of the setup would be as follows: I would virtualize Unraid in Proxmox and perform a full passthrough of the motherboard's SATA controller. I’d do the same for the two WD NVMe drives for the cache, which I plan to run in RAID1 for redundancy. This VM would handle all NAS functions. On the other hand, I would need to create LXCs or VMs for the various services. This is where I have doubts about the best approach: Using Docker inside Unraid: This is probably the easiest way once set up within Proxmox and offers the benefit of community templates. However, I’m worried this might defeat the purpose of isolation. A second Unraid instance just for Docker: This would detach the NAS portion from the services while keeping Unraid’s ease of use for Docker. However, I believe this can't be done on a single license. A single VM/LXC for Docker: I would use a Linux distro, likely Debian from what I see usually recommended, and manage everything with Portainer or a similar tool. I’m unsure if I would need the third NVMe since the others would be passed to Unraid. Multiple VMs/LXCs for each service: I could group similar services, such as Immich + Paperless, or the Arr stack + Jellyfin, or AdGuard + Unbound. This offers the most independence but might be overkill to manage. Again, unsure on how to manage disk space. What do you guys think? Am I missing anything, or perhaps making things more complex than necessary? I’m a bit lost on where to start and unsure if the complexity is worth it over a bare-metal Unraid setup. I also have general issues understaing how the hardware is managed with multiple VM/LXC running. Lastly, I’ve seen that Unraid will soon be bootable directly from NVMe. Should I wait for this release before starting? Any general advice is welcome. Thanks in advance!

I’m slowly rereading all this, since I get lost at some points. Probably means I’m getting into deeper water than I thought, and I might be trying to make everything more complex than what I can handle. I’ll review some of the concepts that I’m not getting, and will check out the YouTube channel you recommended. Thanks again! Thanks for the insight!

Wasn’t aware that bitrot was not an issue anymore. My MOBO is a domestic one and therefore not ECC compatible. In case of a corrupted file, which would give a read error as you say, would it overwrite the preexisting backupc? Or would show up some error in the process? Just a general concern, as said it is my first server, so I’m unsure on what services I will end up making use of. At the moment, it will be connected via 2,5Gb to my PC, and witch 1Gb to the network. It’s my first time here and I have quickly noticed it haha.

First of all, I want to thank you for taking the time to provide such a detailed response. I’ll definitely look into Nextcloud alternatives. It’s just the one I see mentioned most often, but I haven’t settled on it yet. I’ve been reading around, and using Proxmox seems almost like a religion in some communities. I understand the benefits of snapshots and the independence between services, but I’m still unsure how difficult it would be for someone with my limited experience. The same goes for TrueNAS. I’m considering a strategy similar to what you mentioned: backing up all Docker containers and configurations to both the HDD array and a cold storage cloud every night. Additionally, sensitive data like photos or important documents would be backed up from the array to the cloud. However, there’s a lot of content, like the media server library, that won't be backed up due to the sheer amount of storage required. That information is replaceable, but if I’m not mistaken, with a ZFS system I could just roll back to a previous snapshot in case of ransomware. Maybe I’m overthinking this scenario. Regarding bit rot: if it can’t be automatically fixed, from what you said it can at least be detected? I assume that if it happens, the corrupted data could be replaced using either the parity disk or the cloud copy. I’m using WD180EDGZ drives, which are shucked disks and supposedly enterprise-grade. I bought them second-hand; they have about four years of use and roughly 80 power cycles each. Electricity-wise, it would cost a bit under 30€ a year, so I don’t consider that a major factor. Since each disk would likely start a few times a day in normal use, I’m unsure which approach puts less stress on the hardware. I’ve used Docker Compose and Portainer briefly before, on a Raspberry Pi for home automation, DNS, and a few small services. I’ll need to study the differences and the benefits of using them in this new setup. As for using Proxmox alongside Unraid, if I’m already deploying services directly on Unraid, what exactly would be the advantage of adding Proxmox to the mix? Again, thank you so much for your time!

Hi everyone! I’m currently planning my first home server build and I’m looking for some objective advice on choosing the right OS. I realize this is an unRAID-focused community, so the perspective might be a bit leaned towards it, but I’d love to hear your honest take on whether this platform fits my specific goals or if I should look elsewhere. I’ll be using 4x18tb an a 2tb SSD for now and I have the following priorities. I’m quite new, so maybe something just not makes sense. Bitrot Protection: I need a system that ensures my critical data (photos in Immich, documents in Nextcloud) is protected against silent data corruption over time. And what would be my options in case of a ransomware? Disk Longevity: I’m undecided on the best approach for drive health. Is it better to use a system that allows individual disks to spin down when not in use, or is it safer for the mechanical health of these 18TB drives to keep them spinning 24/7? I’d love to hear your experiences on this, since in some places I’ve read the spin up process is the most stressful on the disk. Service Deployment: I’ll be running Immich, Nextcloud, and other Docker-based services. I need a reliable way to manage these along with a solid backup strategy. Performance: to my understanding, performance is limited since UnRaid can uses each drive individually. In what aspects will this be a limitation? I was considering TrueNAS + Proxmox as an alternative, but it might be too much of a learning curve. Given these requirements, do you think unRAID is the most suitable path, or would another OS serve these specific goals better? Thanks in advance for your help!