BVD

Honestly too much to type - I'll hang loose for a bit while you're scoping this out further and just wait to hear back on whether youd like a second set of eyes on it with ya. Best of luck, and enjoy the journey!

This sounds like one of two things: 1. You're using the external storage add-on and its misbehaving 2. inotify's lost its shit Assuming you dont have a backup, that's your first act - browse your nextcloud share from the unraid console (your mapped directory for /data), then go to <yourUser> -> files_trashbin -> files Copy everything out somewhere off the server in case it's your array causing the grief. Once you've confirmed your backups good, then we can worry about the NC side of it.

Whole bunch to go through there, too much to type right now, but a few things to consider: * Your test is for 256k IO using the random read/write algorithm, with sync enabled. * The default zfs dataset has a 128k block size (half the test block size), so two write actions for each 1 from fio. With sync, you're having to physically finish and validate the write to disk before continuing, not an ideal workload for HDDs anyway. * On top of that, weve got a 64 IO depth (which is essentially "how long can my queue be") is essentially halved by the default dataset blocksize; sort of "cancelling it out", down to 32 in effect The most important part though is this - in order to properly test your storage, the test needs to be representative of the workload. I pretty strongly doubt you'll primarily be doing synchronous random r/w 256k IO across some ~20TB of space, but in the event you do have at least some workload like that, youll just ensure that one dataset on the pool is optimally configure to handle it in order to ensure your results are "the best this hardware can provide". Also, would be happy to set aside some time with you still of course! As an FYI (just given the time of your response here), I'm in GMT-5, assuming we're basically opposite hours of eachother, but im certain we could make some time thatd work for us both. You just lemme know if/when you'd like to do so. I'm actually working on some zfs performance documentation geared towards unraid on github currently (going over different containers with recommendations on both how to configure their datasets as well as test+tune, general "databases on zfs" stuff, tunable options from the unraid/hypervisors side and when/how to use them, and so on), and the above post has been enough to kick me in the rear and get back to it. It's been an off and on thing for (months? Hell, idk), but I'll try to share it out as soon as at least *some* portion of it is fully "done". Maybe itll help someone else down the line 👍

Also wanted to mention - if you ever decide you need that extra ~400MB/s or so (1.1GB/s being well within reason for a 10Gb connection), the above would still definitely apply - and the journey's most of the fun!!

Glad to hear it, and happy to help! As luck would have it, I'm actually working on some performance tuning for 40Gb this week, and the number of things that play into it are wild to think about. Things most would never think of like chip architecture, bios settings (NUMA, memory interleaving, etc), driver specific flags, and so on - super interesting stuff!

I couldnt call it a "known issue" per se, just that achieving full 10Gb throughput almost always takes some tuning. 10Gb is a whole other can of worms, and achieving that level of throughput requires both careful planning, and a decent amount of optimization (both host and hypervisor side). For one, you're far more likely to need to worry about your peak CPU frequency, context switching, and high interrupt counts. Beyond that, you're much more likely to encounter IO bottlenecks in "weird" (or at least previously unexpected) places. You'll have to start by determining where the bottleneck is. For instance, are you simply mounting an SMB share? Have you tested NFS to see if you get similar IO behavior? What about share passthrough in the VM config (this usually sucks)? Tried the virtio network driver instead of virtio-net? And so on and so on. Just changing things without at least having an inkling of where you're bottlenecking is a recipe for pain. I'd start by installing something like the netdata docker container; start it up, initiate your 10GB copy, and look for anything that appears to spike in the reported statistics. High IRQ remapping? Single core pegged at 100% utilization? What else does that core have going on if so? What's the reported disk utilization at that time? Once you figure out what the bottleneck is, then you can start doing research on how to correct it; the solution will be unique to your configuration and the cause of the bottleneck, so just be prepared to do a little googling, and some trial and error along the way. Happy hunting!

If you've 30-40 minutes free today, we can do a quick remote session and take a look if youd like? We can probably sort out the cause in 10-15, but buffer never hurt. Shoot me a DM if youd like, I'll be around off and on throughout the day 👍

I honestly dont see them including it in the base OS, instead opting to better support ZFS datasets within the UI for normal cache/share/etc operations. e.g. youd be able to create a zfs cache pool, but only if the optional zfs driver plugin is installed; otherwise, only btrfs and xfs are listed as options. Thatd be how I'd do it anyway - forego any potential legal risk (no matter how small), limit memory footprint by leaving it out of the base for those who dont need/want it, and numerous other reasons.

The last time I tried it was about a month and a half ago, and at least at that time, it was all sorts of borked. The QEMU folks are working on it though - the first kernel with support built in was 5.18, so it's not even fully supported by the hypervisor yet (6.10.3 is still 5.15; 5.18 still seems to have some 'teething' issues last I'd heard). And just to be clear, by 'fully supported', I'm referring to the various alder lake specific components (IPI/thread director/et al) - doesn't mean it 'won't work', just that it 'won't function explicitly as designed'.

Absolutely this - also, can confirm it was 10th gen, with the intro of Xe graphics for their integrated GPU's. Intel had already laid the groundwork to support Xe with the open source community (including working with ffmpeg) prior to release to help ensure a "smooth" release. Plex's tweaks to it were out of tree and significantly behind current though, so they received none of those fixes/enhancements. I assume it has to do with their attempts to lock down transcoding to paid plex pass holders. Same issue with their SQLite implementation - they've hacked the hell out of it such that it's basically unsupportable by anyone but them... which means if it goes paws up, you're SOL when it comes to many manual DB repair options. I hate that plex has become the defacto media streaming tool given all this, and wish I could convince enough of my users to move away from it that I could stop supporting it 😓

@calvados the next time this happens, some other things to look at: * Autosnapshot tools - sanoid/syncoid/auto-snapshot.sh, anything that automatically handles snapshot management and is currently configured on the system can cause this. If you have it, kill *that* process (e.g. sanoid) first, then retry. * intermittent commands - you can try "umount -l /mnt/path" to do a "lazy" unmount; pretty commonly needed for a variety of reasons. Glad you got it sorted for now!

@calvados can you check for anything in use on the pool? ps -aux | grep cxUrPool Short of this, you can also try setting the mounpoint to legacy.

Is your nginx (swag/nginx-proxy-manager) container listed as a trusted proxy in you config.ini file? That's usually where most people get hung up that make it as far as this.

Aside from everything else, doesn't someone else's ownership of the unraid.<anything> domain constitute a problem (if not at least an area of concern) re: trademarks? Or how would all that work from a legal perspective?

Please dont take this the wrong way, and I'm coming at this from a place of respect in helping you protect your data, but if you're not willing to do the legwork to research / find things of this nature, zfs might not be best suited for you... ZFS is super powerful, but can also be quite dangerous if deployed without proper care, or at least a willingness to seek out answers ones self. Again, I understand you may be strapped for time, and I dont mean this as any kind of slight, not at all. Just that you'll need a willingness/time/patience to search out your own answers in order to be successful with zfs in the long run. Apologies in advance for any offense, none meant!

This ones been gone through a number of times on the forum (I think in this thread actually even) - check your docker settings and reconfirm, you'll get it sorted 👍

That can be done, but is sub-optimal - as the file wasnt created on the zfs system, the zfs specific metadata (xattr, etc) aren't applied, instead using whatever it was at creation, which may not match the zfs filesystems settings/config. Worst case is leaving performance on the table though (unless you start accessing that image over the network), so not a huge deal.

XFS on ZFS zvol should be fine 👍

Also, why would we want btrfs on top of zfs?

I updated with a response in my thread (again, apologies for the wait!), but figured I'd check in here as well - The biggest 'boon' to choosing epyc over TR pro is the wealth of motherboard options out there (imo) - the market for epyc is so much bigger that you'll never lack for MB options, available in every shape and size. Lots of niche needs can be filled that way - need a mini-ITX motherboard to fit in a tiny box? Epyc's got em. Dual processor 16 DIMM behemoth? Epyc's the way. Need a specific NIC chipset onboard? You'll probably find one on an epyc MB by some vendor "somewhere". However, I feel that's the only real benefit to it, at least personally... If any of the sWRX80 MB's suit your needs, I feel like TR Pro is the way to go. You get onboard audio (specific review of the audio can be found here, at least for all wrx80 boards minus the newer MSI unit), usually better IO / peripheral connectivity, and generally a better likelihood of the board's individual components/chipsets being tested with 'consumer applications / hardware'. Since they're workstation boards, they're more likely to be tested with components we (as home users) might consider 'normal use' (consumer graphics cards, etc). Essentially you get the best of both worlds - ECC memory, IPMI, loads of memory channels and PCIe lanes (from the server side), along with all the things one would expect when building a computer for themselves these days (onboard audio, bunch of USB ports, etc). Some other points - You might check out some other cooler options - since you're in the 4u rackmount space, might check out dynatron or supermicro's units. There's definitely a price gap between the 16GB and 32GB DIMMS, at least on the second-hand market. I've no problems buying used memory personally, and have seen 16GB sticks go for as little as $2.80-3/GB on various swaps and the like, while 32GB DIMMs seem to typically stay closer to the $5-6/GB range. With 8 channels, you can get your 128GB with 16GB DIMMs, and maybe save some $$$ if you're patient The 'TDP' values listed for TR Pro are all just nonsense imo - I don't think they ever tested them individually, instead opting for "well, this socket supports 280w, so everything's 280w". Consider the listed TDP as the 'worst case scenario'. If you're looking to make it through the next 5+ years, I'd ditch the norco at some point. When they work, they seem to be great. But as the company's dead (along with all the nightmare scenarios I've come across from other users), I'd replace it before it dies; sell it off, and use the recouperated $ to help fund a different chassis from a vendor that's still operational. I obviously tend toward supermicro, as all their stuff is practically lego-like... I still have the first chassis I ever bought from them over 10 years ago, as I just upgraded the backplane from what it came with [3Gb/s SATA] to a 12Gb/s SAS unit, and it's still happily chugging along. But Chenbro, Asus, and several others are options as well. Just my .02 - best of luck with the build, and keep us posted if you would!

Wow, I'm super delayed in responding, sorry about that! Probably no longer useful, but figured I'd respond anyway for posterity: Idle power consumption runs around ~200w, which is pretty typical for me - that's with a couple disks spun up for folks watching plex (direct streams / no transcoding). The highest I've ever seen it pull was just over 670w - this was during a parity check while playing some games on the 2070 (and a whole bunch of other crap). The TR Pro boards only 'support' for ECC insofar as I'm aware, so right now it's 128GB of registered ECC memory. That 763w though... That seems awful high? Is that your max power consumption I assume? Promise I'll be a little more prompt on the next response lol

I was working on typing up a follow-up to the sr-iov thread I did a while back, and got to wondering about the value of it (e.g. how many people here are using a given NIC type). Mine are all Intel or Chelsio, but I see a lot of broadcom/aquantia/mellanox as well mentioned in the forum, and wondered weather one on broadcom (or even chelsio) would be helpful... For example: Intel Chelsio Mellanox Broadcom Aquantia Realtek Marvell

Same benefits with 1Gb - just ensure that whatever intel nic you get is both genuine and has sr-iov support per intel's ark page listing and you'll be fine. They're more expensive, but I typically go straight for new i350 intel oem cards these days. Too much risk of fakes out there on the used market, no matter how careful you are.

The biggest advantage is performance. I've been able to get north of 1.1m IOPs out of mine, something thatd be nearly impossible with any btrfs setup. The disadvantage though is that if one simply dives into zfs without doing any tuning (or worse, just applies something they read online without understanding the implications), they're likely to end up worse off than they would've with virtually any other filesystem/deployment.

4 drives in z2 is always going to be awful. As mentioned above, if only using 4 drives, you should mirror them so you dont have the z2 overhead in a place it doesnt make any sense. Once you get the other 4 drives, then you can redeploy with z2. I'd recommend spending some time doing a bit more research before jumping into zfs though personally - jumping in with both feet is fine, but if unprepared, you'll likely fail to realize the filesystems full benefits.