BVD

The limitation of IOMMU grouping is always on the side of the motherboard (I usually hesitate to make statements like this as fact, but I think it's pretty close here). The ability to segment to individual traces down is completely dependent upon the PCB design and BIOS implementation, which is on the MB side. The 'downside' to enabling the override is, at least in theory, stability - what we're doing is 'ignoring' what the motherboard says it should support. In reality, if you enable it and everything 'just works', it'll more than like continue to 'just work' indefinitely (barring any changes to the MB BIOS). Unfortunately IOMMU grouping and how it all actually works is beyond the scope of this tutorial, but I agree it's a subject that could use clarification. A lot of it boils down to hardware implementation and optionROM size the MB vendor implements into its design - most consumer boards, they only have enough space to load the fancy graphical BIOS they come with, where workstation/server boards still tend towards keyboard driven UI's (leaving more space for other operations).

Whether or not PCIe ACS override is needed is widely system dependent - any time you hit an issue with IOMMU grouping, it's one of the options that are available, but unfortunately not a one size fits all. Glad you got it figured out!!

I've an theory on that, but it's probably an unpopular one - ZFS has gotten so much more popular in recent years, with a lot of folks diving in head first. Hearing how it protects data, the ease of snapshots and replication, how easy it makes backups, that they migrate everything over without first learning what it really is and isn't, what it needs, how to maintain it, and what the tradeoffs are. Then when something eventually goes sideways (neglected scrubs, power outage during a resilver, whatever, they change xattr with data already in the pool, set sync to disabled for better write performance, any number of things both environmental or user inflicted), the filesystem is resilient enough that it 'still works', so it's expected that anything on it still should as well... Hell, I've been neck deep in storage my entire career, and using ZFS since Sun was still Sun, and I *STILL* find myself having to undo stupid crap I did in haste on occasion. The fact that you partitioned the disks first wouldn't change any functional behavior in the driver (similar to running zfs on sparse files, the same code/calls are used). Either 'it's fixed', or I'd simply taxed the driver beyond what it'd optimized for at the time, at least that's my feeling anyway.

Are you running 6.10? I've not tested it since 6.8.3, so perhaps there were updates to the docker zfs driver in between that've made a difference, I'm not sure. I'd say it's more likely though that you're just not 'pushing the limits', so any inefficiencies never really bubble to the surface. I will say though, I likely would've been in a similar situation with btrfs. At the time, I was running some ~160 containers on one system (testing for some work stuff in addition to my home environment), and it got to the point where any time one of the containers was restarted, there were significant latency spikes, even though the hardware layer still had loads of overhead available. I tracked it as far as a long running TXG (but before attempting to commit the TXG) before I realized I not only didn't have time for this, but was way the hell out of my league lol. Something funky going on in spacemap, but I'd no idea what. To me though, it makes some sense - why expend the resources necessary to create a snapshot of every single layer of every single container every single time for no benefit really given the containers are disposable (comparatively higher transactional expense to both CPU and filesystem), when you can instead just give it a single entrypoint and let the lower level IO scheduler handle it with an image or block device? Honestly for most folks it probably doesn't matter though - a single current gen NVME device can handle enough IO these days that even with any number of inefficiencies in configuration, they likely wouldn't notice. And if they do, it'll likely be some higher level code efficiency that crops up before the storage starts to give em any headaches lol. "Back in my day, we had to pool 80 drives, manually align blocks to page sizes, and add a SLOG just to get enough IOPs that the DBA's didn't burn us at the stake! Don't get me wrong, they were still out for blood, and if you saw them at the coffee machine when you went for your refill, you still turned around and came back later... But at least they just wanted you harmed, and not dead!" 🤣

You've got a couple options - 1. Create a zvol instead, format it, and keep docker writing there (which is now a block storage devices) 2. Your plan, creating a fileset and using a raw disk image In either case, couple things you can do - * redundantmetadata=most - containers are throwaway, no real reason to have doubly redundant metadata when you can just pull the container back down anyway; wasted resources * primarycache=none (or metadata at most) - containers might be (probably imperceptably given you're on NVME and not SATA) slower to initially start, but once they do, the OS is already handling memory caching anyway, leaving ZFS often duplicating efforts (and using memory by doing so) * sync=standard - again, containers, who cares if it needs to be re-pulled * atime=off - (I do this for everything) I've got a whole whack of performance tuning notes lying around for one thing or another - if the forum ever gets markdown support, I'll post em up, maybe then can be helpful to someone else lol

I think you probably meant to post that for the ZFS Master plugin? This one just adds the ZFS driver. Maybe there's a hover over help menu or something? Not sure. To the question at hand though, it looks like you're using the docker ZFS driver (using folders instead of .img) - personally, I'd recommend against that. The data within these directories are just the docker layers that get rebuilt each time you update a container. Doing it this way just makes managing ZFS a mess, as you end up with all sorts of unnecessary junk volumes and/or snapshots listed out every time you do a zfs list / zfs get / etc. Plus, it creates so danged many volumes that, once you get to a decent number of containers (or if you've automated snapshots created for them), filesystem latencies can get pretty stupid.

After finding that my go file was getting out of hand, I ended up setting up a script that kicks off at first array start instead which walks through symlinking all the various stuff I had in there instead, just so I could keep the go file limited to things that actually have to be ran at init (or at least 'things that make the most sense at init' lol). At this point, it links my bash and tmux profiles, iterm shell integration, some cron, several other things - this way at least the UI is relatively tidy lol. I'd found some areas where just having everything on the NVME zpool and linking it afterwards just seemed to help with initial startup times and the like. Maybe if user scripts ever gets the ability to pul things in folders I'll change it up, but I spend most of my time in the cli anyway, so I guess we'll just see what happens 🤷‍♂️

Honestly that'd be more than I'd want in the container, at least myself - you can manually do this a few ways, but if it's important to you, there are definitely a few options out there you can investigate and work through! 👍

Negative - the plugin's current design checks on reboot so he doesn't have to generate a new plugin each time a new ZFS release comes out.

No issue with asking him to do so... I just hate asking someone else to do something if I can do it myself lol I guess it might make some sense though now that I think about it a bit more... with zfs support eventually coming, it might actually make sense to have it built into Nerd Tools than to just have on my own separated from the rest lol

After looking at @dmacias's repo, seems like it'd be easy enough to build. Guess I know what I'll be doing this weekend lol

@steini84 or @ich777 would it be possible to include ioztat with the plugin, or do you feel it's better served to something like NerdPack? I've been using it since it's inception, super helpful for quickly tracking down problem-child filesets: https://github.com/jimsalterjrs/ioztat It does require python, but that's the only thing outside of the base OS that's required for us (though I don't know if that requirement precluded it from inclusion, hence the NerdPack comment...) Something else I've been symlinking for a while now is the bash-completion.d file from mainline zfs - it's 'mostly' functional in openzfs, though I've not spent a lot of time poking around at it. https://github.com/openzfs/zfs/tree/master/contrib/bash_completion.d

@BoKKeR Best bet would be to use pip instead - both python and pip are included in dev tools, and it looks like it's one of the supported ways to install tldr

There should be some additional lines above that that'll tell you what exactly is going on - believe it means you're trying to use a container which doesn't support one of the streams used in the source media, while having 'force compliance' set to false. Set it to true, and it'll just drop whatever can't be translated to the chosen container (happens most often with mp4 [which is like the 'universally supports damned near everything' container] -> mkv/etc [which has more restrictions] if I'm remembering correctly)

Having the container means you don't have to care about anything other than having a browser to access. I don't mind paying for good software. I do mind paying a monthly subscription just so I can access the same content on all my devices.

I've mine mapped to the Notes folder from my nextcloud location - you'll have to set up nextcloud to scan for file changes on occasion for this to work properly, but once done, it works out pretty well.

We must be on the same wavelength - I literally just started building my own container for this one, after discovering obsidian a couple weeks ago thanks to Wendell over at Level1Techs... Only to find it in CA this morning! Great work, and thanks!

@SpencerJ just wanted to check in with you as I've been eagerly awaiting the next episode, but it's been quite a while now since the last one, and the one prior was a little later than the previous cadence.... Has the uncast been retired, or perhaps on hiatus for now? Hope it's still going, and looking forward to to the next episode if so!

@ich777 I actually went a similar route tbh - I'd had everything running in one box, and every time I had to take it down for whatever reason (adding a SAS card for the disk shelf, pulling one of the GPUs for another project, etc) it was something of a planning ordeal. These days I have two unraid boxes (the primary, then my older setup acting as something of a backup vault / tinkering playground) as well as one proxmox host. * CARP ensure's routing's always available through the pfsense VMs (one on unraid, one on proxmox) * zfs snapshots ensure that the proxmox host's datasets are never more than 10 minutes out of date (just for those "tier 1" [home edition, lol] services) * if the proxmox host detects any of those services are down for whatever reason, it starts up it's own copies of them locally. Kinda stole the idea from one of @SpaceInvaderOne's old pfsense videos and adapted it. There's some additional logic there to ensure that it only self initiates starting its own services if it's not already started them due to my shutting down the array on purpose, but that's the gist. I just think given how rare it is for folks to be running multiple NAS/servers at home (home NAS users are already niche, but then those running multiple are a niche within a niche), I see a lot of value in the idea of allowing VM's to run with the array down. Maybe once multiple arrays become a thing, it'll make more sense for Limetech to implement some logic like 'this VM doesn't rely on data in array X, so it remains running'. We'll see!

Guessing it's the way libvirt's utilized based on the review I'd initially done when trying to sort this out myself - it's definitely no easy task, but I'd certainly appreciate the effort!!

Any chance you've a static IP set for the container? There's a macvlan issue with some systems which causes in random crashes with that setup (I'd had one within 12 hours, then next didn't happen for 27 days), fixed in 6.10 by switching to IP VLAN.

With rumors regarding zen 3 TR Pro seemingly evaporating overnight (nothing new since December), I've been getting antsy. Sincerely hoping that all the March 2022 release date rumors leading up to the drought turn out to be true... Because in preparation for it's release, I've been deploying more and more of the workloads I'd had planned for the server, and now I've stacked enough on that the 16c 3955wx has been redlining somewhat regularly the last few weeks. Images of a super buff 4'5" body builder trying to backpack a volkswagen have come to mind. Given this, I've spun down had to disable Tdarr's CPU transcodes and just stick with GPU accelerated only. It's helped quite a bit, saving some headroom for those occasional spikes. Some current results on that front: Almost ready to add the last Plex library and queue it up. I figure the 1650S has pretty much paid for itself twice over at this point (at least at the price I'd paid for it back when anyway) - 16TB drives @ ~$320/per, with the 1650S costing me ~$170, I'm honestly pretty stoked! The first 'new' addition since moving to the new hardware was Paperless-ng - I had no idea how much it'd change our life to be honest. We went from having 2 banker boxes full of everything from tax paperwork, to paid bills, work reviews, etc, to about 3 inches of papers left. No more having to go into the boxes each and every time we have a bill come in the mail, no more time wasted trying to find 'that one bill that we know we paid but the stupid bank/lender/government/whatever say they never received payment for... Just plop it on the scanners feed tray, hit a button, and done. It's been a hell of a process to get here, about 5 months in the making as we're using this ancient HP business scanner and I'm too cheap to replace it - it's wicked slow at a decent quality setting, maybe a page every couple minutes... but we're in no hurry, the quality is perfect for Paperless's OCR (converting the scanned image to searchable text), it's able to scan directly to SMB, and it's got a 35 page feed tray. I just put another stack on a couple times a day and leave it at that. Paperless automatically tags them based on content, and we've not had to dig out a single page since setting it up. Next, I opened up the media request website to my sister, now that my parents have gotten the whole 'we can just submit it here and we don't have to change DVD's anymore!? *CLICK CLICK CLICK CLICK CLICK CLICK* " thing out of their system; that might've been a bit of a mistake: Nearly everything in TV_SaturdayMorning is hers, and she literally sat down for a full hour adding each and every one her series... I wasn't expecting that kind of growth lol. That 'last library' to add? ...That's it. There's more, but I think I gotta hit the sack - hoping to get more time to catch this thing up to date in the coming weeks! Brief preview of what I've still got to detail out: Another host - two hypervisors, routing, and a little room to spare Network refresh - IoT takes a bite out of wifi And a couple other odds and ends - several containers, storage crunch woes (thanks sis...), and repurposing the older xeon gold 6139 based system.

Reaffirming the home assistant note - I'd not really thought of it a great deal until recently, as we just finished purchasing and moving into our new home about 4 months ago, and have finally gotten it to the stage where we're ready to implement some of our plans for the place (connected garage door, motion detection and temp control, soil monitoring for the garden, etc)... Say you suffer a double failure - you want to minimize IO during the rebuild to mitigate (as much as one can) the potential for a third failure during reconstruction, so you start the array in maintenance mode. With 16TB drives becoming more commonplace, 20TB drives now available, and 24TB+ drives on the horizon, rebuild times are only going to get longer and longer. If we average 150MB/s across the entirety of reconstruction (whichever then is wildly optimistic), you're looking at ~1.5-2 days downtime. Not a huge deal for a media server. But for critical needs (like heating and cooling, home security, many others), thatd feel like a lifetime.

The kernel driver isnt ready for 12th gen at least as far as gpu partitioning is concerned. Haven't tried it with bleeding edge kernels yet or anything, but at least as of 5.13, the latest "clean/stable" iGPU supported is 10th gen.

I'm kind of hoping that the work they're doing towards moving the UI to a Vue event-driven system makes the whole mobile app idea moot by allowing them to easily code a mobile view/version and baking it in. The effort these mobile app devs have put in is admirable and definitely appreciated, but I think it's not super sustainable with how much and often needs within them evolve over time these days