To re-clarify, we are running the Windows version, on Server 2016. We are not currently running the Linux QDSM (but would like to in order to eliminate a Windows VM).
I might try to put a drawing together in Visio to help illustrate, but... ESXi 6.7 is the operating system on the SuperMicro. It is designed to do little other than run virtual machines, including Windows Server 2016 & Photon 3.0 in our case. We are already running a handful of Photon 3.0 VMs with Docker containers and they work amazingly. In one case we replaced a Windows certificate authority server with a Photon using OpenSSL for CA; this reduced the RAM and CPU utilization from 8GB to 20MB and a couple GHz to 0Hz most of the time (and saved ourselves a Windows OS license to boot).
So, I'm 100% confident I can throw another Photon VM and stick Docker containers in it. All I would need in my case is a containerized version of QDSM, (which it seems may not work until we cough up more money for Enterprise licensing, so I could be dead in the water already). ESXi stores virtual disks in "datastores", which can be a local disk or a NAS/SAN, so the fact that the virtual disks will be on a NAS is mostly for reference; it's not a potential limitation for a virtual environment.
We have a bunch of customers' QBD files on a network share, so our "database" is a whole bunch of random QB? files. The share includes a fair amount of backed up company files as well as all the temp files Quickbooks creates, so it's hard to know what our true "database size" is. It seems we have about 11.5GB of QBW files, which may be the closest thing I've got to an accurate count. It is a Windows Server 2016 file server with all the roles and features installed for CIFS and DFS replication (although we don't currently have a secondary file server stood up for replicating anything to). The file server VM is allocated 4 CPU cores and 8GB RAM; it is using right around 2.1GHz and ~400MB RAM, according to the virtual monitoring in ESXi. With all that said, I have to believe most of that is consumed by the Windows overhead, and although I can speculate and say I believe a Linux VM running QDSM and Samba would need a fraction of those resources, I cannot say for certain how much it would truly need.
With that great big disclaimer in mind, I would think you've got plenty of CPU/RAM to at least try deploying it on your NAS in a container, providing you're able to deploy containers in your NAS (I haven't messed with ContainerStation so I don't know how easy it is/isn't). If it's too much for your system you can always shut down the container and go back to what you're doing today. I think the difficult part here is that no one has created a containerized QDSM yet.