There is no doubt that the rapid falling price of SSDs makes these valid questions. SSDs are the top of any performance/dollar chart. Large SATA drives are the top of any GB/dollar chart.
1a) No, the VMs should run from SSDs, and the NAS should be bigger+slower HDD. Cheap SSDs are not a good place to start, as they are low on the performance curve. You said put money into speed and reliability, that means no cheap SSDs.
1b) True, but it is likely to be less costly to have a fast set of cache SSDs, than a full set of SSDs for everything.
1c) I think this is the route to look into, solid fast SSDs for cache, and Dockers, and VMs. Then HDD for expansive storage.
2) What is not clear is your space requirement for VMs, or even cache. As mentioned above, this is the area to focus on for performance.
3a) I would not mix SSDs and HDDs in the array, even if supported. The SSD will perform for reads, but not writes. The BTRFS based cache pool will allow the VMs to continue to run after an SSD failure.
3b) Mentioned above, I don't think it makes sense to mix SSDs and HDD in the array.
5) Potentially the RAID1/5 (don't use RAID0) could be used for the cache drive instead of BTRFS.
Roughly, I would suggest a pair of 900+GB SSDs in RAID1, for cache and VMs, and three HDDs (3TB-6TB each). But that vastly oversizes the 2-4TB as 6-12TB.
If you really mean 4TB max, not going to need to grow it, you might look at dropping unRAID and use a fast controller with battery backed cache and 6 or more SSDs. Without need to grow, and using all SSD, unRAID's major features (easy expansion and spin down) are not needed.
Also expect more real offerings in the NVMe space this year.