No, definitely not because if you do a Host pass through it see the same CPU.
Maybe it's really related to the filesystem, I can't tell for sure because it works flawlessly on my system... And I've now also tried it on my test system:
This is my test server with the container running, as you can see my test server also has 8GB of RAM installed and it works also flawlessly.
The only difference is that I've installed the game to a BTRFS Cache drive, I really can't tell why it won't work on your system.
Here is the log from my test server: garrysmod.txt