in short, yes, I used way beyond 32GB in windows. I didn't get to trying a different distro of Linux. HOWEVER I did get it to work in unraid by moving the 16GB modules from channels C and D to A and B. I did one start with just the 16GB modules saw 64GB then put the rest back in and have 96GB in unraid. Kind of strange because I reseated it twice the way it was and it didn't work. It looks like somewhere something didn't like the 16GB modules being in channels C and D. May have to do with my motherboard and how it reported the modules to unraid. For reference the Mother Board is a Gigabyte x99P-SLI and it is worth noting that some gigabyte boards (this included) are known to have strange issues where trident z RGB memory does not show on half of the channels in the LED software and it is unable to modify the SPD data on the RAM. although I have trident Z, the 16GB DIMMs are NOT-RGB.
So in hindsight maybe this belonged on another topic but it may still help people if they have a similar issue.
(Edit)
I just noticed the original post also used a Gigabyte board as well. And although people concluded it looked like the memory modules that weren't compatible based off the QVL, some people may have something else happening. Someone mentioned configuration before, in my case it didn't even want to post if a 16GB dim was in the same channel as a 8GB (understandable, just saying); And then the issue I had where moving memory channels fixed it. So making sure that they match based off of what channels they are in may matter, and even then toying with what is in which channel may help.