Yeah, it's getting ridiculous now.
So for Spectre / Meltdown. First you have to have the appropriate malware running locally on your server. (Which BTW, they have never found a single instance of). Then, it has to perform its attack, which will absolutely let it gather what ever info happens to be in the CPUs cache at the time. But, what exactly are the odds of the information that is actually of any value being in the cache at the exact same time as it runs?
Sure, it can request to look at all of the memory locations in your server, and then determine what all the contents are. But, is it going to be able to actually discern what is a password or not? What is simply code? What is simply part of your movie file cached? I mean, I don't know about you, but I tend to not have in ram the phrase: "blahblahblah is my password for RoyalBank, and my account number is 1234".
Sure, it's all possible that it will get a password (if you happen to have one stored on your server, and it is in RAM at the same time). But what exactly are the odds? My feeling is that I'd probably win the lottery two or three times before those attack vectors could discern and analyse anything meaningful from my system. But, the story is different for something like a bank where personal info would constantly be in a cache. Let them take the potentially 30% hit in CPU speed. Myself, I'm going to keep on buying lottery tickets. I'll bet you I come out ahead.