In my opinion, about as much as you worry about getting hit by a meteorite on any given day.
Yes, these are all security vulnerabilities. Are they likely? Probably not. They are proof of concepts. Are they even likely on a home server? Really doubtful.
Sure, in *theory* a piece of malicious software that manages somehow to get itself and installed on your server (very remote odds to begin with) would technically be able to discern through very careful timings the contents of memory (not hard drives, but RAM). But would that information actually be useful? Once again extremely unlikely.
These mitigations when push comes to shove are most for data centers because it would allow (in theory) someone to rent out time on a VM running on a server and be able to discern the data that another company has on their VM running on that same server.
Intel's recommendation to completely disable hyperthreading is to render yourself safe is ignored by everyone.
Personally, I disable all the mitigations for these security concerns that aren't handled via micro-code. I'd rather have the speed of my processor back to what it's supposed to be.
But, if you are a member of ISIS, the Black Panthers, or (god forbid) the Mickey Mouse Club then it might not be a bad idea to use all the mitigations and also disable hyperthreading as the NSA obviously has their fingers on you. They are the ones after all who are responsible for the most malicious piece of software ever (Stuxnet)