I tried to compare settings with another server that doesn't experience this problem, while doing so I noticed that the problem disappeared after editing the settings since it restarts the ntpd.
I wonder if it helps in your case, it might hint where the problem is.
Settings -> Date and Time --> edit something (you can just add a char to the ntp server and delete it) & Apply