Date and Time Clock slowly goes out of sync


IanB

Recommended Posts

Hi,

 

I'm currently running a fairly new / clean install of Unraid 6.4 and am finding that the time (accessible via Settings > Date and Time) runs "slow". I will reset the time by either setting it manually, or by changing the UTP servers. After about ~20 mins though, it's already noticeably behind. If I leave it a few hours, it's a few minutes behind and the difference keeps getting larger and larger the longer I run it.

 

I've attached some diagnostics in case that's useful. But any help would be greatly appreciated. As the time goes more and more out of sync, various services start failing (e.g. syncing to Amazon S3 can fail if the local and remote clock are too far out of sync). 

 

Thanks in advance

Ian

tower-diagnostics-20180121-1625.zip

Link to comment

Does the erroneous clock speed survive a reboot?

 

ntp that is responsible for keeping the time also keeps track of how much drift a specific hardware has. Once in a while, it can totally fail with this and compute a horrendous drift value making the clock run very much too fast or slow.

 

ntpd keeps the drift value in /var/lib/ntp, but I have never checked if unraid saves the content of this directory on reboot.

Link to comment
Just now, bonienl said:

 

yes, drift is stored on flash (see /config folder) and used on reboot.

 

I must be bad at searching - I do find the configuration settings in the /boot/config folder but no file that seems to store any drift value. Or maybe it gets deleted on every boot after the value has been copied to RAM?

 

 

But I have two times had to clear the drift value for other machines where ntpd has goofed and mistaken some manual time update as drift that should be corrected.

Link to comment

Hey,

 

Thanks for the responses!

 

>Does the erroneous clock speed survive a reboot?

 

Yeah, i've rebooted the machine multiple times and have seen this issue again and again.

 

>drift is stored on flash

 

I've checked the flash drive, and /var/lib/ntp and both locations have a "drift" file (which contains "0.000"). As mentioned above, this happens with NPT enabled in my settings, but also with it off and just a date / time set manually. I assume the contents of my drift file(s) means that it thinks the machine clock is completely in sync with the ntp server(s)?

 

5a64d1fd05c84_2018-01-21at17_46.png.64985b577b40f1af2fb5a9b21b8dfae1.png

 

As you can see, it's gone out of sync by ~4 mins since I last synced (which was when I first created this forum post)

 

Ian

Edited by IanB
Added screenshot
Link to comment

Browsing around, I thought this info would be useful:

root@Tower:/mnt/user# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l    3   64    1    0.000    0.000   0.000
 time3.google.co .GOOG.           1 u    1   64    1  138.324  300111.  85.763
 li1535-196.memb .INIT.          16 u    -   64    0    0.000    0.000   0.000

I noticed that when I modify the set of NTP servers, the drift file disappears (from /var/lib/ntp/) so I guess it gets re-created later? This was run right after I added another NTP server, which means my clock is now apparently in sync. I can run it again in ~20 mins to see if it's drifted out.

Edited by IanB
Link to comment

Actually, only leaving it a few minutes shows that it's drifting:

root@Tower:/mnt/user# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l   29   64   77    0.000    0.000   0.000
 time3.google.co .GOOG.           1 u   48   64   17  114.412  12742.3 9923.26
 li1535-196.memb 89.101.218.6     2 u   16   64   77  148.922  14095.7 10737.0

 

Edited by IanB
Link to comment

I tried running `ntpdate` manually and got the following:

root@Tower:~# ntpdate
21 Jan 19:07:43 ntpdate[15518]: no servers can be used, exiting

Is this expected? I checked the `ntp.conf` file and it contained a number of servers defined at the bottom:

 

# Generated entries follow:
server time.google.com iburst
server pool.ntp.org iburst

 

Link to comment

I managed to stop, update, and restart manually doing the following:

 

root@Tower:~# /etc/rc.d/rc.ntpd stop
Stopping NTP daemon...
root@Tower:~# ntpdate -s time.nist.gov
root@Tower:~# /etc/rc.d/rc.ntpd start
Starting NTP daemon:  /usr/sbin/ntpd -g -u ntp:ntp

 

My time is now correctly in sync, but will have to now watch to see how it drifts out. Is there a way for me to validate that the NTP service is periodically updating from the remote servers?

  • Like 1
Link to comment

Since I last ran the `ntpdate` I saw that I was already a few seconds now out of sync. I ran the command, as you suggested (the ntpd service was still running) and it updated:

 

root@Tower:~# ntpdate -u pool.ntp.org
21 Jan 19:37:44 ntpdate[26004]: step time server 87.81.181.2 offset 26.491838 sec

 

Does this suggest that everything appears to work when updating from a remote ntp server, but for some reason, it's not periodically happening?

Link to comment
15 minutes ago, IanB said:

Since I last ran the `ntpdate` I saw that I was already a few seconds now out of sync. I ran the command, as you suggested (the ntpd service was still running) and it updated:

 


root@Tower:~# ntpdate -u pool.ntp.org
21 Jan 19:37:44 ntpdate[26004]: step time server 87.81.181.2 offset 26.491838 sec

 

Does this suggest that everything appears to work when updating from a remote ntp server, but for some reason, it's not periodically happening?

 

There must be something wrong with your machine or hardware.

When i run the command, i have a gap of 0.000631sec within 5min's

Check the voltage of your BIOS-Battery

Edited by Zonediver
Link to comment

>Possible, but related to Intel Skylake

 

Yeah, it's an issue with Skylake X - like the 7980XE that i'm using. Sounds very similar... If the bug turned out to be this kernel issue, how frequently does Unraid receive updates for kernel patches, etc? I've not been running it long enough to know what their update strategy is.

 

Thanks again for the help!

 

Link to comment

Just thought i'd give a quick update, in case anyone else in the future hits this issue...

 

After reading about the kernel issue, I first tried updating my BIOS as my motherboard shipped with a pretty old version. This didn't make any difference. From the description of the kernel issue, it talks about it only happening when Spread Spectrum is enabled, for which my motherboard supported being manually disabled. After I did this though, I unfortunately still found that the clock would drift. There is however a linked (dupe) issue here which suggested another workaround: https://bugzilla.kernel.org/show_bug.cgi?id=198137

 

I was able to follow these steps, and resolve the clock drift:

 

/etc/rc.d/rc.ntpd stop
sudo adjtimex -p
** The tick parameter shows set to 10000
sudo adjtimex -c
** Shows a correct tick value of 10400 (ie.. +4% adjustment)
audo adjtimex --tick 10400
sudo adjtimex -p
** Now checking the command-line "date" to the wall clock shows no appreciable drift over a few minutes.
/etc/rc.d/rc.ntpd start

My server has been running the full day without any noticeable drift. I'm sure that after i reboot, it may lose the adjtimex value and if so, i'll have to look at how to add it to my /config/go. Will keep an eye out for Unraid updates which include any kernel patches for this bug, and test them.

  • Like 2
Link to comment
  • 2 years later...
On 1/21/2018 at 1:27 PM, IanB said:

I managed to stop, update, and restart manually doing the following:

 


root@Tower:~# /etc/rc.d/rc.ntpd stop
Stopping NTP daemon...
root@Tower:~# ntpdate -s time.nist.gov
root@Tower:~# /etc/rc.d/rc.ntpd start
Starting NTP daemon:  /usr/sbin/ntpd -g -u ntp:ntp

 

My time is now correctly in sync, but will have to now watch to see how it drifts out. Is there a way for me to validate that the NTP service is periodically updating from the remote servers?

Thanks.  This fixed it for me.  Now to see if it goes out again...

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.