Hi,

 

I'm currently running a fairly new / clean install of Unraid 6.4 and am finding that the time (accessible via Settings > Date and Time) runs "slow". I will reset the time by either setting it manually, or by changing the UTP servers. After about ~20 mins though, it's already noticeably behind. If I leave it a few hours, it's a few minutes behind and the difference keeps getting larger and larger the longer I run it.

 

I've attached some diagnostics in case that's useful. But any help would be greatly appreciated. As the time goes more and more out of sync, various services start failing (e.g. syncing to Amazon S3 can fail if the local and remote clock are too far out of sync). 

 

Thanks in advance

Ian

tower-diagnostics-20180121-1625.zip

Does the erroneous clock speed survive a reboot?

 

ntp that is responsible for keeping the time also keeps track of how much drift a specific hardware has. Once in a while, it can totally fail with this and compute a horrendous drift value making the clock run very much too fast or slow.

 

ntpd keeps the drift value in /var/lib/ntp, but I have never checked if unraid saves the content of this directory on reboot.

5 minutes ago, pwm said:

ntpd keeps the drift value in /var/lib/ntp, but I have never checked if unraid saves the content of this directory on reboot.

 

yes, drift is stored on flash (see /config folder) and used on reboot.

Just now, bonienl said:

 

yes, drift is stored on flash (see /config folder) and used on reboot.

 

I must be bad at searching - I do find the configuration settings in the /boot/config folder but no file that seems to store any drift value. Or maybe it gets deleted on every boot after the value has been copied to RAM?

 

 

But I have two times had to clear the drift value for other machines where ntpd has goofed and mistaken some manual time update as drift that should be corrected.

Hey,

 

Thanks for the responses!

 

>Does the erroneous clock speed survive a reboot?

 

Yeah, i've rebooted the machine multiple times and have seen this issue again and again.

 

>drift is stored on flash

 

I've checked the flash drive, and /var/lib/ntp and both locations have a "drift" file (which contains "0.000"). As mentioned above, this happens with NPT enabled in my settings, but also with it off and just a date / time set manually. I assume the contents of my drift file(s) means that it thinks the machine clock is completely in sync with the ntp server(s)?

 

 

As you can see, it's gone out of sync by ~4 mins since I last synced (which was when I first created this forum post)

 

Ian

Edited January 21, 20188 yr by IanB
Added screenshot

Browsing around, I thought this info would be useful:

root@Tower:/mnt/user# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l    3   64    1    0.000    0.000   0.000
 time3.google.co .GOOG.           1 u    1   64    1  138.324  300111.  85.763
 li1535-196.memb .INIT.          16 u    -   64    0    0.000    0.000   0.000

I noticed that when I modify the set of NTP servers, the drift file disappears (from /var/lib/ntp/) so I guess it gets re-created later? This was run right after I added another NTP server, which means my clock is now apparently in sync. I can run it again in ~20 mins to see if it's drifted out.

Edited January 21, 20188 yr by IanB

Actually, only leaving it a few minutes shows that it's drifting:

root@Tower:/mnt/user# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l   29   64   77    0.000    0.000   0.000
 time3.google.co .GOOG.           1 u   48   64   17  114.412  12742.3 9923.26
 li1535-196.memb 89.101.218.6     2 u   16   64   77  148.922  14095.7 10737.0

 

Edited January 21, 20188 yr by IanB

I tried running `ntpdate` manually and got the following:

root@Tower:~# ntpdate
21 Jan 19:07:43 ntpdate[15518]: no servers can be used, exiting

Is this expected? I checked the `ntp.conf` file and it contained a number of servers defined at the bottom:

 

# Generated entries follow:
server time.google.com iburst
server pool.ntp.org iburst

 

Probably the ntp service is running, that's why ntpdate can't open the socket (port 123 UDP) and connect to ntp server.

Try from command line:

sudo service ntp stop
sudo ntpdate -s time.nist.gov
sudo service ntp start

Thanks for the suggestion. Unfortunately, it complains about "service":

 

root@Tower:~# sudo service ntp stop
sudo: service: command not found

 

I managed to stop, update, and restart manually doing the following:

 

root@Tower:~# /etc/rc.d/rc.ntpd stop
Stopping NTP daemon...
root@Tower:~# ntpdate -s time.nist.gov
root@Tower:~# /etc/rc.d/rc.ntpd start
Starting NTP daemon:  /usr/sbin/ntpd -g -u ntp:ntp

 

My time is now correctly in sync, but will have to now watch to see how it drifts out. Is there a way for me to validate that the NTP service is periodically updating from the remote servers?

This is also working:

ntpdate -u pool.ntp.org

 

Put this in a cron and let it run every hour.

Edited January 21, 20188 yr by Zonediver

Since I last ran the `ntpdate` I saw that I was already a few seconds now out of sync. I ran the command, as you suggested (the ntpd service was still running) and it updated:

 

root@Tower:~# ntpdate -u pool.ntp.org
21 Jan 19:37:44 ntpdate[26004]: step time server 87.81.181.2 offset 26.491838 sec

 

Does this suggest that everything appears to work when updating from a remote ntp server, but for some reason, it's not periodically happening?

15 minutes ago, IanB said:

Since I last ran the `ntpdate` I saw that I was already a few seconds now out of sync. I ran the command, as you suggested (the ntpd service was still running) and it updated:

 

root@Tower:~# ntpdate -u pool.ntp.org
21 Jan 19:37:44 ntpdate[26004]: step time server 87.81.181.2 offset 26.491838 sec

 

Does this suggest that everything appears to work when updating from a remote ntp server, but for some reason, it's not periodically happening?

 

There must be something wrong with your machine or hardware.

When i run the command, i have a gap of 0.000631sec within 5min's

Check the voltage of your BIOS-Battery

Edited January 21, 20188 yr by Zonediver

>There must be something wrong with your machine or hardware.

 

That's worrying  The hardware is all pretty new, Asus x299 Prime Deluxe with a Intel 7980XE...

7 minutes ago, IanB said:

>There must be something wrong with your machine or hardware.

 

That's worrying  The hardware is all pretty new, Asus x299 Prime Deluxe with a Intel 7980XE...

 

Take a multimeter and check the voltage of the batterie.

This actually looks pretty relevant: https://bugzilla.kernel.org/show_bug.cgi?id=197299

11 minutes ago, IanB said:

This actually looks pretty relevant: https://bugzilla.kernel.org/show_bug.cgi?id=197299

 

Also possible. Deactivate spread spectrum and see what happens.

Edited January 21, 20188 yr by Zonediver

>Possible, but related to Intel Skylake

 

Yeah, it's an issue with Skylake X - like the 7980XE that i'm using. Sounds very similar... If the bug turned out to be this kernel issue, how frequently does Unraid receive updates for kernel patches, etc? I've not been running it long enough to know what their update strategy is.

 

Thanks again for the help!

 

Just thought i'd give a quick update, in case anyone else in the future hits this issue...

 

After reading about the kernel issue, I first tried updating my BIOS as my motherboard shipped with a pretty old version. This didn't make any difference. From the description of the kernel issue, it talks about it only happening when Spread Spectrum is enabled, for which my motherboard supported being manually disabled. After I did this though, I unfortunately still found that the clock would drift. There is however a linked (dupe) issue here which suggested another workaround: https://bugzilla.kernel.org/show_bug.cgi?id=198137

 

I was able to follow these steps, and resolve the clock drift:

 

/etc/rc.d/rc.ntpd stop
sudo adjtimex -p
** The tick parameter shows set to 10000
sudo adjtimex -c
** Shows a correct tick value of 10400 (ie.. +4% adjustment)
audo adjtimex --tick 10400
sudo adjtimex -p
** Now checking the command-line "date" to the wall clock shows no appreciable drift over a few minutes.
/etc/rc.d/rc.ntpd start

My server has been running the full day without any noticeable drift. I'm sure that after i reboot, it may lose the adjtimex value and if so, i'll have to look at how to add it to my /config/go. Will keep an eye out for Unraid updates which include any kernel patches for this bug, and test them.

On 1/21/2018 at 1:27 PM, IanB said:

I managed to stop, update, and restart manually doing the following:

 

root@Tower:~# /etc/rc.d/rc.ntpd stop
Stopping NTP daemon...
root@Tower:~# ntpdate -s time.nist.gov
root@Tower:~# /etc/rc.d/rc.ntpd start
Starting NTP daemon:  /usr/sbin/ntpd -g -u ntp:ntp

 

My time is now correctly in sync, but will have to now watch to see how it drifts out. Is there a way for me to validate that the NTP service is periodically updating from the remote servers?

Thanks.  This fixed it for me.  Now to see if it goes out again...