Local/Universal time is not sync with RTC after reboot

hgupta · February 17, 2025, 5:30pm

Hi Team,

I have below test case:
AIM: disable the chronyd process, and set the date/time/timeZone of rocky to a specific values.

Expectation: Local/Universal and RTC time should be in sync and should be same as set by user.

Observation: once i set the date/time using timedatectl command, I observed that both RTC and Local/Universal are in sync, but when i reboot the system Local/Universal are set to other values were as RTC is still set to same time/date which i set using timedatectl command.

Logs:
Values before reboot

[root@localhost ~]# date
Mon Feb 17 06:18:55 PM IST 2025
[root@localhost ~]# timedatectl
               Local time: Mon 2025-02-17 18:18:59 IST
           Universal time: Mon 2025-02-17 12:48:59 UTC
                 RTC time: Mon 2025-02-17 12:48:59
                Time zone: Asia/Calcutta (IST, +0530)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
[root@localhost ~]# timedatectl show
Timezone=Asia/Calcutta
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=no
TimeUSec=Mon 2025-02-17 18:19:03 IST
RTCTimeUSec=Mon 2025-02-17 18:19:02 IST
[root@localhost ~]# systemctl status chronyd
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-02-17 18:03:28 IST; 16min ago
       Docs: man:chronyd(8)
             man:chrony.conf(5)
   Main PID: 639 (chronyd)
      Tasks: 1 (limit: 48952)
     Memory: 5.1M
        CPU: 137ms
     CGroup: /system.slice/chronyd.service
             └─639 /usr/sbin/chronyd -F 2

Feb 17 18:03:28 localhost systemd[1]: Starting NTP client/server...
Feb 17 18:03:28 localhost chronyd[639]: chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +D>
Feb 17 18:03:28 localhost chronyd[639]: Loaded 0 symmetric keys
Feb 17 18:03:28 localhost chronyd[639]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/drift
Feb 17 18:03:28 localhost chronyd[639]: Using right/UTC timezone to obtain leap second data
Feb 17 18:03:28 localhost chronyd[639]: Loaded seccomp filter (level 2)
Feb 17 18:03:28 localhost systemd[1]: Started NTP client/server.
Feb 17 18:17:21 localhost chronyd[639]: Source 162.159.200.1 replaced with 3.6.43.90 (in.pool.ntp.org)
[root@localhost ~]# cat /tmp/timeInfo
               Local time: Mon 2025-02-17 18:20:55 IST
           Universal time: Mon 2025-02-17 12:50:55 UTC
                 RTC time: Mon 2025-02-17 12:50:55
                Time zone: Asia/Calcutta (IST, +0530)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
[root@localhost ~]#

Disable the chronyd and set the time/date/timezone using timedatectl

[root@localhost ~]# systemctl disable chronyd
Removed "/etc/systemd/system/multi-user.target.wants/chronyd.service".
[root@localhost ~]# systemctl stop chronyd
[root@localhost ~]# systemctl status chronyd
○ chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:chronyd(8)
             man:chrony.conf(5)

Feb 17 18:03:28 localhost chronyd[639]: Loaded 0 symmetric keys
Feb 17 18:03:28 localhost chronyd[639]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/drift
Feb 17 18:03:28 localhost chronyd[639]: Using right/UTC timezone to obtain leap second data
Feb 17 18:03:28 localhost chronyd[639]: Loaded seccomp filter (level 2)
Feb 17 18:03:28 localhost systemd[1]: Started NTP client/server.
Feb 17 18:17:21 localhost chronyd[639]: Source 162.159.200.1 replaced with 3.6.43.90 (in.pool.ntp.org)
Feb 17 18:23:44 localhost chronyd[639]: chronyd exiting
Feb 17 18:23:44 localhost systemd[1]: Stopping NTP client/server...
Feb 17 18:23:44 localhost systemd[1]: chronyd.service: Deactivated successfully.
Feb 17 18:23:44 localhost systemd[1]: Stopped NTP client/server.
[root@localhost ~]#
[root@localhost ~]# timedatectl set-timezone "CST6CDT"
[root@localhost ~]# timedatectl set-time '2016-01-01 01:00'
[root@localhost ~]# cat /tmp/timeInfo
               Local time: Mon 2025-02-17 18:27:19 IST
           Universal time: Mon 2025-02-17 12:57:19 UTC
                 RTC time: Mon 2025-02-17 12:57:19
                Time zone: Asia/Calcutta (IST, +0530)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
[root@localhost ~]# date
Fri Jan  1 01:00:23 AM CST 2016
[root@localhost ~]# timedatectl
               Local time: Fri 2016-01-01 01:00:32 CST
           Universal time: Fri 2016-01-01 07:00:32 UTC
                 RTC time: Fri 2016-01-01 07:00:32
                Time zone: CST6CDT (CST, -0600)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
[root@localhost ~]# timedatectl show
Timezone=CST6CDT
LocalRTC=no
CanNTP=yes
NTP=no
NTPSynchronized=no
TimeUSec=Fri 2016-01-01 01:00:36 CST
RTCTimeUSec=Fri 2016-01-01 01:00:36 CST
[root@localhost ~]#
[root@localhost ~]# systemctl status chronyd
○ chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:chronyd(8)
             man:chrony.conf(5)

Feb 17 06:33:28 localhost chronyd[639]: Loaded 0 symmetric keys
Feb 17 06:33:28 localhost chronyd[639]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/drift
Feb 17 06:33:28 localhost chronyd[639]: Using right/UTC timezone to obtain leap second data
Feb 17 06:33:28 localhost chronyd[639]: Loaded seccomp filter (level 2)
Feb 17 06:33:28 localhost systemd[1]: Started NTP client/server.
Feb 17 06:47:21 localhost chronyd[639]: Source 162.159.200.1 replaced with 3.6.43.90 (in.pool.ntp.org)
Feb 17 06:53:44 localhost chronyd[639]: chronyd exiting
Feb 17 06:53:44 localhost systemd[1]: Stopping NTP client/server...
Feb 17 06:53:44 localhost systemd[1]: chronyd.service: Deactivated successfully.
Feb 17 06:53:44 localhost systemd[1]: Stopped NTP client/server.

Values of time/date after reboot

[root@localhost ~]# date
Mon Sep  2 11:16:55 PM CDT 2024
[root@localhost ~]# timedatectl show
Timezone=CST6CDT
LocalRTC=no
CanNTP=yes
NTP=no
NTPSynchronized=no
TimeUSec=Mon 2024-09-02 23:17:22 CDT
RTCTimeUSec=Fri 2016-01-01 05:20:30 CST
[root@localhost ~]# timedatectl
               Local time: Mon 2024-09-02 23:17:28 CDT
           Universal time: Tue 2024-09-03 04:17:28 UTC
                 RTC time: Fri 2016-01-01 11:20:36
                Time zone: CST6CDT (CDT, -0500)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
[root@localhost ~]# cat /tmp/timeInfo
               Local time: Mon 2024-09-02 19:04:45 CDT
           Universal time: Tue 2024-09-03 00:04:45 UTC
                 RTC time: Fri 2016-01-01 07:07:52
                Time zone: CST6CDT (CDT, -0500)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
[root@localhost ~]# systemctl status chronyd
○ chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:chronyd(8)
             man:chrony.conf(5)

FrankCox · February 17, 2025, 6:42pm

What time does the system bios think it is?

hgupta · February 17, 2025, 6:50pm

System/BIOS time

[root@localhost ~]# hwclock --show
2016-01-01 06:49:51.993453-06:00

iwalker · February 17, 2025, 8:35pm

What is the content of /etc/adjtime ? From what I remember when I’ve had similar issues like this, I believe I just deleted that file, restarted and went from there. My machine is GMT+1, but my adjtime file has the below values:

# cat /etc/adjtime 
0.0 0 0.0
0
UTC

hgupta · February 18, 2025, 7:21am

Before reboot

# date
Tue Feb 18 12:46:04 PM IST 2025
# cat /etc/adjtime
0.0 0 0.0
0
UTC
# systemctl disable chronyd
# systemctl stop chronyd
# timedatectl set-timezone "CST6CDT"
# timedatectl set-time '2016-01-01 01:00'
# date
Fri Jan  1 01:00:00 AM CST 2016
# cat /etc/adjtime
0.0 0 0.0
0
UTC

after reboot:

[root@localhost ~]# date
Mon Sep  2 07:03:50 PM CDT 2024
[root@localhost ~]# cat /etc/adjtime
0.0 0 0.0
0
UTC
[root@localhost ~]#

iwalker · February 18, 2025, 8:21am

You can leave chrony enabled when configuring timezone, you don’t have to stop it. Please post the content of your /etc/chrony.conf perhaps you have something misconfigured here.

Perhaps also reboot and go into your BIOS to check what data/time has been set there and perhaps fix it there before allowing it to boot into Linux.

hgupta · February 18, 2025, 2:40pm

chrony.conf content:

[root@localhost ~]# cat /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (https://www.pool.ntp.org/join.html).

# Use NTP servers from DHCP.
sourcedir /run/chrony-dhcp

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable kernel synchronization of the real-time clock (RTC).
rtcsync

# Enable hardware timestamping on all interfaces that support it.
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

# Allow NTP client access from local network.
#allow 192.168.0.0/16

# Serve time even if not synchronized to a time source.
#local stratum 10

# Require authentication (nts or key option) for all NTP sources.
#authselectmode require

# Specify file containing keys for NTP authentication.
keyfile /etc/chrony.keys

# Save NTS keys and cookies.
ntsdumpdir /var/lib/chrony

# Insert/delete leap seconds by slewing instead of stepping.
#leapsecmode slew

# Get TAI-UTC offset and leap seconds from the system tz database.
leapsectz right/UTC

# Specify directory for log files.
logdir /var/log/chrony

# Select which information is logged.
#log measurements statistics tracking
server in.pool.ntp.org

hgupta · February 18, 2025, 2:47pm

@iwalker , Actually we have an application were we are using rocky as base OS.

in that application we have provided 2 options to user :

he can setup the NTP server and use current time based on timezone.
he can configure time, date and timezone as per his requirement by disabling the chronyd service.

Option 1 works as expected but option 2 fails.

I have tested same thing on rocky OS as well without using my application there also I observed same behavior.

iwalker · February 18, 2025, 3:20pm

And this goes back to what I hinted at before, all to do with the time/date not being synced to the hwclock and is why I asked you to go into the BIOS and check/configure it here as well.

I don’t have this problem when I check timedatectl with and without chrony running, so it means the problem is with your server/system clock.

If Option 1 works, then just leave it at that. I don’t see the need for Option 2 at this point, unless you are going to spend all the time debugging what I suggested relating to the BIOS, sync to hwclock and /etc/adjtime. There’s plenty of results in google about this situation, it’s not Rocky-specific. So any results relating to Linux will generally work similar to what I mentioned already.

As Option 1 works, then that is your solution. A server should always be running NTP of some sorts to stay in sync. So option 2 without NTP is not reliable even if you do manage to get it to sync with hwclock. Because later it will not be syncing with anything, and will drift and display the incorrect time.

hgupta · February 18, 2025, 3:57pm

Thanks @iwalker for your input, i will check with my team on it and get back to you.

Hedges · February 19, 2025, 1:58pm

Sorry if this is obvious to you, but I have seen this issue in my country a lot, where we only have one timezone and “option 2” is often requested when going international and suddenly some users request different timestamps using the web interface.
But the solution in this case actually is easy:

E.g. if I login to a US server, I might see this:

foo@barzoo: date
Mi 19. Feb 08:45:05 EST 2025

But as I do not live at “EST” and it is afternoon already here, he timestamps confuse me a lot. So I do

foo@barzoo: export TZ=GMT
foo@barzoo: date
Mi 19. Feb 13:45:20 GMT 2025

Problem solved,

We also configure our servers to have the RTC running in UTC and there are still people being confused by the RTC not showing the local time, but it never will …

Again, this might be trivial to you … so ignore me if it is …

jlehtone · February 19, 2025, 3:27pm

This. On isolated system one could set arbitrary date, like year (Kōki) 2685 today, and year 1666 tomorrow, but that is not very useful nor intuitive.

If you want to exchange data with other systems, then time should progress in your system predictably, in sync with other systems. That is what the various time services are made for.

When machine starts up, it does not have network connection and therefore it gets initial guess about time from the hwclock. Once time sync service starts, it corrects the time and maintains it to minimize drifting. At some point, at shutdown the latest, the hwclock is updated so that on next start it will be on the ballpark.

Ideally the hwclock is always in UTC time, so it does not have to jump due to changes of timezone, daylight saving time, etc.

hgupta · February 19, 2025, 7:31pm

@Hedges ,

Thanks for input, but in you case you are just change TZ but in my case customer can change any or all 3 things(date,time and TZ) incase of NTP disable.

hgupta · February 20, 2025, 3:43am

@jlehtone ,

Yes i agree with you that is there is no initial connection and NTP is disabled. system time should picked from hwclock.
but as per my observation my hwclock(BIOS) is set to same time/date which i specified manually. where as system time dirft by very large difference.
i also checked that /etct/adjtime is 0 and hwclock is in UTC as well.

what confuse me is that why there is difference in system time and hwclock?

hgupta · February 21, 2025, 10:27am

Hi @jlehtone @Hedges
to resolve above issue(option 2), I created the shell script, and made it part of init.d process.
the script checks if the system clock is synchronized using chronyc. If it is not, it updates the system clock from the hardware clock.
content of script is as below:

#!/bin/bash 
/usr/bin/chronyc tracking > /dev/null 2>&1

if [ $? -ne 0 ]; then

/usr/sbin/hwclock --hctosys

fi

Can you please share your view on it?
Does it look feasible or is there anything else I can add upon it?
please share your views.