ntpd goes maintenance

ntpd goes maintenance

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 1556 bytes --]

Hi, I have a virtualized XStreamOS system that started giving problems lately.
The configured ntp service goes in maintenance during the night, clock goes minutes behind, CIFS fails in the morning.
Here is the error I find:

Sep 12 23:24:35 xstorage ntpd[3819]: [ID 702911 ntp.error] Clock offset exceeds panic threshold.
Sep 12 23:24:35 xstorage ntpd[3819]: [ID 702911 ntp.error] svc:/network/ntp:default placed into maintenance. Set system clock by hand before clearing.
Sep 12 23:24:35 xstorage svc.startd[9]: [ID 748625 daemon.error] network/ntp:default transitioned to maintenance by request (see 'svcs -xv' for details)
Sep 12 23:26:01 xstorage svc.startd[9]: [ID 122153 daemon.warning] svc:/network/ntp:default: Method or service exit timed out.  Killing contract 112.
Sep 12 23:26:01 xstorage svc.startd[9]: [ID 748625 daemon.error] network/ntp:default failed: transitioned to maintenance (see 'svcs -xv' for details)
 
Here is the ntp.conf:

server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst
driftfile /var/ntp/ntp.drift
statsdir /var/ntp/ntpstats/
filegen peerstats file peerstats type day enable
filegen loopstats file loopstats type day enable
 
I used to have the AD PDC as server, but I changed it because I was not sure its time server was correctly running
 
Any idea?

Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 


[-- Attachment #1.2: Type: text/html, Size: 3388 bytes --]

Re: [developer] ntpd goes maintenance

[Marcel Telka]

On Wed, Sep 13, 2023 at 09:17:31AM +0200, Gabriele Bulfon via illumos-developer wrote:
> Any idea?

It is very likely this issue:
https://bugs.ntp.org/show_bug.cgi?id=3552

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] ntpd goes maintenance

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 1122 bytes --]

wow....I'll need to update that system...don't think there is any workaround...
 
Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 




----------------------------------------------------------------------------------

Da: Marcel Telka <marcel@telka.sk>
A: illumos-developer <developer@lists.illumos.org>
Data: 13 settembre 2023 9.20.53 CEST
Oggetto: Re: [developer] ntpd goes maintenance

On Wed, Sep 13, 2023 at 09:17:31AM +0200, Gabriele Bulfon via illumos-developer wrote:> Any idea?It is very likely this issue:https://bugs.ntp.org/show_bug.cgi?id=3552-- +-------------------------------------------+| Marcel Telka   e-mail:   marcel@telka.sk  ||                homepage: http://telka.sk/ |+-------------------------------------------+------------------------------------------illumos: illumos-developerPermalink: https://illumos.topicbox.com/groups/developer/Ta16f432d35bc27f5-Mc72403b491c147bd7babd30bDelivery options: https://illumos.topicbox.com/groups/developer/subscription

[-- Attachment #1.2: Type: text/html, Size: 2221 bytes --]

Re: [developer] ntpd goes maintenance

[Marcel Telka]

On Wed, Sep 13, 2023 at 09:55:12AM +0200, Gabriele Bulfon via illumos-developer wrote:
> wow....I'll need to update that system...don't think there is any workaround...

There isn't.  Upgrade won't help.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] ntpd goes maintenance

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 1306 bytes --]

Ohps I didn't notice the bug was so recent...so how do I keep that system in sync with the time of the PDC?
Insanely this never happened for more than a year, and started doing this after a recent reboot of the entire Proxmox VE.
 
Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 




----------------------------------------------------------------------------------

Da: Marcel Telka <marcel@telka.sk>
A: illumos-developer <developer@lists.illumos.org>
Data: 13 settembre 2023 9.56.27 CEST
Oggetto: Re: [developer] ntpd goes maintenance

On Wed, Sep 13, 2023 at 09:55:12AM +0200, Gabriele Bulfon via illumos-developer wrote:> wow....I'll need to update that system...don't think there is any workaround...There isn't.  Upgrade won't help.-- +-------------------------------------------+| Marcel Telka   e-mail:   marcel@telka.sk  ||                homepage: http://telka.sk/ |+-------------------------------------------+------------------------------------------illumos: illumos-developerPermalink: https://illumos.topicbox.com/groups/developer/Ta16f432d35bc27f5-M962de61957a3ba5c66f61bc7Delivery options: https://illumos.topicbox.com/groups/developer/subscription

[-- Attachment #1.2: Type: text/html, Size: 2533 bytes --]

Re: [developer] ntpd goes maintenance

[Marcel Telka]

On Wed, Sep 13, 2023 at 10:01:13AM +0200, Gabriele Bulfon via illumos-developer wrote:
> Ohps I didn't notice the bug was so recent...so how do I keep that system in sync with the time of the PDC?

Try to help with root causing and/or fixing the bug? :-))

> Insanely this never happened for more than a year, and started doing this after a recent reboot of the entire Proxmox VE.

Yes, I see the same issue too.  Usually few ntpd restarts make it to run properly.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] ntpd goes maintenance

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 1546 bytes --]

I'd love to help, but I'm not that much into that type of code...
Though reading the bug link, has anyone already tried disabling the optimization flag on the ntpd tree and rebuilt illumos gate with this change?
 
Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 




----------------------------------------------------------------------------------

Da: Marcel Telka <marcel@telka.sk>
A: illumos-developer <developer@lists.illumos.org>
Data: 13 settembre 2023 10.15.10 CEST
Oggetto: Re: [developer] ntpd goes maintenance

On Wed, Sep 13, 2023 at 10:01:13AM +0200, Gabriele Bulfon via illumos-developer wrote:> Ohps I didn't notice the bug was so recent...so how do I keep that system in sync with the time of the PDC?Try to help with root causing and/or fixing the bug? :-))> Insanely this never happened for more than a year, and started doing this after a recent reboot of the entire Proxmox VE.Yes, I see the same issue too.  Usually few ntpd restarts make it to run properly.-- +-------------------------------------------+| Marcel Telka   e-mail:   marcel@telka.sk  ||                homepage: http://telka.sk/ |+-------------------------------------------+------------------------------------------illumos: illumos-developerPermalink: https://illumos.topicbox.com/groups/developer/Ta16f432d35bc27f5-Me4529686fa8e1aa783e27941Delivery options: https://illumos.topicbox.com/groups/developer/subscription

[-- Attachment #1.2: Type: text/html, Size: 2656 bytes --]

Re: [developer] ntpd goes maintenance

[Jason King]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

What might be useful is to build it with CFLAGS += -g -msave-args (I’ve not built ntp anytime this millennium so can’t say offhand the exact incantation to get those added to its build) and running ctfconvert on the resulting binary and library. That’d make the stack traces a bit more useful (as well as allow viewing of any structs).


From: Gabriele Bulfon via illumos-developer <developer@lists.illumos.org>
Date: Wednesday, September 13, 2023 at 3:31 AM
To: illumos-developer <developer@lists.illumos.org>
Subject: Re: [developer] ntpd goes maintenance
I'd love to help, but I'm not that much into that type of code...
Though reading the bug link, has anyone already tried disabling the optimization flag on the ntpd tree and rebuilt illumos gate with this change?

Gabriele


Sonicle S.r.l. : http://www.sonicle.com<https://www.sonicle.com/>
Music: http://www.gabrielebulfon.com<http://www.gabrielebulfon.com/>
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets




----------------------------------------------------------------------------------

Da: Marcel Telka <marcel@telka.sk>
A: illumos-developer <developer@lists.illumos.org>
Data: 13 settembre 2023 10.15.10 CEST
Oggetto: Re: [developer] ntpd goes maintenance

On Wed, Sep 13, 2023 at 10:01:13AM +0200, Gabriele Bulfon via illumos-developer wrote:

> Ohps I didn't notice the bug was so recent...so how do I keep that system in sync with the time of the PDC?



Try to help with root causing and/or fixing the bug? :-))



> Insanely this never happened for more than a year, and started doing this after a recent reboot of the entire Proxmox VE.



Yes, I see the same issue too.  Usually few ntpd restarts make it to run properly.



--

+-------------------------------------------+

| Marcel Telka   e-mail:   marcel@telka.sk  |

|                homepage: http://telka.sk/ |

+-------------------------------------------+



------------------------------------------

illumos: illumos-developer

Permalink: https://illumos.topicbox.com/groups/developer/Ta16f432d35bc27f5-Me4529686fa8e1aa783e27941

Delivery options: https://illumos.topicbox.com/groups/developer/subscription
illumos<https://illumos.topicbox.com/latest> / illumos-developer / see discussions<https://illumos.topicbox.com/groups/developer> + participants<https://illumos.topicbox.com/groups/developer/members> + delivery options<https://illumos.topicbox.com/groups/developer/subscription> Permalink<https://illumos.topicbox.com/groups/developer/Ta16f432d35bc27f5-Mfdc7e6e925c3c48bfcc93df1>

[-- Attachment #2: Type: text/html, Size: 7675 bytes --]

Re: [developer] ntpd goes maintenance

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 5168 bytes --]

Stumbled upon this again on a sparc T4 with last recent illumos for sparc.
Enabled debug in svc ntp and this is the last lines of /var/ntp/ntp.debug :

Finding interface for addr 212.237.55.238 in list of addresses
findlocalinterface: kernel maps 212.237.55.238 to 192.168.122.8
Searching for addr 192.168.122.8 in list of addresses - FOUND
Found interface #4 lanaggr0 for address 212.237.55.238
peer_refresh_interface: 192.168.122.8->212.237.55.238 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000: new interface: fd=24, bfd=-1, name=lanaggr0, flags=0x19, scope=0, , sin=192.168.122.8, bcast=192.168.122.255,, mask=255.255.255.0
Finding interface for addr 217.61.62.224 in list of addresses
findlocalinterface: kernel maps 217.61.62.224 to 192.168.122.8
Searching for addr 192.168.122.8 in list of addresses - FOUND
Found interface #4 lanaggr0 for address 217.61.62.224
peer_refresh_interface: 192.168.122.8->217.61.62.224 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000: new interface: fd=24, bfd=-1, name=lanaggr0, flags=0x19, scope=0, , sin=192.168.122.8, bcast=192.168.122.255,, mask=255.255.255.0
Finding interface for addr 188.213.165.209 in list of addresses
findlocalinterface: kernel maps 188.213.165.209 to 192.168.122.8
Searching for addr 192.168.122.8 in list of addresses - FOUND
Found interface #4 lanaggr0 for address 188.213.165.209
peer_refresh_interface: 192.168.122.8->188.213.165.209 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000: new interface: fd=24, bfd=-1, name=lanaggr0, flags=0x19, scope=0, , sin=192.168.122.8, bcast=192.168.122.255,, mask=255.255.255.0
Finding interface for addr 185.19.184.35 in list of addresses
findlocalinterface: kernel maps 185.19.184.35 to 192.168.122.8
Searching for addr 192.168.122.8 in list of addresses - FOUND
Found interface #4 lanaggr0 for address 185.19.184.35
peer_refresh_interface: 192.168.122.8->185.19.184.35 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000: new interface: fd=24, bfd=-1, name=lanaggr0, flags=0x19, scope=0, , sin=192.168.122.8, bcast=192.168.122.255,, mask=255.255.255.0
read_network_packet: fd=24 length 48 from 188.213.165.209
addto_syslog: ntpd exiting on signal 10
finish(SIGBUS)
filegen_unregister(peerstats)
filegen_unregister(loopstats)
filegen_unregister(clockstats)
filegen_unregister(rawstats)
filegen_unregister(sysstats)
filegen_unregister(protostats)
filegen_unregister(cryptostats)
filegen_unregister(timingstats)


Don't know if t his is the same bug,

Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 

 


Da: Jason King <jason.brian.king@gmail.com>
A: illumos-developer <developer@lists.illumos.org>
Data: 13 settembre 2023 15.44.23 CEST
Oggetto: Re: [developer] ntpd goes maintenance




What might be useful is to build it with CFLAGS += -g -msave-args (I’ve not built ntp anytime this millennium so can’t say offhand the exact incantation to get those added to its build) and running ctfconvert on the resulting binary and library. That’d make the stack traces a bit more useful (as well as allow viewing of any structs).
 
 
From: Gabriele Bulfon via illumos-developer <developer@lists.illumos.org>
Date: Wednesday, September 13, 2023 at 3:31 AM
To: illumos-developer <developer@lists.illumos.org>
Subject: Re: [developer] ntpd goes maintenance

I'd love to help, but I'm not that much into that type of code...
Though reading the bug link, has anyone already tried disabling the optimization flag on the ntpd tree and rebuilt illumos gate with this change?

 

Gabriele

 

 

Sonicle S.r.l. : http://www.sonicle.com

Music: http://www.gabrielebulfon.com

eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets

 





----------------------------------------------------------------------------------

Da: Marcel Telka <marcel@telka.sk>
A: illumos-developer <developer@lists.illumos.org>
Data: 13 settembre 2023 10.15.10 CEST
Oggetto: Re: [developer] ntpd goes maintenanceOn Wed, Sep 13, 2023 at 10:01:13AM +0200, Gabriele Bulfon via illumos-developer wrote:> Ohps I didn't notice the bug was so recent...so how do I keep that system in sync with the time of the PDC? Try to help with root causing and/or fixing the bug? :-)) > Insanely this never happened for more than a year, and started doing this after a recent reboot of the entire Proxmox VE. Yes, I see the same issue too.  Usually few ntpd restarts make it to run properly. -- +-------------------------------------------+| Marcel Telka   e-mail:   marcel@telka.sk  ||                homepage: http://telka.sk/ |+-------------------------------------------+ ------------------------------------------illumos: illumos-developerPermalink: https://illumos.topicbox.com/groups/developer/Ta16f432d35bc27f5-Me4529686fa8e1aa783e27941Delivery options: https://illumos.topicbox.com/groups/developer/subscription


illumos / illumos-developer / see discussions + participants + delivery options Permalink

[-- Attachment #1.2: Type: text/html, Size: 12969 bytes --]