Verification: chrony’s state in one command#

Why#

The non-regression battery (Final verification chapter) validates a change; the Munin graphs (Munin chapter) show trends over time. In between, day-to-day operation needs a simple gesture: a complete snapshot of the service’s state, in a single command, readable in a few seconds.

chronyc can chain several reports in a single invocation (the -m option), and four of them are enough to cover the whole service: clock discipline (tracking), served load (serverstats), source states (sources -v) and their statistics (sourcestats -v).

How#

chronyc -m tracking serverstats 'sources -v' 'sourcestats -v'

-m treats each argument as a separate command; the quotes protect the ones that carry an option ('sources -v').

Real output from this server:

Reference ID    : 50505330 (PPS0)
Stratum         : 1
Ref time (UTC)  : Sun Jul 19 23:58:28 2026
System time     : 0.000000117 seconds fast of NTP time
Last offset     : +0.000000430 seconds
RMS offset      : 0.000000751 seconds
Frequency       : 0.015 ppm fast
Residual freq   : +0.016 ppm
Skew            : 0.151 ppm
Root delay      : 0.000000001 seconds
Root dispersion : 0.000005789 seconds
Update interval : 4.0 seconds
Leap status     : Normal
NTP packets received       : 27085049
NTP packets dropped        : 1954441
Command packets received   : 1185
Command packets dropped    : 0
Client log records dropped : 1731901
NTS-KE connections accepted: 70
NTS-KE connections dropped : 0
Authenticated NTP packets  : 314
Interleaved NTP packets    : 443
NTP timestamps held        : 1744
NTP timestamp span         : 16025
NTP daemon RX timestamps   : 0
NTP daemon TX timestamps   : 25129144
NTP kernel RX timestamps   : 3493864
NTP kernel TX timestamps   : 59
NTP hardware RX timestamps : 21635723
NTP hardware TX timestamps : 384

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#* PPS0                          0   2   377     3   +148ns[ +553ns] +/-  696ns
#? GPS                           0   4   377    19  -2621us[-2621us] +/-  248us
^- 145.238.80.80                 1   9   377    81   -553us[ -552us] +/- 1605us
^- 2001:638:610:cecf::7b         1   9   377   135   -154us[ -153us] +/- 9813us
^- 139.143.5.30                  1   8   377   247   -535us[ -537us] +/- 5244us
^- 193.11.166.8                  1  10   377   531   -533us[ -535us] +/-   18ms
^- 129.6.15.25                   1   9   377   129   -529us[ -529us] +/-   44ms
^- 185.142.225.68                1  10   377  1101   +969us[ +923us] +/- 8319us
                             .- Number of sample points in measurement set.
                            /    .- Number of residual runs with same sign.
                           |    /    .- Length of measurement set (time).
                           |   |    /      .- Est. clock freq error (ppm).
                           |   |   |      /           .- Est. error in freq.
                           |   |   |     |           /         .- Est. offset.
                           |   |   |     |          |          |   On the -.
                           |   |   |     |          |          |   samples. \
                           |   |   |     |          |          |             |
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
PPS0                        6   3    22     +0.016      0.146    +40ns   277ns
GPS                        45  24   703     +1.883      1.989  -3307us   720us
145.238.80.80               8   5   60m     -0.019      0.141    -58us    75us
2001:638:610:cecf::7b       8   5   60m     -0.001      0.072   -164us    40us
139.143.5.30                7   4   30m     -0.013      0.139    -71us    34us
193.11.166.8                7   5  103m     +0.014      0.068   -483us    56us
129.6.15.25                12   7   68m     -0.028      0.051   -205us    48us
185.142.225.68              9   5  129m     +0.068      0.236  +1169us   344us

Block 1: tracking, the clock discipline#

FieldReading
Reference ID (PPS0)the active reference really is the PPS, not a fallback source (Chrony chapter)
Stratum : 1the server announces the level directly attached to a physical reference (Overview chapter)
Ref timethe instant of the last applied correction
System timethe instantaneous gap between the system clock and NTP (here ≈ 0.1 µs)
Last offsetthe last correction applied by the loop
RMS offsetthe precision metric: standard deviation of the error, here ≈ 0.75 µs, around the microsecond this project targets
Frequencythe total correction applied to the crystal (0.015 ppm); such a small value means the thermal compensation already absorbs the predictable part of the drift (Chrony chapter, tempcomp)
Residual freqthe frequency error left after all corrections: close to zero on a healthy system
Skewthe uncertainty on the estimated frequency; 0.151 ppm here, the value settles below ~0.05 ppm in steady state (it transiently rises after a restart)
Root delay / Root dispersiondelay and uncertainty accumulated toward the reference: near zero, the reference is local
Update interval : 4.0 sthe correction cadence, consistent with the PPS refclock’s poll 2 (2² = 4 s)
Leap status : Normalno pending leap second (Leap seconds chapter)

Block 2: serverstats, the service delivered#

FieldReading
NTP packets received / dropped27 million requests served, ~7% rejected: the application-level ratelimit at work (Chrony chapter)
Command packetslocal chronyc requests, none rejected
Client log records droppedthe client-tracking table (clientloglimit) saturates and recycles entries: expected on a busy public server
NTS-KE connections accepted70 successful NTS negotiations (NTS chapter); dropped 0: the ntsratelimit limit never had to intervene
Authenticated NTP packetsthe NTP requests actually protected by NTS
Interleaved NTP packets, timestamps held/spanserver-side support of interleaved mode (xleave): those timestamps are retained (~1,700, over ~4.5 h) for clients that request it
NTP daemon TX timestamps≈ 25 million: served responses are timestamped in user space, because hardware TX timestamping is blocked by the macb driver bug (kernel TX 59, hardware TX 384, PHC chapter)
NTP kernel/hardware RX timestamps86% of processed requests get a hardware timestamp (21.6 M out of 25.1 M), the figure expected with phc2sys’s -R 1 setting (PHC chapter); the rest falls back to the kernel

Block 3: sources -v, the source states#

The detailed reading grid (legend, #*, #?, ^- symbols, octal Reach) is in the Chrony chapter. The three checks for this block: the PPS is selected (#*, ±696 ns estimated uncertainty), the GPS in #? with a few milliseconds of offset is normal (second-labeling role, noselect), and all six network sources show Reach 377 (8 answers out of 8).

Block 4: sourcestats -v, per-source statistics#

ColumnMeaning
NPnumber of measurement points kept in the statistical window
NRnumber of runs of residuals with the same sign: well below NP/2, it signals correlated residuals (regression still converging)
Spanduration covered by the window (60m = 60 minutes)
Frequency / Freq Skewestimated drift against this source, and the uncertainty of that estimate (ppm)
Offsetestimated offset against this source
Std Devdispersion of the measurements

Three readings on the example: the PPS shows a dispersion of a few hundred nanoseconds over its short window; the GPS is noisy (sub-millisecond dispersion) with zero consequence (noselect); the network sources spread from a few tens to a few hundred µs of apparent offset, the order of magnitude of network asymmetry, not of the clocks (Performance comparison chapter).

The ten-second reading#

  1. Reference ID = PPS0 and #* on the PPS line: the right source is in charge.
  2. RMS offset around a microsecond or better.
  3. Residual freq close to zero.
  4. Reach 377 on every source.
  5. Network sources mutually consistent within a few hundred µs.
  6. NTS-KE accepted growing, dropped at zero.

Pitfalls to avoid#

  • Judging the server on a snapshot. A single reading is neither an average nor a trend: a high Skew after a restart, an old LastRx at Poll 10, a momentarily large network offset are normal transient states. Duration is judged on Munin (Munin chapter), compliance with the battery of the Final verification chapter.
  • Being alarmed by the GPS in #? or its millisecond offset. That is nominal operation: this source only labels the second (noselect, Chrony chapter).
  • Comparing network sources’ offsets to the local precision. The displayed hundreds of µs mostly measure path asymmetry (Performance comparison chapter), not the local clock.
  • Confusing Frequency, Residual freq and Skew. Applied correction, remaining error, estimation uncertainty: three different quantities. A small Frequency with tempcomp active is a sign of good health, not an anomaly.
  • Forgetting the Unix socket under the hardened service. Without -h /run/chrony/chronyd.sock, chronyc can fail without a clear message (Chrony chapter, pitfalls).

Verify#

This chapter is the verification: the single command above, run on every visit. Its two complements: the Munin graphs for the trend (24 h minimum), and acceptance.sh for non-regression after a change (Final verification chapter).

📚 Going further