Monitoring with Munin#

Why#

Instantaneous numbers (chronyc tracking) confirm the server works right now. But real validation of a Stratum 1 happens over time: does frequency drift follow a day/night cycle tied to temperature? Does the hardware timestamping rate degrade under load? Munin builds historical RRD graphs (day/week/month) that make these trends visible at a glance, impossible to get from a single instantaneous command.

Why Munin rather than Prometheus#

Prometheus is today’s most widespread monitoring tool, so the question deserves a reasoned answer. Four reasons tipped the balance toward Munin on this particular server:

  • The write profile. Munin stores RRD files: fixed size, written once every 5 minutes, with zero growth. Prometheus writes continuously (a time-series database with a WAL journal): exactly the write profile the Filesystem chapter works to eliminate in order to preserve the SD card.
  • No permanent daemon. Here, monitoring only exists once every 5 minutes, confined to core 0 at minimum priority (taskset -c 0 nice -n 19 ionice -c 3, Public telemetry chapter). Prometheus is a resident process consuming CPU and RAM around the clock, on a machine whose entire purpose is microsecond-level scheduling stability.
  • The attack surface. The Prometheus model relies on an exporter listening on an HTTP port, one more network service to harden and filter. Munin, configured as done here, produces static PNGs served by the already-hardened nginx: no additional exposed port.
  • The actual need. A single host, a handful of graphs, a 5-minute refresh aligned with the public page. No fleet, no required alerting, no ad hoc queries. Prometheus shines precisely where this project has no need.

In exchange, Munin offers neither fine resolution (5 minutes, versus a typical ~15 s scrape), nor a query language (PromQL), nor built-in alerting.

This remains a personal choice, consistent with this server’s constraints, not a general truth: a reader comfortable with Prometheus is entirely welcome to use it. The clean way would then be to host Prometheus on another machine (the administration network) and expose on the Pi only a minimal exporter restricted to that network: storage and computation leave the time server, only metric reads remain.

How#

Metrology plugins#

apt install -y munin munin-node

# Chrony plugin shipped with the package
ln -sf /usr/share/munin/plugins/chrony /etc/munin/plugins/chrony

# Community chrony_status plugin (munin-contrib)
curl -fsSL https://raw.githubusercontent.com/munin-monitoring/contrib/master/plugins/chrony/chrony_status \
    -o /usr/share/munin/plugins/chrony_status
chmod +x /usr/share/munin/plugins/chrony_status
ln -sf /usr/share/munin/plugins/chrony_status /etc/munin/plugins/chrony_status

Custom plugin: median-filtered PHC offset#

cat > /usr/share/munin/plugins/ptp_offset <<'EOF'
#!/bin/sh
# Offset between the PTP hardware clock (/dev/ptp0) and CLOCK_REALTIME

if [ "$1" = "config" ]; then
    echo "graph_title PHC to System Clock Offset"
    echo "graph_vlabel offset (ns)"
    echo "graph_category time"
    echo "offset.label PHC Offset"
    echo "offset.type GAUGE"
    echo "offset.draw LINE2"
    exit 0
fi

PHC_OFFSET=$(/usr/local/sbin/phc-offset-median /dev/ptp0 5)
[ -z "$PHC_OFFSET" ] && PHC_OFFSET="U"
echo "offset.value $PHC_OFFSET"
EOF
chmod +x /usr/share/munin/plugins/ptp_offset
ln -sf /usr/share/munin/plugins/ptp_offset /etc/munin/plugins/ptp_offset

mkdir -p /etc/munin/plugin-conf.d/
cat > /etc/munin/plugin-conf.d/ptp_offset.conf <<'EOF'
[ptp_offset]
user root
EOF

This plugin delegates to a shared script (phc-offset-median) that repeats the reading 5 times at real-time priority and keeps the median, see the PHC chapter for the full justification: a single reading is vulnerable to one-off scheduling spikes that would produce a misleading sawtooth graph.

Munin configuration#

cat > /etc/munin/munin.conf <<'EOF'
htmldir /var/time
graph_strategy cron
html_strategy disabled
graph_data_size custom 35d
timeout_fetch_all_nodes 240

[time.example.org]
    address 127.0.0.1
    use_node_name yes
EOF

graph_strategy cron regenerates PNG graphs on Munin’s cron cadence rather than on demand (CGI), consistent with the periodic-regeneration philosophy already seen in the Public telemetry chapter. html_strategy disabled outright disables Munin’s native HTML page generation: this project serves its own static page (previous chapter) and only consumes the PNGs, generating unused HTML pages on top would waste CPU and space on the /var/time tmpfs every cycle. timeout_fetch_all_nodes 240 (4 minutes) leaves a margin under Munin’s 5-minute cron cycle: if metric collection took longer than that, it would be cut off before overlapping the next cycle, instead of letting two runs pile up.

Pitfalls to avoid#

A plugin without the executable bit is silently ignored, no error, no log, just a missing graph. After creating or copying any Munin plugin, chmod +x is not optional. This is the single most common mistake when adding a custom or downloaded plugin.

  • Testing a plugin as the wrong user. munin-node runs as root by default on Debian and drops privileges to whatever user is specified in plugin-conf.d/ (often munin, sometimes root for plugins needing elevated privileges like ptp_offset). A manual test sudo -u munin /usr/share/munin/plugins/ptp_offset can fail even though the plugin works fine via munin-node, not a bug, just a different privilege context. For an accurate test, invoke the plugin under plain sudo (root), or explicitly check the user= set in its config.
  • Looking for munin-plugins-extra on a recent distribution. This package has been merged into munin-node on recent Debian releases, check with apt-cache search before assuming it’s missing.
  • Sizing graph_data_size without thinking through the actual retention needed. A single RRA (Round Robin Archive) at fixed resolution (here 5 minutes over 35 days) is enough for day/week/month views, but doesn’t cover a fine-grained yearly view, a deliberate trade-off rather than an oversight, consistent with the “volatile logs” philosophy of the rest of this project (Filesystem chapter).

Verify#

# Manual plugin execution, as munin-node would do it
sudo /usr/share/munin/plugins/ptp_offset
# Should return "offset.value <number>", not "offset.value U"

# Test the config the plugin declares
sudo /usr/share/munin/plugins/ptp_offset config

munin-run chrony
# Should return numeric values for every declared metric

# Manually generate graphs (instead of waiting for the cron)
munin-cron --force-root
ls -la /var/time/*.png | head
Backward fail2ban Public telemetry Forward
time.orlinum.fr : Documentation - © time.orlinum.fr, Documentation sous licence CC BY-SA 4.0