Final verification and acceptance tests#

Why#

Every previous chapter has its own “Verify” section, targeting a single component. But a system made of twenty individually working parts doesn’t guarantee they work well together, and manually testing twenty points on every configuration change is slow and error-prone. The solution: a single acceptance script, run with one command, that audits the whole system and categorizes each check as OK / WARN / CRITICAL.

How#

Structural pattern#

#!/bin/bash
OK_COUNT=0; WARN_COUNT=0; CRIT_COUNT=0

ok()       { printf "  [ OK ]       %s\n" "$1"; OK_COUNT=$((OK_COUNT+1)); }
warn()     { printf "  [ WARN ]     %s (%s)\n" "$1" "$2"; WARN_COUNT=$((WARN_COUNT+1)); }
critical() { printf "  [ CRITICAL ] %s (%s)\n" "$1" "$2"; CRIT_COUNT=$((CRIT_COUNT+1)); }

Each check follows this pattern: measure a value, compare it to a threshold, and call ok/warn/critical based on the result. For example, for the RMS offset (Chrony chapter):

RMS=$(chronyc tracking 2>/dev/null | awk '/^RMS offset/ {print $4}')
if [ "$(echo "$RMS < 0.000001" | bc -l)" = "1" ]; then
    ok "RMS offset = ${RMS}s (< 1µs)"
elif [ "$(echo "$RMS < 0.00001" | bc -l)" = "1" ]; then
    warn "RMS offset = ${RMS}s" "target < 1µs"
else
    critical "RMS offset = ${RMS}s" "significant drift"
fi

Check categories covered#

A complete acceptance script for this kind of server typically covers:

Assembling a minimal script#

The fragments above combine into a real, short but already useful script, to be extended chapter by chapter with its own thresholds:

cat > acceptance.sh <<'EOF'
#!/bin/bash
OK_COUNT=0; WARN_COUNT=0; CRIT_COUNT=0

ok()       { printf "  [ OK ]       %s\n" "$1"; OK_COUNT=$((OK_COUNT+1)); }
warn()     { printf "  [ WARN ]     %s (%s)\n" "$1" "$2"; WARN_COUNT=$((WARN_COUNT+1)); }
critical() { printf "  [ CRITICAL ] %s (%s)\n" "$1" "$2"; CRIT_COUNT=$((CRIT_COUNT+1)); }

# Critical services
for svc in chrony nginx fail2ban auditd; do
    if systemctl is-active --quiet "$svc"; then
        ok "$svc active"
    else
        critical "$svc inactive" "systemctl status $svc"
    fi
done

# RMS offset (Chrony chapter)
RMS=$(chronyc tracking 2>/dev/null | awk '/^RMS offset/ {print $4}')
if [ -z "$RMS" ]; then
    critical "RMS offset unreadable" "is chronyc reachable?"
elif [ "$(echo "$RMS < 0.000001" | bc -l)" = "1" ]; then
    ok "RMS offset = ${RMS}s (< 1µs)"
elif [ "$(echo "$RMS < 0.00001" | bc -l)" = "1" ]; then
    warn "RMS offset = ${RMS}s" "target < 1µs"
else
    critical "RMS offset = ${RMS}s" "significant drift"
fi

# PPS/GNSS
if [ -e /dev/pps0 ]; then
    ok "/dev/pps0 present"
else
    critical "/dev/pps0 missing" "is pps-gpio module loaded?"
fi

# System: no active swap
if [ "$(swapon --show | wc -l)" -eq 0 ]; then
    ok "no active swap"
else
    warn "active swap detected" "see Filesystem chapter"
fi

echo
echo "ACCEPTANCE SUMMARY"
printf "  OK       : %3d\n" "$OK_COUNT"
printf "  WARN     : %3d\n" "$WARN_COUNT"
printf "  CRITICAL : %3d\n" "$CRIT_COUNT"
echo
if [ "$CRIT_COUNT" -gt 0 ]; then
    echo ">>> FAILURE - $CRIT_COUNT critical issue(s)"
elif [ "$WARN_COUNT" -gt 0 ]; then
    echo ">>> PARTIAL SUCCESS - $WARN_COUNT warning(s)"
    echo ">>> Monitoring recommended"
else
    echo ">>> FULL SUCCESS"
fi
EOF
chmod +x acceptance.sh

This version fits in about twenty checks to stay readable in this guide. This project’s actual script runs closer to a hundred, one per row of the table above, added incrementally chapter by chapter throughout the install.

Pitfalls to avoid#

A test that fails systematically, regardless of the system’s actual state, is worse than no test. It creates a false habit of ignoring warnings (“that WARN is normal, it’s always there”). This happened for real on this project with a TLS check: the test looked for the exact string Protocol : TLSv1.3 in openssl s_client’s output to confirm the negotiated version. A minor OpenSSL version change was enough to shift that exact spacing (Protocol : TLSv1.3, two spaces instead of one): the test failed permanently, even though TLS 1.3 was indeed negotiated on every connection. When a test fails consistently on a system that otherwise looks healthy, the first question isn’t “what’s broken on the server side?” but “does the test itself measure what it claims to measure?”

• WARN/CRITICAL thresholds copied without adapting to the actual context. A 60°C temperature threshold might be perfectly normal in a temperate climate and systematically exceeded in a hotter one, without reflecting a real problem. Calibrate thresholds against actual observed system behavior, not a generic value found elsewhere.
• Letting permanent WARNs settle in as background noise. On a system with dozens of checks, some findings stem from a real, known, accepted limitation (a documented hardware bug, a third-party tool’s false positive). Leaving them as WARN on every run would contradict the principle above: that is exactly how the habit of ignoring warnings is born. The right treatment is to turn them into an explicit exception inside the test itself: the known case reports OK, with its justification in the message, as this project’s test suite does for hardware TX timestamping (“macb Pi 5 driver bug, expected”, PHC chapter). WARN then stays reserved for transitional states meant to disappear (a feature not yet deployed), and the normal output tends toward zero warnings: any WARN that shows up is a fresh signal worth attention, never ambient noise.

Verify#

./acceptance.sh

Expected output at the end of the script:

ACCEPTANCE SUMMARY
  OK       :  83
  WARN     :   0
  CRITICAL :   0

>>> FULL SUCCESS

Run this script after every configuration change, not just at initial install: it’s the best safety net against silent regressions. 0 CRITICAL is the minimum threshold for production readiness; every WARN should be explained (see above), not simply ignored.