AT&T U-Verse and NTP – adufray.com

I recently upgraded my AT&T modem from the NVG 589 to the Pace 5268AC at the insistence of a technician out to repair my ONT. I had been having some packet loss issues and hoped the upgrade would help (it seemed to). However, I noticed immediately my NTP packets were being filtered. I thought this was odd, but found the following resources:

After studying all of these pages and looking at countless lines of tcpdump output, I confirmed a few things:

NTP traffic is indeed filtered both inbound and outbound
It is not an outright block, but a rate limit
The outbound filter applies only to traffic with source port 123/udp, an ephemeral port is fine

Sadly, the Network Time Foundation’s reference implementation of ntpd only permits using an ephemeral source port when using ntpdate and not from ntpd itself. Therefore, while behind an AT&T network, it seems not possible to run ntpd without some kind of middleware piece to mangle your UDP packets (e.g., iptables). This makes things very difficult in a modern home with all the following devices using NTP:

Network security cameras
iPhones/iPads/Macs
Media players, like Apple TVs
Wireless access points
“Smart” TVs

I was able to configure a local NTP server using OpenBSD’s OpenNTPd, which uses ephemeral source ports by default. Most of my devices I was able to reconfigure to use my local time server, however many devices do not expose that configuration to the end user - in particular iOS devices and Apple TVs. In fact, they were the noisiest devices on the network attempting to sync time. In order to resolve those devices, I had to add stub DNS zones to my local DNS. I created replacement zones for:

time.apple.com
time-ios.g.aaplimg.com

Once I pointed those hostnames to my local NTP server, everything seemed to resolve itself. Even systems that I cannot change from ntpd to OpenNTPd can now sync time using my local servers, which is pretty great.

I also looked at using Chrony instead of OpenNTPd, but after installation on my FreeBSD server I received this scary message:

Unfortunately, this software has shameful history of several vulnerabilities
previously discovered.  FreeBSD Project cannot guarantee that this spree had
come to an end.  Please type ``pkg delete chrony'' to deinstall the port
if tight security is a concern.

I liked the idea of using Chrony because my system monitoring tool of choice, Check_MK, already had a plugin to monitor Chrony, and as far as I can tell didn’t have one for OpenNTPd. On top of that, Chrony is now the default NTP server in Red Hat Enterprise Linux starting with version 7.

Since I trust OpenBSD’s security track record vs. a group I’ve never heard of and the most alarming message I’ve ever seen installing a port, I stuck with OpenNTPd. I was also able to make quick work of a Check_MK plugin, despite ntpctl’s absolutely terrible output. (Seriously, who designed that?) Here’s the plugin - you just need to add the sample output’s command either in your plugins directory or directly to check_mk_agent itself, prepending it with <<<openntpd>>> of course.

#!/usr/bin/python

# 2017 (c) adufray.com
# bsd license
# openntpd check_mk plugin to fit with ntp checks
# note: only supports servers/peers, not sensors

ntp_default_levels = (10, 200.0, 500.0) # stratum, ms offset

# Example output from agent:
# $ ntpctl -s a | paste - - | nl | egrep -E '^     1[[:space:]]|[*]' | cut -f 2- | sed -e 's/[[:space:]][[:space:]]*/ /g'
# 4/4 peers valid, clock synced, stratum 4
# 69.164.202.202 from pool us.pool.ntp.org * 1 10 3 2s 30s -0.053ms 8.154ms 1.675ms

def inventory_openntpd(info):
    if info[0]:
        return [(None, "ntp_default_levels")]

def check_openntpd(_no_item, params, info):
    if not info[0]:
        yield 2, "No status information, openntpd probably not running"
        return
    if "clock unsynced" in " ".join(info[0]):
        yield 2, "%s" % " ".join(info[0])
        return

    # Prepare parameters
    crit_stratum, warn, crit = params

    # Check offset and stratum, output a few info texsts
    offset = float(info[1][-3][0:-2])
    stratum = int(info[1][-6])

    # Check stratum
    infotext = "stratum %d" % stratum
    if stratum >= crit_stratum:
        yield 2, infotext + " (maximum allowed is %d)" % (crit_stratum - 1)
    else:
        yield 0, infotext

    # Check offset
    status = 0
    infotext = "offset %.4f ms" % offset
    if abs(offset) >= crit:
        status = 2
    elif abs(offset) >= warn:
        status = 1
    if status:
        infotext += " (levels at %.4f/%.4f ms)" % (warn, crit)
    yield status, infotext, [ ("offset", offset, warn, crit, 0, None) ]

    # Show additional information
    if info[1][1] == "from":
       yield 0, "reference: %s" % "/".join(info[1][0:4:3])
    else:
       yield 0, "reference: %s" % "/".join(info[1][0:2])

check_info["openntpd"] = {
   'check_function':          check_openntpd,
   'inventory_function':      inventory_openntpd,
   'service_description':     'NTP Time',
   'has_perfdata':            True,
   'group':                   'ntp_time',
}

I suppose this will be a solid enough solution until Poul-Henning Kamp’s masterpiece, Ntimed, is released!

ADUFRAY