[sf-lug] "resolved": Re: Comcast Business DNS problems - yet again ([ns1.]linuxmafia.com 96.95.217.99 ...)

[Michael Paoli]

And, it appears at least for the moment, the issue is "resolved"
by 2023-09-24 10:56AM US/Pacific PST8PDT -07:00.
Though persistence of that resolution may still be an issue.

Some relevant bits:

---------- Forwarded message ---------
From: Rick Moen <rick at linuxmafia.com>
Date: Sun, Sep 24, 2023 at 10:56 AM

Coincidentally, I got woken up by a Comcast middle-grade service rep. in
Denver, who tried a few things culminating with removing the
SecurityEdge service from our account.  In quick testing, I am
reasonably sure this has fixed (for now) the problem, and all I have to
do now is (according to the NANOG worthy) repeat this bushwah ad
infinitum every time Comcast decides to roll out a firmware update.

I am _certain_ neither Cheryl nor I ever elected to add the SecurityEdge
product to our account.
https://business.comcast.com/enterprise/products-services/cybersecurity-services/comcast-business-securityedge

linuxmafia:~# dig sflug.com axfr @96.86.170.229 | wc -l
38
linuxmafia:~# traceroute -nTp 53 96.86.170.229
traceroute to 96.86.170.229 (96.86.170.229), 30 hops max, 40 byte
packets
 1  96.95.217.102  2.658 ms  3.009 ms  2.752 ms
 2  100.92.140.67  14.323 ms  19.645 ms 100.92.140.66  19.094 ms
 3  96.216.9.185  12.476 ms 96.216.9.177  12.033 ms 96.216.9.185  11.143
ms
 4  68.85.154.117  10.678 ms 68.85.154.113  10.206 ms  9.875 ms
 5  96.108.99.245  11.514 ms  16.932 ms 96.108.99.249  21.265 ms
 6  68.86.143.89  15.108 ms  15.630 ms  14.518 ms
 7  162.151.87.226  14.724 ms 162.151.86.58  13.462 ms  17.528 ms
 8  162.151.79.134  19.282 ms 162.151.78.186  16.072 ms  15.112 ms
 9  68.85.103.154  16.792 ms 68.85.191.206  15.629 ms  15.249 ms
10  73.189.65.18  28.179 ms  22.635 ms  25.685 ms
11  96.86.170.229  25.125 ms  30.377 ms  29.757 ms
linuxmafia:~#

linuxmafia:~# dig @96.86.170.225 +noall +answer +nottl comcast.com. A
;; connection timed out; no servers could be reached
linuxmafia:~#

Querying a NON-nameserver now (correctly) fails.

https://mailman.nanog.org/pipermail/nanog/2023-September/223337.html

On Fri, Sep 22, 2023 at 8:49 PM Michael Paoli
<michael.paoli at cal.berkeley.edu> wrote:
>
> Alas, Comcast Business is at it yet again.
> They've again broken DNS, notably
> at their gateway device, between most notably
> [ns1.]linuxmafia.com 96.95.217.99
> and The Internet, when and where
> [ns1.]linuxmafia.com 96.95.217.99
> is client, and any Internet IP addresses beyond that gateway device, are
> server, e.g. when
> [ns1.]linuxmafia.com 96.95.217.99
> functions as secondary and primary is elsewhere on
> The Internet.  The DNS problem is much more pervasive than that, with
> that gateway device, but the above is most notable impact,
> particularly for those beyond that local subnet,
> e.g. The rest of The Internet using
> [ns1.]linuxmafia.com 96.95.217.99
> as authoritative nameserver for many DNS zones.
> Notably it can't get its updates for zones it's secondary for,
> due to Comcast reintroducing apparently same problem they introduced a
> while ago.  And this was after Comcast Business not that long ago
> replaced the device to "fix" the issue ... and now the issue has
> returned yet again.
>
> Anyway, Rick is again working with Comcast Business to get them to fix
> their DNS mess up yet again.
>
> Meantime, I'm going to reimplement some workarounds where feasible,
> notably for zones where I also have DNS admin access to the primary
> servers.  Alas, had that in place until fairly recently, took it out
> after Comcast Business fixed the earlier issue.  Looks like time to
> put it back again.  I and I think this time around I probably leave
> it in place for 90+ days after Comcast Business "fixes" their issue
> ... in case they break it yet again not too long after "fixing" it.
> Or maybe I do up to 6 months or a year.  Not exactly gettin' the warm
> 'n fuzzy feelings from Comcast Business.
>
> And for the curious, on Comcast, might want to see if your DNS actually
> truly works, or if they subvert your DNS traffic too, and don't let you
> actually get that directly from The Internet.  And yes, there may be
> workarounds, oh, ... like run DNS on a different port, or (ugh) do DNS
> over TLS or HTTPS).
>
> Anyway, some technical details on the breakage (and thus far looks to be
> exactly same issue as before):
>
> On Thu, Sep 21, 2023 at 11:05 PM Michael Paoli
> <michael.paoli at cal.berkeley.edu> wrote:
> > Oh dear ...
> > $ hostname && ip -4 a s | fgrep inet | fgrep -v 127.
> > linuxmafia.com
> >     inet 96.95.217.99/29 brd 96.95.217.103 scope global eth2
> > $ hostname && ip -4 a s | fgrep inet | fgrep -v 127.
> > linuxmafia.com
> >     inet 96.95.217.99/29 brd 96.95.217.103 scope global eth2
> > $ (set -x; dig @$(dig +short com. NS | head -n 1) +noall +authority
> > sflug.com. NS)
> > ++ dig +short com. NS
> > ++ head -n 1
> > + dig @b.gtld-servers.net. +noall +authority sflug.com. NS
> > $ nc -vz b.gtld-servers.net. 53
> > b.gtld-servers.net [192.33.14.30] 53 (domain) open
> > $
> > # hostname && ip -4 a s | fgrep inet | fgrep -v 127.
> > linuxmafia.com
> >     inet 96.95.217.99/29 brd 96.95.217.103 scope global eth2
> > # traceroute -nTp 53 192.33.14.30
> > traceroute to 192.33.14.30 (192.33.14.30), 30 hops max, 40 byte packets
> >  1  192.33.14.30  2.795 ms  3.157 ms  3.672 ms
> > #
>
> Note in the above that in reality 192.33.14.30
> is absolutely not 1 hop away.
>
> And in the below, take note that at
> present here's absolutely nothing
> on or currently using 96.86.170.228:
>
> > # traceroute -nTp 53 96.86.170.228
> > traceroute to 96.86.170.228 (96.86.170.228), 30 hops max, 40 byte packets
> >  1  96.86.170.228  1.916 ms  1.615 ms  1.230 ms
> > # dig @96.86.170.228 +short www.google.com. A
> > 142.251.46.228
> > #
>
> > On Thu, Sep 21, 2023 at 10:44 PM Michael Paoli
> > <michael.paoli at cal.berkeley.edu> wrote:
> > >
> > > Okay ... let me have a look and see what's up ....
> > >
> > > On Thu, Sep 21, 2023 at 6:59 PM Rick Moen <rick at linuxmafia.com> wrote:
> > > >
> > > > I was getting a whole lot of weird errors in my logfiles about
> > > > 96.86.170.229 not being authoritative.  Some basic command-line
> > > > checks show I'm not getting AXFR from it.
> > > >
> > > > [rick at linuxmafia]
> > > > ~ $ dig sflug.com ns @a.gtld-servers.net. +short
> > > > ns1.sf-lug.org.
> > > > ns1.linuxmafia.com.
> > > > nsy.sunnysidex.com.
> > > > nsx.sunnyside.com.
> > > > [rick at linuxmafia]
> > > > ~ $ dig sflug.com. soa @ns1.sf-lug.org. +short
> > > > ns1.sf-lug.org. jim.well.com. 1695151540 10800 3600 3600000 86400
> > > > [rick at linuxmafia]
> > > > ~ $ dig sflug.com. axfr @ns1.sf-lug.org. +short
> > > > ; Transfer failed.
> > > > [rick at linuxmafia]
> > > > ~ $ dig ns1.sf-lug.org. +short
> > > > 96.86.170.229
> > > > [rick at linuxmafia]
> > > > ~ $