[sf-lug] Draft of a note to my Comcast account manager
Rick Moen
rick at linuxmafia.com
Sat Sep 23 20:07:39 PDT 2023
Whether coincidentally or not, the "manager" of my Comcast customer
account -- probably on the sales/marketing side -- telephoned me today
while I was on the road, asking about customer satisfaction. I think he
mostly wanted to sell me additional services. We chatted about the
current problem, which I described briefly in layman-friendly terms, and
(as I'd pulled off the road to take the call) asked him to please send
me more by e-mail, which he did.
I don't want to rag on the nice gentleman, but telling him exactly what
the issue is _might_ put pressure somewhere useful. Here's a draft of
what I may send him tomorrow night. (Comments welcome.)
From: Rick Moen <rick at linuxmafia.com>
To: "Leonard, Jeffrey" [address redacted]
Cc:
Bcc:
Subject: Current DNS problem
Quoting Leonard, Jeffrey [address redacted]:
> Jeffrey here with Comcast Business following up with you to make sure
> your services for your business are built long-term for you the right
> way.
Thank you for reaching out. Here's the crux of things, Jeffrey:
Can Comcast Business tell me how to have outbound DNS queries function
normally and _not_ be forced through the transparent proxy apparently
built into all Technicolor CGA4332COM gateways with Comcast's current
firmware? For example, would enabling "passthrough mode" disable that
mandatory (yet undisclosed) proxying of all DNS port 53 traffic?
Or is there _any_ other way to disable or go around the port 53 proxy?
Any "yes" answer will save this account. If not, Comcast will soon lose
this account permanently. That's the bottom line.
Why? Because an undisclosed, new DNS proxy breaks all authoritative
nameservice to/from my DNS server. And because it falsifies the origin
of _all_ DNS information, intruding on Comcast customer an undisclosed
security and privacy risk.
If I have to leave Comcast over this, you can bet I will also be
prominent in making this deal-killing problem widely known, too.
As a senior sysadmin, I'm better positioned than most in so doing.
Thursday, Sept, 21, my authoritative DNS nameserver suddenly became, by
around 1:02pm, unable to communicate for DNS zone transfers with (any
and all) other authoritative nameservers, an essential function for all
such sources of DNS information. Calling Support, I was told "emergency
support" trouble ticket CR109553011 had been opened and would be
escalated: Support acknowledged that my description was too technical
for them, but said they'd need to have a Comcast on-site technician
visit, as the next step.
On-site tech Alberto arrived at 10am Friday. I showed him the problem:
All zone transfers fail. Port 53 DNS queries to a remote DNS server 10
hops away get a (wrong) answer from one hop away, i.e., the gateway box.
Also, DNS queries to a remote machine that is _not a DNS server_ get a
DNS reponse from one hop away, i.e., the gateway box.
In other words, the Technicolor box is intercepting and messing with all
outbound DNS traffic, and preventing all zone transfers across DNS port
53 from working. Alberto took a cellphone photo of my evidence, and
asked me to send his boss Nick Varela the evidence, which I did at
10:47am. I was told _this_ would result in the matter being immediately
escalated to 2nd or 3rd level Support.
12 hours later, 10pm, I'd heard nothing, so logged into
business.comcast.com, and looked at the ticket. Here's what I saw:
Ticket Status Service Address Date Opened Last Action
CR109553011 Closed Internet 1105 ALTSCHUL AVE Sep 22, 2023 Sep 22, 2023
UNIT HMOFC, MENLO
PARK CA 94025
Summary: Cant Connect/Surf
_That_ is the "emergency support" ticket. Problem description "Cant
Connect/Surf" (dead wrong) and "Closed" (dead wrong).
I called Support _again_ (Friday evening, yesterday). A bright young
man took down my problem description again, resulting in this (just
re-checked):
Ticket Status Service Address Date Opened Last Action
CR109708661 Assigned Internet 1105 ALTSCHUL AVE Sep 23, 2023 Sep 23, 2023
UNIT HMOFC, MENLO
PARK CA 94025
Summary: Gateway Configuration
Not progress, but at least not wrong and closed. _But_, nobody has
telephoned me except a marketing query asking me to do a survey,
nobody has admitted the now-obvious truth that your box is imposing
an undisclosed, new proxy on all DNS traffic, and nobody's telling
me how to turn it off (or saying "we require it; sorry it breaks your
DNS").
Why do I say this is a covert, undisclosed breach of all customers'
security & privacy? Let me show you. You may have heard of
"traceroute": It's a command-line tool to, as the name implies, trace
the series of router "hops" to a destination on the Internet. We'll be
using it.
CASE 1: DNS FOR DOMAIN SFLUG.COM.
My server helps publish DNS data for domain sflug.com, as a secondary
nameserver. To do this, it periodically pulls down the "zone" data, the
DNS contents, from the primary server at IP address 96.86.170.229 .
Here, we use traceroute in the normal "ICMP" way, (correctly) showing
that 96.86.170.229 is 10 hops away. (The -n flag means numeric;
don't bother with names, just show IPs.)
# traceroute -n 96.86.170.229
traceroute to 96.86.170.229 (96.86.170.229), 30 hops max, 40 byte packets
1 96.95.217.102 2.471 ms 2.755 ms 2.570 ms
2 100.92.140.66 17.655 ms 20.439 ms 100.92.140.67 19.480 ms
3 96.216.9.185 17.928 ms 16.996 ms 16.585 ms
4 68.85.154.113 16.111 ms 68.85.154.117 15.725 ms 15.363 ms
5 96.108.99.249 14.972 ms 96.108.99.245 22.969 ms 96.108.99.249 14.137 ms
6 68.86.143.93 13.733 ms 19.038 ms 68.86.143.89 18.125 ms
7 162.151.87.226 18.228 ms 162.151.86.58 13.888 ms 162.151.87.226 15.427 ms
8 162.151.79.134 15.357 ms 15.344 ms 162.151.78.186 12.990 ms
9 68.85.103.154 13.999 ms 68.85.191.206 13.178 ms 12.772 ms
10 96.86.170.229 22.183 ms 27.416 ms 26.889 ms
#
Traceroute is flexible, though: Instead of tracing using ICMP packets,
we can trace the route using TCP packets (one of the types DNS goes
over) and target the destination's port 53, where DNS queries go.
That's the "Tp 53" bit we're adding:
# traceroute -nTp 53 96.86.170.229
traceroute to 96.86.170.229 (96.86.170.229), 30 hops max, 40 byte packets
1 96.86.170.229 2.726 ms 2.310 ms 2.198 ms
#
Wait, one hop? How can that be correct? Answer: It isn't. The query
is intercepted immediately at the Comcast gateway box, and never sent to
the requested destination. Hidden software in the box's firmware sends
it, instead, to one of Comcast's DNS servers -- overriding my
instructions.
What happens when my server requests the sflug.com "zone" information
from IP 96.86.170.229? We can simulate this using another command-line
tool, "dig", which is for doing DNS queries. We'll request data of type
AXFR, which is what does transfers of complete zones:
# dig sflug.com axfr @96.86.170.229
; <<>> DiG 9.4.2 <<>> sflug.com axfr @96.86.170.229
;; global options: printcmd
; Transfer failed.
#
Above is exactly what started happening to _all_ of my server's
authoritative DNS service, for a couple of dozen Internet domains,
mine and others: My DNS nameserver tries to contact the remote peer
nameserver for a zone update (or the peer tries to do the same reaching
to my nameserver), and the transfer simply fails. Comcast's router
hijacks the request for that zone, diverts it to a Comcast DNS
nameserver, and it or something else handling the request cannot comply,
so this critical bit of Internet infrastructure simply breaks.
CASE 2: ANY TYPE OF DNS QUERY AT ALL, BY ANYTHING
Every thing that accesses Internet services ends up needing DNS
resolution. Consider, for example, Web site www.comcast.com .
To get there, your Web browser makes a request to your device's
DNS resolver, which looks up the IP address, and the Web browser
then fetches a page from that IP.
Let's see how that works on my laptop, through Comcast's gateway,
again, simulating the lookup using "dig":
% dig www.comcast.com +short
www.comcast.com.edgekey.net.
e523.dscb.akamaiedge.net.
23.4.226.74
%
If I omit the "+short" flag, the more-verbose output includes where the
DNS answer came from:
;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
Which is a Comcast DNS nameserver.
Correct and uneventful, no problem, right? But if I address that query
to a NON-nameserver, an IP address out on the Internet like 1.2.3.4 that
cannot and should not resolve DNS, I should get no answer, right? Well,
surprise.
% dig www.comcast.com @1.2.3.4 +short
www.comcast.com.edgekey.net.
e523.dscb.akamaiedge.net.
104.100.85.21
%
Even though it's a different IP than before, it's still a correct and
valid answer, but the problem is: 1.2.3.4 cannot _give_ an answer to
DNS questions. Again, omitting "+short" reveals where it supposedly came
from:
;; SERVER: 1.2.3.4#53(1.2.3.4)
This is obviously falsified. IP 1.2.3.4 is not a nameserver at all.
This lie gets told only for (all) _remote_ query targets. If I make the
same query directed to the laptop I'm typing this on, IP 10.1.10.229,
I (correctly) get query failure.
% dig www.comcast.com @10.1.10.229
; <<>> DiG 9.10.6 <<>> www.comcast.com @10.1.10.229
;; global options: +cmd
;; connection timed out; no servers could be reached
%
CONSEQUENCES OF FALSIFYING DNS ARE BAD AND FAR-REACHING
I can barely start to describe to you how much undisclosed damage this
does to customers' security and privacy. Almost _all_ Internet
operations critically rely on correct and secure DNS. With this
undisclosed change, Comcast customers cannot trust the accuracy and
honesty of any DNS information for anything. Comcast is in a position
to control, to finely monitor, to alter, to selectively censor and
prohibit, to manipulate any DNS results at any time.
This change was not disclosed to us customers, not to people like me
who need _real_ port 53 communication for our DNS nameservers to
function, and not to any other customers, either.
I suspect this was deliberately pushed out to customers' gateway boxes
as an invisible firmware update: That is why the Technicolor model
CGA4332COM swapped into service in my house about a month ago worked
fine until Thursday when it got slipstreamed a firmware update.
And customer were NOT TOLD that, henceforth, all DNS would be
trapped and routed through Comcast's DNS servers, _irrespective_
of their configuring DNS servers of their own preference. Customers'
instructions about nameserver identity will henceforth be silently
overridden.
BREACH OF CONTRACT; POTENTIAL TORT LIABILITY
I'll be blunt: This sabotage of DNS access reneges on our Comcast
Business contract terms. I pay not only for general Internet
connectivity but also for five fixed IP addresses. Fixed IP addresses
exist to run servers. That is the functionality I pay good money for,
and the mandatory port 53 proxy convertly prevents that functionality
in a particularly sneaky manner. Comcast's conduct almost certainly
breaches the implied covenant of good faith and fair dealing and may
constitute other business torts. It also may raise hostile scrutiny
from Federal and state regulators.
On a personal level, though, again, I will be satisfied enough if
Comcast Engineering or someone else at Comcast can, in a reasonable
amount of time, provide a solution that restores real TCP/UDP port
53 access to my installation. I am open to anything that works.
If that does not happen, I will soon be ending our business relationship
and investigating additional remedies.
-- Rick Moen
rick at linuxmafia.com
(650) 283-7902
More information about the sf-lug
mailing list