[sf-lug] Draft of a note to my Comcast account manager
Michael Paoli
michael.paoli at cal.berkeley.edu
Sat Sep 23 21:59:00 PDT 2023
So ... my first pass, general impressions and thoughts ...
Context - the initial target of this communication is sales person.
So, reading/skimming - I'm thinking aren't any parts in it that are
"horrible" or uncalled for. But I'm thinking, however, optimizing for
the target - and hoped for effect ... I might be inclined to reorganized
very roughly as follows ... and I'm not even that sure on it at this
point ... I may want to ruminate on it a bit more, and have at least a
second pass ... and probably some more detailed feedback, after I've
mulled over it a bit more, so, I might also alter my evaluation.
Anyway, first pass, I'm roughly thinking, etc. ...:
Yes, there's significant problem, but perhaps rather than hitting as
hard and detailed on that - at least in initial / earlier bits of the
communication, start with that it's not only a problem, but was recently
"fixed" by Comcast, only to recur again shortly thereafter - and with
the relevant references on that (most notably including Comcast
Business's service tickets or their other reference numbers.
So, I'm thinking probably lead with that.
And then delve into the seriousness of the problem, starting with how it
threatens the business relationship, and not only that, is having other
Comcast Business customers seriously question their relationship with
Comcast Business, and at least some bits of this information and
communications are open to the public, so folks are watching and may
well be noticing if Comcast Business will in fact be able to
satisfactorily resolve this ... or not, and the matter may garner more
attention if Comcast Business cannot or will not satisfactorily resolve
the issue. And as for technical details, sales person would likely get
highly lost by those, so I might be inclined to include those - but
much father down in the email, and mention that, e.g. something like
"for the relevant technical details (and further background), see
further below in this email" ... something roughly like that. So
sales/marketing person won't get lost in that and never make it to
points also relevant to sales/marketing person, yet be well included in
the email, so if/where they pass it along (directly or indirectly) to
relevant technical folks, at least those technical details will also
all be there in the email.
I'm thinking perhaps bringing up stuff like tort and contract violation,
or things of that matter, perhaps better left for later communications,
if Comcast Business fails to, cannot, or will not fix the issue. I'm
thinking a more vague "may garner more attention" or something like
that is probably sufficient (and likely closer to optimal?) at this
point ... and including that it is already starting to get at least some
of that attention (notably that it's hit SF-LUG list, though probably no
need to specifically call that out ... I think something like
"communications ... open to the public" may be sufficient to get the
general point across.
And I'm thinking then go to the technical details and background - or
perhaps include them referring to them as (large) footnotes or included
addendum or the like. May want to also put the customary
signature/contact bits before that so they're not so easily overlooked
- and then probably also at the very tail end of the email.
Anyway, I'm still mulling it over ... but that's kind'a what I think
and come up with for first impressions and related thoughts. I'm also
thinking bit how to best organize the technical and (more detailed)
background bits - not sure how to best present both of those in what
would hopefully be optimal ordering and arrangement.
Also not sure I've necessarily got optimal ordering/structure/approach
on the other bits - but that's what I seem to come up with on those at
least with first pass impressions/thoughts on the matter.
On Sat, Sep 23, 2023 at 8:07 PM Rick Moen <rick at linuxmafia.com> wrote:
>
> Whether coincidentally or not, the "manager" of my Comcast customer
> account -- probably on the sales/marketing side -- telephoned me today
> while I was on the road, asking about customer satisfaction. I think he
> mostly wanted to sell me additional services. We chatted about the
> current problem, which I described briefly in layman-friendly terms, and
> (as I'd pulled off the road to take the call) asked him to please send
> me more by e-mail, which he did.
>
> I don't want to rag on the nice gentleman, but telling him exactly what
> the issue is _might_ put pressure somewhere useful. Here's a draft of
> what I may send him tomorrow night. (Comments welcome.)
>
>
>
> From: Rick Moen <rick at linuxmafia.com>
> To: "Leonard, Jeffrey" [address redacted]
> Cc:
> Bcc:
> Subject: Current DNS problem
>
> Quoting Leonard, Jeffrey [address redacted]:
>
> > Jeffrey here with Comcast Business following up with you to make sure
> > your services for your business are built long-term for you the right
> > way.
>
> Thank you for reaching out. Here's the crux of things, Jeffrey:
>
>
> Can Comcast Business tell me how to have outbound DNS queries function
> normally and _not_ be forced through the transparent proxy apparently
> built into all Technicolor CGA4332COM gateways with Comcast's current
> firmware? For example, would enabling "passthrough mode" disable that
> mandatory (yet undisclosed) proxying of all DNS port 53 traffic?
>
> Or is there _any_ other way to disable or go around the port 53 proxy?
>
>
> Any "yes" answer will save this account. If not, Comcast will soon lose
> this account permanently. That's the bottom line.
>
> Why? Because an undisclosed, new DNS proxy breaks all authoritative
> nameservice to/from my DNS server. And because it falsifies the origin
> of _all_ DNS information, intruding on Comcast customer an undisclosed
> security and privacy risk.
>
> If I have to leave Comcast over this, you can bet I will also be
> prominent in making this deal-killing problem widely known, too.
> As a senior sysadmin, I'm better positioned than most in so doing.
>
>
> Thursday, Sept, 21, my authoritative DNS nameserver suddenly became, by
> around 1:02pm, unable to communicate for DNS zone transfers with (any
> and all) other authoritative nameservers, an essential function for all
> such sources of DNS information. Calling Support, I was told "emergency
> support" trouble ticket CR109553011 had been opened and would be
> escalated: Support acknowledged that my description was too technical
> for them, but said they'd need to have a Comcast on-site technician
> visit, as the next step.
>
> On-site tech Alberto arrived at 10am Friday. I showed him the problem:
> All zone transfers fail. Port 53 DNS queries to a remote DNS server 10
> hops away get a (wrong) answer from one hop away, i.e., the gateway box.
> Also, DNS queries to a remote machine that is _not a DNS server_ get a
> DNS reponse from one hop away, i.e., the gateway box.
>
> In other words, the Technicolor box is intercepting and messing with all
> outbound DNS traffic, and preventing all zone transfers across DNS port
> 53 from working. Alberto took a cellphone photo of my evidence, and
> asked me to send his boss Nick Varela the evidence, which I did at
> 10:47am. I was told _this_ would result in the matter being immediately
> escalated to 2nd or 3rd level Support.
>
> 12 hours later, 10pm, I'd heard nothing, so logged into
> business.comcast.com, and looked at the ticket. Here's what I saw:
>
> Ticket Status Service Address Date Opened Last Action
> CR109553011 Closed Internet 1105 ALTSCHUL AVE Sep 22, 2023 Sep 22, 2023
> UNIT HMOFC, MENLO
> PARK CA 94025
>
> Summary: Cant Connect/Surf
>
> _That_ is the "emergency support" ticket. Problem description "Cant
> Connect/Surf" (dead wrong) and "Closed" (dead wrong).
>
>
> I called Support _again_ (Friday evening, yesterday). A bright young
> man took down my problem description again, resulting in this (just
> re-checked):
>
> Ticket Status Service Address Date Opened Last Action
> CR109708661 Assigned Internet 1105 ALTSCHUL AVE Sep 23, 2023 Sep 23, 2023
> UNIT HMOFC, MENLO
> PARK CA 94025
>
> Summary: Gateway Configuration
>
> Not progress, but at least not wrong and closed. _But_, nobody has
> telephoned me except a marketing query asking me to do a survey,
> nobody has admitted the now-obvious truth that your box is imposing
> an undisclosed, new proxy on all DNS traffic, and nobody's telling
> me how to turn it off (or saying "we require it; sorry it breaks your
> DNS").
>
>
> Why do I say this is a covert, undisclosed breach of all customers'
> security & privacy? Let me show you. You may have heard of
> "traceroute": It's a command-line tool to, as the name implies, trace
> the series of router "hops" to a destination on the Internet. We'll be
> using it.
>
> CASE 1: DNS FOR DOMAIN SFLUG.COM.
>
> My server helps publish DNS data for domain sflug.com, as a secondary
> nameserver. To do this, it periodically pulls down the "zone" data, the
> DNS contents, from the primary server at IP address 96.86.170.229 .
>
> Here, we use traceroute in the normal "ICMP" way, (correctly) showing
> that 96.86.170.229 is 10 hops away. (The -n flag means numeric;
> don't bother with names, just show IPs.)
>
> # traceroute -n 96.86.170.229
> traceroute to 96.86.170.229 (96.86.170.229), 30 hops max, 40 byte packets
> 1 96.95.217.102 2.471 ms 2.755 ms 2.570 ms
> 2 100.92.140.66 17.655 ms 20.439 ms 100.92.140.67 19.480 ms
> 3 96.216.9.185 17.928 ms 16.996 ms 16.585 ms
> 4 68.85.154.113 16.111 ms 68.85.154.117 15.725 ms 15.363 ms
> 5 96.108.99.249 14.972 ms 96.108.99.245 22.969 ms 96.108.99.249 14.137 ms
> 6 68.86.143.93 13.733 ms 19.038 ms 68.86.143.89 18.125 ms
> 7 162.151.87.226 18.228 ms 162.151.86.58 13.888 ms 162.151.87.226 15.427 ms
> 8 162.151.79.134 15.357 ms 15.344 ms 162.151.78.186 12.990 ms
> 9 68.85.103.154 13.999 ms 68.85.191.206 13.178 ms 12.772 ms
> 10 96.86.170.229 22.183 ms 27.416 ms 26.889 ms
> #
>
> Traceroute is flexible, though: Instead of tracing using ICMP packets,
> we can trace the route using TCP packets (one of the types DNS goes
> over) and target the destination's port 53, where DNS queries go.
> That's the "Tp 53" bit we're adding:
>
> # traceroute -nTp 53 96.86.170.229
> traceroute to 96.86.170.229 (96.86.170.229), 30 hops max, 40 byte packets
> 1 96.86.170.229 2.726 ms 2.310 ms 2.198 ms
> #
>
> Wait, one hop? How can that be correct? Answer: It isn't. The query
> is intercepted immediately at the Comcast gateway box, and never sent to
> the requested destination. Hidden software in the box's firmware sends
> it, instead, to one of Comcast's DNS servers -- overriding my
> instructions.
>
> What happens when my server requests the sflug.com "zone" information
> from IP 96.86.170.229? We can simulate this using another command-line
> tool, "dig", which is for doing DNS queries. We'll request data of type
> AXFR, which is what does transfers of complete zones:
>
> # dig sflug.com axfr @96.86.170.229
>
> ; <<>> DiG 9.4.2 <<>> sflug.com axfr @96.86.170.229
> ;; global options: printcmd
> ; Transfer failed.
> #
>
> Above is exactly what started happening to _all_ of my server's
> authoritative DNS service, for a couple of dozen Internet domains,
> mine and others: My DNS nameserver tries to contact the remote peer
> nameserver for a zone update (or the peer tries to do the same reaching
> to my nameserver), and the transfer simply fails. Comcast's router
> hijacks the request for that zone, diverts it to a Comcast DNS
> nameserver, and it or something else handling the request cannot comply,
> so this critical bit of Internet infrastructure simply breaks.
>
>
>
> CASE 2: ANY TYPE OF DNS QUERY AT ALL, BY ANYTHING
>
> Every thing that accesses Internet services ends up needing DNS
> resolution. Consider, for example, Web site www.comcast.com .
> To get there, your Web browser makes a request to your device's
> DNS resolver, which looks up the IP address, and the Web browser
> then fetches a page from that IP.
>
> Let's see how that works on my laptop, through Comcast's gateway,
> again, simulating the lookup using "dig":
>
> % dig www.comcast.com +short
> www.comcast.com.edgekey.net.
> e523.dscb.akamaiedge.net.
> 23.4.226.74
> %
>
> If I omit the "+short" flag, the more-verbose output includes where the
> DNS answer came from:
> ;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
> Which is a Comcast DNS nameserver.
>
> Correct and uneventful, no problem, right? But if I address that query
> to a NON-nameserver, an IP address out on the Internet like 1.2.3.4 that
> cannot and should not resolve DNS, I should get no answer, right? Well,
> surprise.
>
> % dig www.comcast.com @1.2.3.4 +short
> www.comcast.com.edgekey.net.
> e523.dscb.akamaiedge.net.
> 104.100.85.21
> %
>
> Even though it's a different IP than before, it's still a correct and
> valid answer, but the problem is: 1.2.3.4 cannot _give_ an answer to
> DNS questions. Again, omitting "+short" reveals where it supposedly came
> from:
> ;; SERVER: 1.2.3.4#53(1.2.3.4)
>
> This is obviously falsified. IP 1.2.3.4 is not a nameserver at all.
>
> This lie gets told only for (all) _remote_ query targets. If I make the
> same query directed to the laptop I'm typing this on, IP 10.1.10.229,
> I (correctly) get query failure.
>
> % dig www.comcast.com @10.1.10.229
>
> ; <<>> DiG 9.10.6 <<>> www.comcast.com @10.1.10.229
> ;; global options: +cmd
> ;; connection timed out; no servers could be reached
> %
>
>
>
> CONSEQUENCES OF FALSIFYING DNS ARE BAD AND FAR-REACHING
>
> I can barely start to describe to you how much undisclosed damage this
> does to customers' security and privacy. Almost _all_ Internet
> operations critically rely on correct and secure DNS. With this
> undisclosed change, Comcast customers cannot trust the accuracy and
> honesty of any DNS information for anything. Comcast is in a position
> to control, to finely monitor, to alter, to selectively censor and
> prohibit, to manipulate any DNS results at any time.
>
> This change was not disclosed to us customers, not to people like me
> who need _real_ port 53 communication for our DNS nameservers to
> function, and not to any other customers, either.
>
> I suspect this was deliberately pushed out to customers' gateway boxes
> as an invisible firmware update: That is why the Technicolor model
> CGA4332COM swapped into service in my house about a month ago worked
> fine until Thursday when it got slipstreamed a firmware update.
>
> And customer were NOT TOLD that, henceforth, all DNS would be
> trapped and routed through Comcast's DNS servers, _irrespective_
> of their configuring DNS servers of their own preference. Customers'
> instructions about nameserver identity will henceforth be silently
> overridden.
>
>
> BREACH OF CONTRACT; POTENTIAL TORT LIABILITY
>
> I'll be blunt: This sabotage of DNS access reneges on our Comcast
> Business contract terms. I pay not only for general Internet
> connectivity but also for five fixed IP addresses. Fixed IP addresses
> exist to run servers. That is the functionality I pay good money for,
> and the mandatory port 53 proxy convertly prevents that functionality
> in a particularly sneaky manner. Comcast's conduct almost certainly
> breaches the implied covenant of good faith and fair dealing and may
> constitute other business torts. It also may raise hostile scrutiny
> from Federal and state regulators.
>
> On a personal level, though, again, I will be satisfied enough if
> Comcast Engineering or someone else at Comcast can, in a reasonable
> amount of time, provide a solution that restores real TCP/UDP port
> 53 access to my installation. I am open to anything that works.
>
> If that does not happen, I will soon be ending our business relationship
> and investigating additional remedies.
>
> -- Rick Moen
> rick at linuxmafia.com
> (650) 283-7902
More information about the sf-lug
mailing list