[conspire] Screenshot mentality

[Rick Moen]

Back when I worked for SourceForge, Inc., a major corporate site where
we installed SourceForge for internal use delegated to a secretary all
contact with us.  She opened up a support ticket with us for some
alleged problem, and there was no competent description of the problem,
just about a dozen hideously overlarge MS-Windows bitmaps as 'screenshots'
-- that completely failed to show the problem.

I've found this weird fixation with screenshots as supposed raw
diagnostic data, damned near everywhere.  Helldesk (er, 'helpdesk') 
staff tend to be deluged with them.  What dumbfounds me, though, is when
helpdesk people _request_ it.


I work in a department of system administators, called 'Ops'.  Most
internal infrastructure at the firm is managed by a different, very
Microsoft-centric department called 'Corporate IT' (that uses an opaque
and generally bad trouble-ticket system called Servicedesk).  I knew
there was a culture gap, but:




Corp. IT servers must cease using ii52-36 = 10.92.4.56 and ii39-7 10.92.8.74 DNS
Servicedesk Request ID : 24367
Requested by Rick Moen on May 10, 2013 05:45 PM

Some Corp IT servers are still sending DNS queries to two dnsint hosts
that are being decommissioned:
ii39-7    10.92.8.74
ii52-36   10.92.4.56

I see in those dnsint hosts DNS querylogs requests originating from
these office IP addresses:

10.15.30.191
10.15.2.20

I'm not sure what host the former is (but suspect it's Corp. IT).  The
latter is AD server PMGI-DC3, according to my records.

As another specific, I note that this Corporate IT IP is also still
sending DNS queries to obsolete Savvis dnsint host ii39-7's IP
(10.92.8.74):

10.15.14.151

Here, FYI, are the new Savvis dnsint servers.  You should use their IPs
as DNS server targets instead of those of ii39-7 and ii52-36:
Internal slave ii15-11, Savvis Cage 340, backnet IP 10.92.13.86
Internal slave ii18-11, Savvis Cage 340, backnet IP 10.92.13.85

Please make sure that those two and any other Corp. IT machines cease
querying ii39-7 and ii52-36 for DNS, as they will soon go away
completely.




[Followup:]

Another affected Corp. IT box:
10.10.2.21; // Corp. IT DNS server in Taiwan

Another:
10.18.2.20 Florida AD server

Just to make sure we're really clear about this, this problem needs to
be addressed immediately, as ii39-7 and ii52-36 are going away and any
machine still looking to get DNS from them will have a serious problem.




from: Corp. IT guy
Date: May 14, 2013 06:40 PM

I am looking into this and here is what I have so far on 10.15.2.20 and
10.18.2.20:
requests for [domain name] DNS zone are forwarded off to 10.92.6.112

10.15.30.191 is a management IP address (DRAC) for one of the Corp
servers and DNS on this interface is not configured (according to dell
open manage software).
since this particular box came from OPS, we might need to reboot it and
get into BIOS to see if DNS is configured there.

10.15.14.151 is a desktop box with name jambopt960b (probably James
Bustamante's box). He might be querying ii39-7 and ii52-36 manually.

Please provide some screen grabs of what you are looking at, so we can
troubleshoot further.

Thank you,




>From :   Rick Moen
On : May 17, 2013 11:44 AM

> Please provide some screen grabs of what you are looking at, so we 
> can troubleshoot further.

Really?  Seriously?  You don't believe something exists until I take a
screenshot?  You needed a photograph?

The raw data are on display in [Ops trouble ticket URL], based on BIND9
querylogs I gathered 2013-05-07 on old dnsint hosts ii52-36 and ii39-7.
You can look there.

I am just re-collecting that raw data using queries of the last couple
of days.

dnsint host 1 of 2 (ii52-36):

[root at ii52-36 (11:36:02) ~]# awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.15
[root at ii52-36 (11:36:07) ~]# awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.18
[root at ii52-36 (11:36:11) ~]# awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.10 
[root at ii52-36 (11:36:18) ~]#

dnsint host 2 of 2 (ii39-7):

[root at ii39-7 (11:38:18) ~]# awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.15
10.15.2.19
10.15.2.20
10.15.2.21
[root at ii39-7 (11:38:33) ~]#  awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.18
10.18.2.20
10.18.2.21
[root at ii39-7 (11:38:59) ~]#  awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.10
10.10.2.20
10.10.2.21
[root at ii39-7 (11:39:18) ~]# awk '{print $4}' /var/named/chroot/logs/query.log* | sed 's/#.*//'| sort -u | grep ^10.10
10.10.2.20
10.10.2.21
[root at ii39-7 (11:39:18) ~]# ls -l /var/named/chroot/logs/query.log*
-rw-r--r-- 1 named named 10933668 May 17 11:41 /var/named/chroot/logs/query.log
-rw-r--r-- 1 named named 20971532 May 17 08:07 /var/named/chroot/logs/query.log.0
-rw-r--r-- 1 named named 20971558 May 17 00:40 /var/named/chroot/logs/query.log.1
-rw-r--r-- 1 named named 20971570 May 16 17:12 /var/named/chroot/logs/query.log.2
[root at ii39-7 (11:41:25) ~]# 

There.  Those are logfile entries from Corp IT servers within the past
three days.  If you tell me you won't believe that console prompt
session until I've taken a screenshot of it and attached it to
ServiceDesk, let me know and I can do so.

Meanwhile, please be advise that decommissioning of ii52-36 and ii39-7
will happen as soon as Ops can arrange it.  If Corp. IT servers have
problems because they are still querying ii52-36 and ii39-7, that will
be a shame but we'll have given ample warning.