[sf-lug] On the value of verbatim diagnostic data

[Rick Moen]

A little story about the importance of providing verbatim symptoms,
preferably copied and pasted directly from the command line.

In investigating a problem, I noticed a DNS nameserver wasn't 
answering remote DNS queries.  I ssh'ed into the nameserver, to check:

  [rmoen at ki42-18 (14:25:19) ~]$ service named status
  rndc: error: none:0: open: /etc/rndc.key: permission denied
  rndc: could not load rndc configuration
  named dead but subsys locked
  [rmoen at ki42-18 (14:25:43) ~]$ 

I wasn't paying enough attention, but Web-searched on 'named dead but
subsys locked'.  (If you spot the problem, hold that thought; I was a bit
asleep at the wheel, at that moment.)

I was thinking 'Wait, is process BIND = named dead?  Why?'  And I was
looking for clues.  Web-search hits like
http://www.linuxforums.org/forum/networking/186272-named-dead-but-subsys-locked.html
give what would ordinarily be sound advice:

   That means that BIND crashed or was shutdown uncleanly, and there is
   a stale lock file /var/lock/subsys/named.  Make sure BIND is not running, 
   delete the lock file, then start the service.

So, I tried:

[rmoen at ki42-18 (15:32:41) ~]$ rm /var/lock/subsys/named
rm: remove write-protected regular empty file `/var/lock/subsys/named'?  y
rm: cannot remove `/var/lock/subsys/named': Permission denied
[rmoen at ki42-18 (15:32:45) ~]$ 

And _then_ I immediately spotted my bonehead error:  I wasn't the root
user.  D'oh!  As it turned out, BIND was not dead at all.  (Cause of server
nonresponse was actually an unrelated factor, a missing route.)

However, suppose when I say the initial 'named dead but subsys locked'
post, I had gone onto a Linux community mailing list and asked

   Hey, what does the error 'named dead but subsys locked' mean
   when I try to get service status on BIND?  

Somebody might quite reasonably have replied 'That means that BIND
crashed or was shutdown incorrectly...', which is _usually_ the right
answer but not in this case.

Whereas, if I'd posted:

  [rmoen at ki42-18 (14:25:19) ~]$ service named status
  rndc: error: none:0: open: /etc/rndc.key: permission denied
  rndc: could not load rndc configuration
  named dead but subsys locked
  [rmoen at ki42-18 (14:25:43) ~]$ 

  What does that error 'named dead but subsys locked' indicate?

Someone could have easily spotted 'rmoen' instead of 'root' and said 
"Dude, your username doesn't have privilege to do that.  su to root.'