[conspire] Slice of life

[Carl Myers]

Yeh, good point - I believe the "short time later" played no small part in the
outtage I was referring to - sadly, problems aren't always noticeable until the
client's DNS caches clear out.

The advantage of the system I described is, changes can be made without having
to make the request - to beta, or gamma.  Then, after careful QA and review (and
a "request" which is possibly signed off by a manager of some sort) those
changes can be promoted to production in a controlled way, without accidentally
promoting other changes that were not reviewed.  It gets the best of both worlds
by letting devs avoid some of the process, while still ensuring that changes are
controlled.

That said, as I said before, it does suffer from some problems too (like
instilling developer laziness).

I am quite biased, being a "build engineer" myself and having spent my 4 year
career focused on developer tools more than end-user apps, but I believe
strongly that people who do not have such a system in place are doomed to
implement it, piece by piece, gradually over time (or, suffer through horrific
problems and lose valuable productivity, and eventually, developers fed up with
the situation).

On an unrelated note, I also liked your mention of the advantage of scripting
over crappy GUI tools - what an exquisite example to use when someone asks "Why
should I bother to learn this crap anyways when I can just use a GUI?".  The
more I use the CLI, the more comfortable I get with it and the more productive
it makes me.


On Fri, Sep 18, 2009 at 02:12:49PM -0700, Rick Moen wrote:
> Date: Fri, 18 Sep 2009 14:12:49 -0700
> From: Rick Moen <rick at linuxmafia.com>
> To: conspire at linuxmafia.com
> Organization: Dis-
> Subject: Re: [conspire] Slice of life
> 
> Thanks, Carl.  As you say, it turns out that doing change control
> requests does small miracles in keeping the production, customer-facing
> environment stable.  And yes, DNS can be a particular point of pain,
> especially when the master daemon nameserver won't restart because of a
> syntax problem hidden somewhere in one of several conffiles / include
> files or one of a thousand-plus zonefiles -- but you have to guess
> where, because the BIND9 daemon's abysmal built-in checking means it
> can't tell you where.
> 
> Just to be clear, what I posted was a _request_.  All requests may be
> deployed only after passing review by a fellow senior sysadmin and an
> Operations manager.
> 
> Worst-case scenario is that the responsible party runs the change
> procedure on a Friday afternoon, leaves to go fishing, and everything
> breaks a short while later.  At that point, the existence of a specific,
> scriptable backout procedure means the poor-bastard on-call tech can
> revert the change.
> 
> -- 
> Cheers,            "Orthodoxy is my doxy.  Heterodoxy is someone else's doxy."
> Rick Moen               -- William Warburton, Bishop of Gloucester (1698-1779)
> rick at linuxmafia.com
> 
> _______________________________________________
> conspire mailing list
> conspire at linuxmafia.com
> http://linuxmafia.com/mailman/listinfo/conspire

-- 
Carl Myers 
PGP Key ID 3537595B
PGP Key fingerprint 9365 0FAF 721B 992A 0A20  1E0D C795 2955 3537 595B

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://linuxmafia.com/pipermail/conspire/attachments/20090918/c2773ed7/attachment.pgp>