[conspire] letsencrypt.org - not that hard, even from "scratch", and for wildcards, ...

Fri May 8 06:30:03 PDT 2020

> From: "Ruben Safir" <ruben at mrbrklyn.com>
> Subject: Re: [conspire] Lets Encrypt
> Date: Thu, 7 May 2020 10:08:34 -0400

> On Wed, May 06, 2020 at 10:01:09PM -0700, Michael Paoli wrote:
>> >From: "Rick Moen" <rick at linuxmafia.com>
>> >Subject: Re: [conspire] Lets Encrypt
>> >Date: Wed, 6 May 2020 20:46:07 -0700
>>
>> >Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
>> >
>> >>>From: "Ruben Safir" <ruben at mrbrklyn.com>
>> >>>Subject: [conspire] Lets Encrypt
>> >>>Date: Wed, 6 May 2020 16:11:55 -0400
>>
>> >>>Has anyone used it successfully?
>> >>
>> >>Heck yeah!  Been usin' it for years now!
>> >>It's not *that* hard.
>> >
>> >I think Ruben might appreciate a look at your implementation scripts,
>> >Michael.
>>
>> Yup, ... watch this space for updates.  ;-)
>
> I am sick of banging my head on this.  I wish I could do this
> with a wildcard domain, which evidently needs to be done through
> DNS.  There instructions (IMO) are completely incoherent on this.

Stop banging your head.  Read(/skim) some documentation.  It's not all
that hard!  Uhm, no, their (not there) instructions aren't that difficult
to comprehend.  Let's see ...:

Ain't that hard, let's see - from (relative) "scratch" ...
o a relatively minimal Debian stable installation
o not generally covered: generating key, generating CSR, changing DNS,
   rationale on not covering those bits:
   o key and CSR you'd need to do anyway for any CA, not specific to
     letsencrypt.org
   o DNS maintenance needs be done anyway, so not a DNS tutorial or the
     like, will presume reader knows how to make and verify requisite
     DNS changes

Okay, start with aforementioned Debian installation.
Debian stable now likely more than suffices on sufficiently new
client - we'll use certbot - widely supported, from the project itself,
simple enough, and will well do the needed (there are lots of other
clients also available - or heck, don't like 'em?  Write your own - the
API is quite well documented, so any capable programmer can write their
own client if they so choose).

Let's make sure our information on package versions available is current,
and that we're current ... I'll show also config lines I have in
/etc/apt/sources.list:

#  grep '^[ \t]*[^ \t#]' /etc/apt/sources.list
deb http://deb.debian.org/debian/ buster main non-free contrib
deb-src http://deb.debian.org/debian/ buster main non-free contrib
deb http://deb.debian.org/debian-security buster/updates main contrib non-free
deb-src http://deb.debian.org/debian-security buster/updates main  
contrib non-free
deb http://deb.debian.org/debian/ buster-updates main contrib non-free
deb-src http://deb.debian.org/debian/ buster-updates main contrib non-free
#  apt-get -y update
#  apt-get -y upgrade
#  apt-get --no-install-recommends -y install certbot python-certbot-doc man

In the above, I give the --no-install-recommends option to avoid pulling
in additional stuff I don't need, and prefer not to drag in here.
That option can also be configured in the apt configuration if one wants
that for default behavior.
I do also explicitly pull in the documentation, to make things easier -
including checking against the exact version installed.
Oh, I did say it was relatively minimal ... also installed the man package,
so we can now usefully do:
$  man certbot

And how did I determine the package for that documentation?:
$  apt-cache search certbot-doc

By default, certbot likes to do stuff as root and automatically.
For illustration purposes here, we're not gonna do that.  No
privileged operations needed - a regular unprivileged user will
quite suffice as we're going to do it.  That also means we'll have to
pass along a few more options/arguments and/or configure things a bit
differently to do it that way - that's fine - mostly want to show that
any user can get the certs - no particular privilege needed.
Heck, could even install under user's HOME directory and not need root
to even install the client software - but I'm not going to show
that here.  I also figure if you can manage DNS and install certs,
you can probably install certbot.

#  let's disable certbot's cron-driven stuff:
#  find /etc/cron* -name '*certbot*' -print
/etc/cron.d/certbot
#  grep -n '^[ \t]*[^ \t#]' /etc/cron.d/certbot
14:SHELL=/bin/sh
15:PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
17:0 */12 * * * root test -x /usr/bin/certbot -a \! -d  
/run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q  
renew
#

Hmmm, ed would be nice ...
#  apt-get --no-install-recommends -y install ed

and comment out that certbot cron line:
#  ed /etc/cron.d/certbot
775
17
0 */12 * * * root test -x /usr/bin/certbot -a \! -d  
/run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q  
renew
s/^/#/p
# 0 */12 * * * root test -x /usr/bin/certbot -a \! -d  
/run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q  
renew
w
776
q
#

For example/illustration here, I'll use domain example.com and email
postmaster at example.com - substituting those in for the actual domain
I ran these tests on.

So, let's say we've generated our key and CSR.
We'll put the CSR, in PEM format, in file:
CSR.pem
On the CSR, we request these for CN and SAN
CN = *.example.com
X509v3 Subject Alternative Name (SAN):
DNS:*.example.com, DNS:example.com
For regular user ID here, I've created an account named "test" - no special
privileges.
So, hereafter, the stuff shown with prompt of "$ " is done with the test ID.

$  man certbot

So, we read/skim it.  Doing it as non-root, there are a fair number
of non-default options we're going to want ... and consistently so.
We'll need to register first (that sets up our API key, so we can do
subsequent requests using that, without needing to register again).
And when we request certs, likewise we'll want a quite same set of
options each time - and with some of those options in common to both.
So, rather than type that all out repeatedly, let's create a few
fairly simple scripts to handle that - and also a file with the
common options ... actually two such files.
Let's do these files/scripts:
.my_certbot_common - for common settings (sourced in from others), we
   also set some options so that it won't explicitly ask us some
   questions (you may want to review and possibly set these differently)
my_certbot.register - to register (generally only need do that once)
my_certbot.getcert - to get actual cert
my_certbot.getcert.test - like above, but use staging/test cert - so it
   essentially does everything the same, except instead of the production
   server, it instead uses test/staging server, and the certificates
   issued are not valid (not CA trusted root signed - and clearly so) -
   test/staging is also not rate limited, and is suitable for testing
   when wants to test everything out but not get an actual cert.  We'll
   also configure this one slightly different on the email (which I
   think it respects?  Documentation says it does for registration at
   least) - hoping that will suppress reminders about expiring cert (in
   this case we don't care for or want the reminder ... the reminders
   are also clear enough that they're invalid test/staging certs, so one
   can also fully test the reminder email stuff - if so desired)
.my_certbot.getcert_common - the two above scripts will have a whole lot
   in common - so lets shove most of that commonality here, for easier and
   more consistent maintainability

$  vi .my_certbot_common my_certbot.register  
.my_certbot.getcert_common my_certbot.getcert my_certbot.getcert.test
$  chmod u+x my_certbot.*

And, let's have a look at what we have for and in our relevant files now:
$  ls -ld .my_certbot_common my_certbot.register  
.my_certbot.getcert_common my_certbot.getcert my_certbot.getcert.test
-rw------- 1 test test 374 May  8 11:37 .my_certbot.getcert_common
-rw------- 1 test test 428 May  8 11:21 .my_certbot_common
-rwx------ 1 test test 877 May  8 12:09 my_certbot.getcert
-rwx------ 1 test test 895 May  8 12:24 my_certbot.getcert.test
-rwx------ 1 test test 750 May  8 12:09 my_certbot.register
$  more .my_certbot_common my_certbot.register  
.my_certbot.getcert_common my_certbot.getcert my_certbot.getcert.test  
| expand -t 4
::::::::::::::
.my_certbot_common
::::::::::::::
#  file to be used to source in our common settings

MY_CERTBOT_DEFAULT_EMAIL='postmaster at example.com'
MY_CERTBOT_CONFIG_DIR="$HOME"/etc/letsencrypt
MY_CERTBOT_LOGS_DIR="$HOME"/var/log/letsencrypt
MY_CERTBOT_WORK_DIR="$HOME"/var/lib/letsencrypt
MY_CERTBOT_COMMON_OPTIONS="--agree-tos --no-eff-email --config-dir  
$MY_CERTBOT_CONFIG_DIR --logs-dir $MY_CERTBOT_LOGS_DIR --work-dir  
$MY_CERTBOT_WORK_DIR --manual-public-ip-logging-ok"
::::::::::::::
my_certbot.register
::::::::::::::
# !/bin/sh

#  Register with letsencrypt.org - generally only need to do this once.
#  see also:
#  "$HOME"/.my_certbot_common - for where relevant local data is stored
#  certbot(1)

#  vi(1) :se tabstop=4

set -e # exit on failures

umask 077 # security / least privilege principle

. "$HOME"/.my_certbot_common # get our common settings

#  ensure that we have our needed directories:
for dir in \
     "$MY_CERTBOT_CONFIG_DIR" \
     "$MY_CERTBOT_LOGS_DIR" \
     "$MY_CERTBOT_WORK_DIR"
do
     [ -d "$dir" ] &&
         continue # already have this directory, check next
     mkdir -p "$dir"
     [ -d "$dir" ] || {
         echo "$0: failed to create directory $dir, aborting" 1>&2
         exit 1
     }
done

certbot \
     --email "$MY_CERTBOT_DEFAULT_EMAIL" \
     $MY_CERTBOT_COMMON_OPTIONS \
     register
::::::::::::::
.my_certbot.getcert_common
::::::::::::::
#  file to be used to source in our common settings when getting cert

#  get our common settings
. "$HOME"/.my_certbot_common || exit

MY_CERTBOT_GETCERT_COMMON_OPTIONS="$MY_CERTBOT_COMMON_OPTIONS  
--duplicate --force-renewal --manual --preferred-challenges dns --csr  
$HOME/CSR.pem --cert-path $HOME/cert.pem --fullchain-path  
$HOME/fullchain.pem --chain-path $HOME/chain.pem"
::::::::::::::
my_certbot.getcert
::::::::::::::
# !/bin/sh

#  Get letsencrypt.org CA signed cert
#  See file: "$HOME"/.my_certbot.getcert_common
#  for locations of: expected CSR file, and where output files (including
#  cert) will be written
#  see also: certbot(1)

#  vi(1) :se tabstop=4

set -e # exit on failures

umask 077 # security / least privilege principle

. "$HOME"/.my_certbot_common # get our common settings

#  get our common settings when getting cert:
. "$HOME"/.my_certbot.getcert_common

#  ensure that we have our needed directories:
for dir in \
     "$MY_CERTBOT_CONFIG_DIR" \
     "$MY_CERTBOT_LOGS_DIR" \
     "$MY_CERTBOT_WORK_DIR"
do
     [ -d "$dir" ] &&
         continue # already have this directory, check next
     mkdir -p "$dir"
     [ -d "$dir" ] || {
         echo "$0: failed to create directory $dir, aborting" 1>&2
         exit 1
     }
done

certbot \
     --email "$MY_CERTBOT_DEFAULT_EMAIL" \
     $MY_CERTBOT_GETCERT_COMMON_OPTIONS \
     certonly
::::::::::::::
my_certbot.getcert.test
::::::::::::::
# !/bin/sh

#  Get test (invalid) letsencrypt.org cert
#  See file: "$HOME"/.my_certbot.getcert_common
#  for locations of: expected CSR file, and where output files (including
#  cert) will be written
#  see also: certbot(1)

#  vi(1) :se tabstop=4

set -e # exit on failures

umask 077 # security / least privilege principle

. "$HOME"/.my_certbot_common # get our common settings

#  get our common settings when getting cert:
. "$HOME"/.my_certbot.getcert_common

#  ensure that we have our needed directories:
for dir in \
     "$MY_CERTBOT_CONFIG_DIR" \
     "$MY_CERTBOT_LOGS_DIR" \
     "$MY_CERTBOT_WORK_DIR"
do
     [ -d "$dir" ] &&
         continue # already have this directory, check next
     mkdir -p "$dir"
     [ -d "$dir" ] || {
         echo "$0: failed to create directory $dir, aborting" 1>&2
         exit 1
     }
done

certbot \
     --test-cert \
     --register-unsafely-without-email \
     $MY_CERTBOT_GETCERT_COMMON_OPTIONS \
     certonly
$

And, expectedly, we have relatively little difference between the
my_certbot.getcert and my_certbot.getcert.test programs:
$  diff <(<my_certbot.getcert expand -t 4) <(<my_certbot.getcert.test  
expand -t 4)
3c3
< # Get letsencrypt.org CA signed cert
---
> # Get test (invalid) letsencrypt.org cert
36c36,37
<     --email "$MY_CERTBOT_DEFAULT_EMAIL" \
---
>     --test-cert \
>     --register-unsafely-without-email \
$

So, let's see if we can smoothly do the registration, and get our cert!
Registration:
$  ./my_certbot.register
Saving debug log to /home/test/var/log/letsencrypt/letsencrypt.log

IMPORTANT NOTES:
  - Your account credentials have been saved in your Certbot
    configuration directory at /home/test/etc/letsencrypt. You should
    make a secure backup of this folder now. This configuration
    directory will also contain certificates and private keys obtained
    by Certbot so making regular backups of this folder is ideal.
$

Well, that was certainly easy.
We have our CSR file in the appropriate place:
$  ls -l CSR.pem
-rw------- 1 test test 1001 May  8 10:04 CSR.pem
$

So, let's get a cert - we'll get a test cert - if that works fine,
actual production cert would be trivially different to obtain (for
production, just use my_certbot.getcert.test instead of
my_certbot.getcert.test).

$  ./my_certbot.getcert.test
...
Please deploy a DNS TXT record under the name
_acme-challenge.example.com with the following value:

urC99wtxH-Xla7_pQxAefCbb2JG7NEIIeeXRez4qO44

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - -
Press Enter to Continue
...
Please deploy a DNS TXT record under the name
_acme-challenge.example.com with the following value:

dmS_i4cDPXo_JvwKfMQRn196i4sTDdu8aewPxe2RGxo

Before continuing, verify the record is deployed.
...
Press Enter to Continue
...
Server issued certificate; certificate written to /home/test/cert.pem
  - Congratulations! Your certificate and chain have been saved at:
    /home/test/fullchain.pem
...
$

And done!

And peeking a bit at the (test) cert:
$  openssl x509 -in cert.pem -text -noout | sed -ne '/Issuer:/p;/Not  
After/p;/Subject: CN = /p;/Subject Alternative Name/{N;p;q}'
         Issuer: CN = Fake LE Intermediate X1
             Not After : Aug  6 11:30:02 2020 GMT
         Subject: CN = *.example.com
             X509v3 Subject Alternative Name:
                 DNS:*.example.com, DNS:example.com
$

So, in conclusion ...
wildcard and/or (multi-name/domain) CA signed certs from letsencrypt.org,
ooh, and no superuser privileges required on client at all to obtain
such certs.  Not that hard.  Sure, need to read a bit of documentation,
maybe poke at it slightly to possibly work out a (minor) kink or two,
but pretty straight-forward.

And ... subsequent certs?  Easy peasy.  In our example shown, just
drop the CSR in place in the file (and probably not a bad idea
to remove the cert.pem chain.pem fullchain.pem files - presumably having
safely saved them elsewhere first).  Then just run the program again,
make the requisite DNS validation entries ... boom, done.
Definitely not rocket science.
And if/when that's "too much work" - or one gets tired of the
manual step parts of that ... well, can automate (much!) more of it.
Generally, with privilege, certbot can install certs for you,
automagicly checking and replacing those (if you trust it that much
and want it to do that for you).  Even without that, can use
other options/programs, e.g. "hook" programs, to automate otherwise
manual steps (such as DNS updates).  The certbot program has quite
a set of options for configuring various ways for it to automate
DNS updates (and clean out those "temporary" entries after, when it's
done validating and they're no longer needed).  Validation can also
be done other ways, e.g. (for non-wildcard) using http (or https),
just place the requisite files on the webserver.  It can even
fire up a minimal temporary web server and serve up that data (which
might be particularly useful and convenient if you're actually wanting
to use the certs for something other than http/https - e.g.
SMTP STARTTLS).

http://linuxmafia.com/pipermail/conspire/2020-May/010723.html