Preventing Domain Expiration
By Rick Moen
If you study accounting and finance, one of the tidbits taught is that financial fraud (via embezzlement and otherwise) is far more pervasive than anyone's willing to admit. It's a loss syndrome businesses (e.g., banks) see no advantage in mentioning: Having been taken for a ride is pretty embarrassing, after all.
Internet users have an equally pervasive — and oddly similar — problem: accidental Internet domain expiration. Your Linux user group or other nonprofit group (or, hey, even your company) is relying on some vaguely defined chain of command to make sure the domain keeps getting renewed, making the assumption everything's fine as long as no disaster has yet happened (which tactic is called "management by exception" in business school — usually just before they cue the ominous music). Somebody drops the ball, the domain everyone's relying on expires when nobody's looking, and when the dust settles you find that a domain squatter's grabbed it.[1] Yes, there are companies that make domain snatching their core business. They do well at it, too. Too well for my taste.
Victims of such raids sometimes attempt to recover using legal threats, usually trademark-based, and the ICANN Uniform Domain-Name Dispute-Resolution Policy (UDRP) to wrestle back their domains, but it's more common to pay the squatter's ransom: That might range from hundreds to tens of thousands of US dollars, depending on the domain and what the market will bear.
Equally common, though, especially for less wealthy victims, is to just quietly concede, watch the squatter deploy a so-called "search engine" site where your Net presence had been, and move your presence to some entirely new domain name you use as a replacement. Every year, I see this happen to individuals and groups: Suddenly, the established domain is a squatter site, and everyone has a new e-mail address, for reasons nobody wants to discuss.
But there's a better way. It doesn't have to happen. It can be prevented.
First up, out on the Net, I found a nice little Bourne shell script by Ryan Matteson (matty91 at gmail dot com) called "domain-check" (http://prefetch.net/code/domain-check — not recommended: see alternative Perl reimplementation d-check, below), which queries the public WHOIS data in real time to check pending domain expiration dates, and integrates nicely with cron and optionally SMTP e-mail notification, in order give responsible parties advance notice of the need to take action. (In this article, as elsewhere, all-caps "WHOIS" refers to the TCP port 43 protocol defined in RFC 3912 for remote information lookup about domain names, etc. Not all TLDs offer that service, as detailed below.)
Ryan's script's only dependencies are awk, whois, and date, whose executable paths must all be correctly set in the script body (and require fixing on typical Linux systems). Plus, you probably need to add a line defining shell environment variable MAIL to point to a proper system outbound mailer, if you wish to do e-mail advisories. (On my system, /usr/bin/mail fits nicely.)
Once you have that set, it's fairly self-explanatory:
$ domain-check -h
Usage: /usr/local/bin/domain-check [ -e email ] [ -x expir_days ] [ -q ]
[ -a ] [ -h ]
{[ -d domain_namee ]} || { -f domainfile}
-a : Send a warning message through email
-d domain : Domain to analyze (interactive mode)
-e email address : Email address to send expiration notices
-f domain file : File with a list of domains
-h : Print this screen
-s whois server : Whois sever to query for information
-q : Don't print anything on the console
-x days : Domain expiration interval (eg. if domain_date < days)
Example output:
$ domain-check -d linuxmafia.com Domain Registrar Status Expires Days Left -------------------------------- ----------------- -------- ------- --------- linuxmafia.com TUCOWS INC. Valid 17-jul-2010 1057
Ryan's implementation of domain-check has two problems: One is that he has inadvertently made its licence technically proprietary (as of v. 1.4), by failing to include rights to modify or redistribute, in his otherwise generous licence statement. Ryan's aware of this oversight, but hasn't yet fixed it at press time.
The other: It can parse the expiration date fields from only a few top-level domains (TLDs), missing some really important ones such as .ORG. In particular, if you run it with e-mailed output (where it really shines, generally, e.g., running as a weekly cronjob to check a list of domains), it says nothing at all about domains within the many TLDs it simply can't handle.
Mind you, as editor Ben Okopnik and I were testing Ryan's script, we realised that adding to it support for additional TLDs could prove non-trivial, and we respect Ryan's accomplishment, as far as it's gone: A brief survey of the 250 country-code TLDs ("ccTLDs", such as .uk and .us) and 21 generic TLDs ("gTLDs", such as .com, .net, .org, .info, etc.) showed dozens of variations in the way expiration dates and registrar names, each needing its own parsing code.
Ryan might appreciate some help with that task: Experienced shell coders might want to send Ryan patches, especially to fill out its currently rather thin TLD coverage. However, we right away spotted the licensing issue, on top of that — and so, for ourselves, decided to switch tactics.
Introducing Ben's domain-check
(2018 update: domain-check is no longer maintained, and always had some design problems. Recommended compatible replacement is Jesse Monroy's d-check, likewise available from my Web site, as well as from GitHub.)
Ben Okopnik fired up his mighty Perl kung fu, and crafted a second implementation, likewise called "domain-check", which now is available with GPL licensing terms at my Web site. It works a treat. Here's how it goes from the command line — obviously fashioned after Ryan's good example:
$ domain-check -d=linuxmafia.com Processing linuxmafia.com... Host Registrar Exp.date/Days left ============================================================================== linuxmafia.com TUCOWS, INC. 17-jul-2010 / 1057
And, of course, it supports the same e-mailed reporting mode that in Ryan's script is so nicely cron-friendly — with the bonus improvement of relying on Perl and a WHOIS client solely, and finding them via PATH without any need to tweak the script.
The Two WHOIS Clients
At present, Ben's domain-check will use, if present, the fairly sophisticated, configurable, and cache-enabled WHOIS client "jwhois" by default, on a hunch that "jwhois" is usually, generally a small bit smarter, and its caching on disk of recently received WHOIS data is usually an advantage, relative to the regular "whois" (/usr/bin/whois) implementation — with automatic fallback to the latter client. However, the WHOIS client comparison is, upon further examination, a mixed bag. For one thing, "jwhois's" results caching (defaulting to a seven-day retention period) can become a problem: Suppose it's Wednesday today, you last checked your friend's domain example.com on Sunday, and that domain's due to expire this coming Saturday. You run domain-check (and it finds "jwhois"); domain-check reports that your friend's weekend expiration is still looming.
But maybe, he/she has (unbeknownst to you) already paid for that renewal, and it took effect yesterday. domain-check won't pick this datum up (while using "jwhois" with 7-day retention), and so issues a false alarm, because it's still using the cache-hit of Sunday's data, now three days old (but already obsolete).
You can ameliorate this situation by, say, reducing the cache period (near the bottom of /etc/jwhois.conf) to 2 hours instead of the default 168 hours = 1 week — but the point is that "jwhois's" default reliance on old data can be misleading.
Nor is it always or unambiguously the case that "jwhois" is "a bit smarter". This is where things get interesting (part one, of two). The worldwide Internet domain system's "whois" data, showing contact information for each domain's owners & operators, which registrar it's enrolled through, when it will expire, and at what IPs its DNS nameservers can be found, is (like DNS itself) yet another distributed information system, with "whois" information for each TLD (among those that offer it) publicly accessible (if at all) either the WHOIS protocol, or Web-based lookup methods, or both, that can query one or more database server holding that data.
Which TLDs offer meaningful information lookup via WHOIS, and at what WHOIS server hostnames in each case? If you're reasonably lucky (regarding the six or seven TLDs you typically care about, no matter where in the world you are), the WHOIS client software you use (which on Linux will be either /usr/bin/whois or /usr/bin/jwhois) already has this knowledge built in. However, the various TLD operators, including the administrators of the 250 country-code TLDs, have an unsettling tendency to move things around, change where their WHOIS data is, terminate WHOIS service, start WHOIS service — without (much) notice. They're supposed to inform IANA of all such changes, whereupon IANA would update its TLD information pages (1, 2), but you will be "shocked, shocked!" to hear that compliance is spotty. In parallel to this official process the two client programs' authors attempt to track TLD changes, themselves. Sometimes, one of the two Linux WHOIS clients will reflect (in its auto-selection of the correct WHOIS server for a given TLD, or its claim that none exists) better information than IANA has. Sometimes, IANA has better data (and, if the system really worked, it would have the latest and best — but doesn't). More often than not, the best data are on relevant Wikipedia pages (1, 2, 3, 4). Some of the linked subpages are really entertaining: If your sense of humour is as warped as mine, check out the reasons why ".vrsn-end-of-zone-marker-dummy-record.root" is a valid TLD, and note the reasons why, in 2007, .arpa is still a robust TLD with six active subdomains — and, by the way, a useful WHOIS server.
The biggest reason Ben and I have so far favoured the jwhois client is that its internal knowledge about which WHOIS server to use for particular TLDs and subdomains is highly configurable via configuration file /etc/jwhois.conf (but beware of the mixed blessing of results caching). Whereas, the other WHOIS client is not. However, in the middle with wrestling with both clients, seeking to give domain-check the broadest possible TLD coverage, Ben found it prudent to hack domain-check's parsing code that handles its (optional) file listing which domains to check, to support an optional two-column format: domain, and what WHOIS server hostname to use for that domain. To help users, I've constructed a prototype domains file, showing a test-case host within each supported TLD (often, the "NIC" = network information centre that runs Internet infrastructure for that country or other TLD authority), plus the currently correct WHOIS host for that TLD. Separately, I am maintaining a separate file of more-verbose notes/information, but the latter is intended solely for interested humans, and isn't readable by domain-check.
Now, I figure most people who deal in domains are following this account without major problems, but a minority of readers may be thinking "What's this about determining expiration data via WHOIS?", and a smaller minority are still stuck on "What's this about domains expiring?" I should explain:
It's a Wacky World, out There
(This is part two of "Where things get interesting".)
One of the reasons I really enjoy travelling to remote and diverse parts of the world, on occasions when I have time and money for it, is that you encounter people living their lives using, quite naturally, radically different basic assumptions, sometimes assumptions differing in subtle but important ways. In return, you're rewarded with the cheerful fact that you and your people will tend to strike other nations as slightly odd and nutty, too — and may even agree. (An American comedian and entertainer named Garrison Keillor and his radio programme "A Prairie Home Companion" finally made me realise, similarly, that my own crowd of Scandinavian-Americans are extremely quirky people — manias for strong coffee and white fish, going nuts on Midsummer Day, mocking self-deprecation, and all.)
Getting back to the subject, exploring WHOIS data can earn you that same shock of unexpected strangeness, right at home. One of my first test cases for the unfolding development of domain-check was .au, i.e., our esteemed friends in Australia. Hmm, I thought, why not check Linux Users of Victoria?
$ whois luv.asn.au | more Domain Name: luv.asn.au Last Modified: Never Updated Registrar ID: R00016-AR Registrar Name: Connect West Status: OK Registrant: Linux Users of Victoria Inc. Registrant ID: None given Eligibility Type: Other Registrant ROID: C0793852-AR Registrant Contact Name: THE MANAGER Registrant Email: Visit whois.ausregistry.com.au for Web based WhoIs Tech ID: C0793854-AR Tech Name: Stuart Young Tech Email: Visit whois.ausregistry.com.au for Web based WhoIs Name Server: black-bean.cyber.com.au Name Server IP: 203.7.155.4 Name Server: ns2.its.monash.edu.au Name Server IP: 130.194.7.99 Name Server: core1.amc.com.au Name Server IP: 203.15.175.32 Name Server: lists.luv.asn.au Name Server IP: 203.123.80.10
Hullo? Where's the expiration data? Turns out, none of .au offers that information via WHOIS. Nor does the public whois information browsable at the indicated Web host. What?
Well, upon inquiry, I was enlightened: It's deemed a privacy issue, and Australians using .au domains presumably suffer fewer domain snatches, and similar abuses. The same appears to be true in .de (Germany) and some others. Presumably, domain owners (as opposed to the general public) can look up their own domains' expiration data via their logged-in individual domain records, in addition, of course (in theory), to getting notification of upcoming expirations. On the downside, TLDs that conceal that data from the public prevent public-spirited neighbours from helping warn domain owners notice upcoming problems, keep people from planning for legitimate opportunities to re-register domains their owners no longer want, etc.
(By the way, if you are really serious about protecting your privacy as a domain holder, .au doesn't really qualify. .to (Kingdom of Tonga) is among the few that do.)
However, it gets stranger: There are particular country-code domains (I won't name names) where expiration data is available, and open to the public, but appears not to matter. That is, you'll find what appears to be a good test case, notice that its expiration date of record is three years ago, and then notice that the domain still works, anyway.
Your mileage may differ, but, for me, that was a culture shock: In my sadly clock-driven, Westernised existence, Internet domain expiration is a real calamity: Your domain's DNS stops working within a day (if not instantly), and you may or may not even be allowed to buy it back (re-register or renew it) at all. If you are, it may involve a ghastly "Sorry, you were asleep at the wheel" surcharge.
Some TLDs, evidently, just aren't like that, so domain-check may address a non-problem for domains in your national TLD. It's up to you to check, I suppose.
My prototype setup of domain-check runs via a weekly cronjob that runs every Sunday night, and e-mails me a notice about which domains, among the roughly 150 I have domain-check monitor, are now within 90 days from expiration, plus a separate one about what domains, if any, have already expired. You might ask, armed with that weekly briefing, what do I do? That brings us to:
The Difficult Part
This might be you: You own a domain you care considerably about, but every year, like clockwork, you put off renewal until the last minute, to "get the most for your money". You pay for only one extra year at a time, not three or four, for the same reason. Maybe you dislike your current registrar (in TLDs like com/org/net where a choice is offered), but never move your domain because that would require sending more money, and you have a vague notion that, some year, you'll move registrars right around the time of your next renewal. Maybe you literally wait until the final day, and panic and shout on the telephone to your registrar if there's a hitch, until your renewal goes through. You're now reading this article and realise I'm about to tell you you're being foolish, but nothing's going to change your mind, let alone your habits.
Why is that foolish behaviour? Because every bit of that attitude greatly increases the risk of accidental expiration. You should, actually, consider moving to a better registrar at any time, and not waiting, because competing registrars almost all credit you for the full time remaining at your prior registrar, upon your domain's arrival. That is, if you have 100 days remaining at Registrar A when you initiate a transfer to Registrar B (and pay the latter for a year of service), you start with 365 + 100 days on your domain's expiration clock. So, you lose nothing at all. The bank interest you save by buying only one year in advance instead of 3-4 is absolutely negligible compared to the painful cost of recovering from accidental expiration (where that is possible), not to mention the transaction cost of swooping in and continually renewing annually to "save money" (let alone the cost of doing that mere days or hours before expiration, as many do).
I might be able to convince you, the reader, that the above syndrome is unwise, but I won't convince your friends or the organisations you care about — whose domains you might want to watch over. Which brings us back to the question: Armed with the knowledge that someone's domain expiration is imminent, what do you do about it?
Several non-technical problems become evident, as one attempts to look after friends' domains — and I really should have realised that the technical challenges of writing and debugging domain-check would be the iceberg's tip, but didn't:
- The dangerous lure of last-minute renewal syndrome is widespread. Even after reaching the right person, saying "Dude, your domain's three days from expiration", and getting acknowledgement, the owner may not care. Oh well, it's his/her funeral. At least you tried.
- Increasingly among registrars, there's something called "auto-renew" on domain registrations: The registrar accepts credit-card data from the owner, and then at some number of months, weeks, or days from expiration (sometimes, three or more scheduled attempts) tries to charge the credit card for a year's fee, without the need for an explicit renewal action from the owner. Registrars like the customer-retention effect of this setup, so it is often the default arrangement (e.g., at Dotster). At least one major registrar reportedly carries out the auto-renew attempt only on a covered domain's day of expiration, and never before. Problem: From your perspective as a sympathetic watcher of someone else's domain, you have no idea if an auto-renew is likely or not, so you cannot easily distinguish a domain on auto-renew from one at risk.
- Most perniciously, there is the syndrome of broken communication, which I'll detail below.
- Last, don't forget, not all domain expirations are unintended.
Imagine a Linux user group, or a science fiction fan association that puts on an annual convention, or some other similar group that relies on an Internet domain. You're trying to get their attention to an upcoming expiration. Domain matters are probably delegated to someone technical who's believed to be handling them. The people who run the group generally are most often other people, who may not understand domain matters at all, and may assume, if you ask them about it, that you must be referring to the Web site, will forward your mail to the HTML guy / webmaster / hosting company / listadmin, and will never realise their category error.
The domain guy may be gone from e-mail for a month. He/she might have believed the responsibility was taken over by somebody else. The contact e-mail addresses shown in WHOIS for the domain may be wrong or outdated, or just unmonitored. Your warning e-mails might be mistaken as spam or a sales solicitation (strangers showing concern seems outlandish to many), and blackholed or ignored. Or everyone in the group might be assuming someone else is taking care of it. Or maybe their mail server is misconfigured or otherwise mishandling or misrouting some or all incoming mail about the domain.
Ultimately, this isn't your problem — sufficiently hapless organisations' negligence will cause them loss despite even heroic efforts to help, and that can't be helped — but it's nice to know the most common failure modes.
If you see a domain's days remaining rapidly approaching zero, and nothing's happening, one of four explanations logically applies:
- It's set to auto-renew, on or near the last day, "but thanks for letting us know".
- The owner specifically intends to let it expire, actually, "but thanks for caring".
- The owner's stuck in a deliberate, conscious case of last-minute syndrome.
- The registrar's various reminder attempts (if any) have failed completely or failed to reach the right, motivated party, for some reason as yet undetermined, and everyone's otherwise asleep at the wheel. (Registrars make mistakes. Also, domain contact information has been known to get corrupted, changed in error, hijacked / misappropriated, etc.)
As the concerned outsider, your main worry is the last scenario, which is the classic domain-loss one — which is relevant to the current question, of what you do with your knowledge of the impending expiration. The naive answer is: "Look in WHOIS, find the listed domain contacts, send them "Dude, you need to renew your domain" e-mail, check that domain off your list, and pat yourself on the back for being a good neighbour.
That's naive because, odds are, that's exactly what the registrar did, and it didn't work. Thus, you may want to be a little creative about whom to contact, and how. E.g., look on the Web site for maintained information about who currently runs the group. Bear in mind that he/she/they may not, initially, know what you're talking about (e.g., fob you off on the webmaster). Ask politely that someone in charge send you wording like "Thanks, we know about it", mention that you'll cease pestering them when that happens, and keep following up at intervals.
Be really, really diplomatic. Your queries may, themselves, cause a political kerfuffle within domain-owning groups, and cause considerable unintended irritation. People will get bothered, often despite being the wrong people to bother (e.g., the webmaster), and may get cranky. A harassed domain-admin may write back and say "It's on auto-renew, jerk." Don't be offended. Stress that you didn't know, and merely want to help them avert mishaps. From time to time, you just might get lucky and hear "Thank you."
Anyway...
I should stress that my cronjob was the result of only a few minutes' work, shortly before penning the initial draft of this article. It wouldn't be difficult to write a less-feeble shell script to do slightly more useful notifications, e.g., tailored e-mail warning texts at the 90, 60, and 30-day marks, with each being sent to groups of people appropriate to each domain, rather than all notifications being sent just to one person for all domains monitored.
However, as is so often the case with system administration, perfectionism is not your friend: Waiting around to do this job right had already caused me to delay some months from doing even this much, while I pondered the problem — and in the meantime a volunteer group I know (but will not name here) was obliged to spend about US $500 to ransom its domain back. Ouch and damn.
Moral: Do the 80% solution, the one that avoids disaster, today. Don't be proud.
And don't be a single point of failure, either. I'm encouraging all my 'Nix-using (including, without prejudice, MacOS X) friends to run this thing, and help out with redundant, overlapping checks, too.
How about you? The domain you save from disaster probably won't be your own, but it could easily be one you care about dearly, or that a friend cherishes.
Alternatively, you could think of this as your best shot at ruining a domain squatter's day. Either way, it's awfully good news for decent folk.
[1] 2011 clarification: Expiration is not, however, an instant disaster. I apologise if I seem to have so implied. In the first 40 days, the domain can still be renewed by just paying the normal renewal. Some registrars will accept renewal money from anyone; others won't. For the following 30 days ('redemption period' in the public whois records), registrars increase substantially the fees required for renewal. For the last 5 days ('locked' status), the domain is being prepared for deletion. At 75 days out from expiration, it lapses back to unregistered, and can be grabbed by others. For more, see Mike Davidson's explanation.
Rick is [2010: was] a member of The Answer Gang.
His first computer was his dad's slide rule, followed by visitor access
to a card-walloping IBM mainframe at Stanford (1969). A glutton for
punishment, he then moved on (during high school, 1970s) to early HP
timeshared systems, People's Computer Company's PDP8s, and various
of those they'll-never-fly-Orville microcomputers at the storied
Homebrew Computer Club -- then more Big Blue computing horrors at
college alleviated by bits of primeval BSD during UC Berkeley summer
sessions, and so on. He's thus better qualified than most, to know just
how much better off we are now.
When not playing Silicon Valley dot-com roulette, he enjoys
long-distance bicycling, helping run science fiction conventions, and
concentrating on becoming an uncarved block.
Rick has run freely-redistributable Unixen since 1992, having been roped
in by first 386BSD, then Linux. Having found that either one
sucked less, he blew
away his last non-Unix box (OS/2 Warp) in 1996. He specialises in clue
acquisition and delivery (documentation & training), system
administration, security, WAN/LAN design and administration, and
support. He helped plan the LINC Expo (which evolved into the first
LinuxWorld Conference and Expo, in San Jose), Windows Refund Day, and
several other rabble-rousing Linux community events in the San Francisco
Bay Area. He's written and edited for IDG/LinuxWorld, SSC, and the
USENIX Association; and spoken at LinuxWorld Conference and Expo and
numerous user groups.