Home | Mail | Resume | Spyware and Adware
Yours truly

I was quoted in a September 19, 2004 New York Times article, Barbarians at the Digital Gate by Tim O'Brien and Saul Hansell, on the topic of spyware and adware. The article is a good introduction to the topic, but focuses on largely on the business aspects of the issue. Here I'm detailing some of the technical means for addressing the problem, as well as observations on the nature and origin of adware and related malware.

As with many of the pieces on this site, I periodically revisit items to update and revise them as appropriate. First published September 20, 2004.

July 17, 2005 Update. The New York Times has carried several additional stories along this same vein, some better than others. I've addressed these at the end of the essay.

Spyware, Adware, Windows, GNU/Linux, and Software Culture

The Times piece involved a couple of phone interviews with Tim O'Brien, as well as a few email exchanges. My views were fairly presented, though as is usually the case, an awful lot of conversation got boiled down to a very few lines in print, so I'm taking the opportunity to expand on several of these.

For starters, I'll note that I run GNU/Linux on my own personal desktop, both at home and at work, and that the problems delineated in the article simply don't exist for me there. While I strongly favor Linux, I consider my bias grounded in experience and reality. I've certainly had years of experience with both types of systems.

I run herd over a small posse of legacy MS Windows systems at work, a youth center in Napa, CA. I'm also called on periodically to do maintenance on PCs used by adult staff in various businesses. I have to say the the whole issue of spyware, adware, viruses, worms, and other annoyances (generally: malware) really opened my eyes to the problems MS Windows users face.

Among topics the article didn't address for reasons of space and focus:

There were also a few general observations I had on the spyware / adware / malware issue. Briefly (and there's more at depth later on most of these points):

Keeping things clean

I've found that it is possible, at least with luck and a lot of work, to keep Microsoft systems clean.

Among the most effective, of course, is to install GNU/Linux on the box. Very simply: no Linux system I've used or am aware of has had any level of adware infestation. And were they to have a problem, rooting it out would be largely trivial. I'm strongly recommending Ubuntu GNU/Linux, a Debian-based distribution, to GNU/Linux newcomers.

Assuming you're not prepared to go to that level, here's what I've done at work, where my hands are tied (grants, boss, etc.). I don't believe you can get robust results with DOS-based systems: Win3x/95/98/ME. Especially WinME, which is probably the worst of a long line of bad OS products Microsoft has produced.

As I said in extended comments to Mr. O'Brien (with whom I spoke & corresponded), I've got an advantage over many systems administrators in that I'm running a lab for kids: I am the word of God, and I can simply decree that specific programs and/or functionality aren't available. I also run a couple of GNU/Linux servers in the lab which provide certain functionality, some of which is used in keeping things sane. This includes Samba, Apache, Dansguardian, Squid, and numerous utilities. I've also got Cygwin installed on the desktop systems, which simplifies and extends administrative management considerably. All of these tools are FSF Free Software (often called Open Source), meaning several things, but mostly: you can install and use them for free, and modify them if you choose to do so.

Uninstall MS Outlook and Outlook Express

These are a pair of virus-propagation utilities which offer a largely unsatisfactory level of email functionality. Given that the kids don't (currently) have email, and that I've got other options for providing 'em with same if we should choose to do so, simply eliminate the problem by removing it.

This, incidentally, is a good example of security via minimum exposure. If you don't need to offer specific functionality, then don't. Unfortunately it means that you have to give slightly more thought to your system configuration than a default, kitchen-sink installation generally means.

If you must provide email functionality, Mozilla (more below) offers a "Thunderbird" client, and Eudora is a popular small-organization choice (advertising-supported). Both, incidentally, use open and transportable mailbox formats making your future migration to GNU/Linux far easier. Mozilla has a utility for migrating your proprietary MS Outlook format PST (mailbox) files.

Install Mozilla Firefox (or another non-MSIE browser)

Mozilla Firefox, "Rediscover the web", as the slogan says.

Opera is another popular choice, though in its free incarnation it has certain adware characteristics (similar to Eudora above).

Installing Firefox addresses a large host of evils in one swell foop, including:

Mozilla is about taking back control of the web. Very nice, that.

Uninstall other dodgy software

There's a whole mess of software on your MS Windows computer not because it's of any particular use to you, or because you asked for it, but because of marketing arrangements between your hardware or OS vendor and other companies. The mess of Internet service provider icons, for example.

Most of these are relatively harmless. I did find one program, Viewpoint, apparently provided by Yahoo, wanted to upgrade, and was suddenly talking about putting search bars and buttons everywhere. I decided that that particular collection of bits was no longer welcome and uninstalled it. Possibly an overreaction, but any additional icon on a desktop means another twenty minutes of answering questions from kids ("What does this do? This wasn't here yesterday?"), even if it doesn't do anything particularly annoying. Prune ruthlessly. And a note to vendors: stay out of our faces, you're going to have a much better survival profile. When in doubt, Google for the software by title, adding "spyware" or "adware", to find others' discussions. In many cases, the distinction between useful software and malware is grey.

Block MSIE web access

There are a number of methods to prevent users from accessing Microsoft Internet Explorer. Unfortunately, few of them work effectively. The program is too thoroughly entwined in the workings of legacy MS Windows and various Microsoft products to make removing a few icons a fix.

I'm addressing the full method in a forthcoming technical article, but one relatively effective trick is to direct all MSIE traffic to a proxy, except for a small set of hand-picked sites which must get through. For example, windowsupdate.microsoft.com. Doing this on multiple workstations for multiple users is a headache, but can be accomplished with scripting tools, your domain login's "LOGON.BAT" file, and in my case, an Apache webserver given a virtual host who's sole purpose in life is to tell people not to use MSIE.

This breaks some stuff, you have to decide whether you value a few conveniences over a generally working system. There are sites which only work under MSIE (that's their problem, not mine, is my response). There are also specific tool under MS Windows which require MSIE, notably Windows Media Player. Some third-party tools such as anti-virus software will get caught by your proxy. Monitoring my webserver's logs is useful at identifying any such issues, and if necessary, adding a site to the pass-through list.

Using Web-Filtering Software

In my case, it's Dansguardian. As mentioned above, it's FSF Free Software, and comes with its own highly tuned filters. You'll need to adjust them to your needs, slightly, which mostly means adding sites to exception and/or ban lists, done by editing a set of well-documented, easily understood, text files. Blocking is based on several characteristics, including specific domains, keywords, content-type, and extensions.

You need to keep an eye on what's passing or not passing through the filters, which means this is a bit of an ongoing task. For the most part, done on an as-needed basis. After an initial week or so of adjustments, I find I rarely need to touch things more than once a month or so.

For the really bad guys: firewall-level blocking

The NY Times article doesn't mention one particular homepage hijacking site, (and don't click that link yet) http://www.domainsponsor.com/. This is an organization which apparently registers a large number of "typo" URLs -- domain names close to, but not quite. Kids, as you might guess, tend to have middlin' to po' typing and spelling skills, so "disny.com" and similar expressions show up. When this happens, your browser is redirected to the page above. And if you are foolish enough to surf with MSIE, your homepage (the page you see when first starting your browser) is reset to one of DomainSponsor's choosing. If you still want to follow the link after all that, go ahead.

Their own webpage (and WHOIS record) indicates DomainSponsor is owned by Oversee.net. Netblock NET-65-235-246-0-1, CIDR 64.235.246.0/24, ASN 25973 (Mzima Networks, Inc.).

My evolving attitude on 'Net citizenship is rapidly approaching a "take no prisoners" status, and is based on the principle of network hygiene: bad behavior (viruses, malware, phishing, attacks, zombies) reflect bad network management and oversight practices, something I've come to appreciate in my ongoing antispam activities. If a site demonstrates that it's a sufficiently bad neighbor that it's going to do things like hijack browsers' home pages, regardless of how poorly designed the browser is, that that particular neck of the 'Net has no business whatsoever swapping bits with my network. While a Web filter can work with domain names or content, what you want is a firewall in which you can explicitly block some or all traffic from a single Internet (or IP) address -- or an arbitrarily large range of them. Locally, this particular source of malice is blocked by several redundant methods.

For the truly dedicated, there are extensive lists of IP space associated with organizations or countries from which some feel there is more harm than good in allowing traffic through. For the malware proponents: beware that the Net may be comprised of small players, but there are many of them, powerful in aggregate, and with long memories. As the recent case of Savvis shows, the effect can be ultimately persuasive.

Anti-virus software.

It's not an option. And it's not enough to install it, you have to keep it up-to-date -- at one mail service provider I worked for, this meant updating every six hours (via an automated script). And you need to run it.

My current choice has become something of a PITA following the latest upgrade to the AV software itself, as its auto-upgrade feature isn't working. Which adds yet another item to the list of things I've got to get fixed or updated to feel moderately comfortable about the state of my systems.

The big names are Sophos, Command Software (now Authentium), Symantec, and Trend Micro, in no particular order and with copious omissions, I'm sure. There's also an FSF Free Software alternative, ClamAV, worthy of note.

Oh, and a request: if you install AV software on your mail system, turn off the notification feature. Anti-virus software itself is a nontrivial contributor to the spam problem. The messages are all-too-often misdirected. Really, it's not your problem, here.

Anti-adware/spyware software.

I'm using Ad-Aware from LavaSoft, with largely good results if somewhat mixed operational experiences. The free version of the software is highly interactive, and it's literally a ninety-step process to get all ten systems updated. Lately, downloads and scans have been mysteriously hanging, as I commented to Tim O'Brien during one phone interview. There are other products, I'd recommend installing at least one.

Be alert for "rogue" anti-adware / anti-spyware programs. Sad but true: there are programs advertised as removing spyware/adware which do anything but. Spyware Warrior's Rogue/Suspect Anti-Spyware Products and Web Sites page is strongly recommended. Note that such products may be advertised by Google -- research any such software before you install it. I am not responsible for advertising content on this page.

Coming from the GNU/Linux side of the house, one major gripe against all of the products is the reluctance with which they support automation or silent background operation. Instead, the products launch at login time (why not scan periodically or as-needed?), display splash-screens or tray icons, and often allow non-administrative users to disable or close them. From a systems management perspective: a nightmare.

Eternal vigilance.

Keeping your systems clean is an ongoing chore. Updates need to be downloaded, logs need to be read, users need to be monitored (having them cancel in-process scans is a major factor). One frustration, of course, is that log-ons, already slow for domain users, become slower still as your arsenal of system defenses swing slowly into action. Users are understandably frustrated by this and want to have things happen faster, and will close down what they see as "things in the way".

When you do find a problem (or worse: a suspected problem), you've got another hassle on your hands: trying to sort out the good, the bad, and the ugly. Default tools for getting systems information on MS Windows systems are primitive at best, often unhelpful, and vary widely across various OS products, and even among releases of the same product. In particular, getting a task of running processes, identifying how they were run, and finding out which are or are not malevolent, is a nontrivial task. Even once you've got a list, sorting out the mess is a chore.

The Task Manager is the usual first course of action, but it's a poor tool for the job: it provides little information, you can't print the output, and you can't filter to processes of interest. The site HijackThis at SpyChecker is useful in that it lists many people's process list dumps, often with analysis. While you can't always find out what's running, you can usually get close. Often simply entering an executable's name into Google (say: example.exe), will give useful information. I've found that there are malicious programs with innocent looking names and innocent programs with malicious-looking ones, it's difficult to be sure. Under WinXP, there's a 'TASKLIST.EXE' program which lists processes similarly to a Linux 'ps' command.

You want to check both your Startup folder(s) (if you have multiple users) and the "Run" Windows Registry key, both of which specify programs to be run at startup. Anything running out of temporary folders is immediately suspect.

With the right tools, you can run a portscan of your system to see how it's talking on your network. GNU/Linux offers a great tool for this, 'nmap', which is available on many "bootable" Linux distributions. These are small (or not so small) collections of GNU/Linux utilities that run from a CDROM, floppy disk, USB pen drive, or other removable media, and don't require installation on your hard drive. LNX-BBC and Knoppix are among the two best known, the former being technically oriented and the latter a full end-user desktop on CD ROM. But that's another essay.

Experiences

So with all of that, how effectively have we cleaned things up? Remember that this is a lab with over 300 member accounts. Kids between the ages of 6 and 18, many of them not particularly computer literate, and roaming pretty widely over the Web. In six months, I've seen a grand total of one browser homepage hijack (fixed), and one virus infection (detected and neutralized). Pretty good record.

How effectively are we locked down?

Within the computer lab, I have logs of all scans performed since late March, 2004. As I write, there are a total of 2609 scans on all ten lab systems. Even with everything locked down, we're still finding a new object about every three scans, with 961 objects found total. Again, this is the locked-down environment.

It's important to note that AdAware looks for numerous "signatures" of evil stuff, and, well, there are different classes of evil evildoers of evil. Most of the results are Web "cookies", or tracking data placed on the system by a remote website. Other than presenting a privacy concern, these don't directly affect system performance or represent malicious software. Occasionally a Registry key is detected (another bit of system information). I've had no known processes detected.

Are there any adults in the house?

If so, watch out.

When I've run scans on systems used by adults in business roles, what I'm typically finding is 350 - 420 or more items on a system when performing a first time scan (Tim and I discussed this statistic several times, I'm surprised it didn't make it into the article). While I noted in the "eternal vigilance" paragraph above that many ongoing detected objects are relatively minor, this is generally not the case for first-time scans of in-use systems. I'm finding literally hundreds of cookies and Registry entries, and running processes or programs on disk associated with known spyware and adware. Often system performance is very slow as a result of adware and spyware (not helped by older or less-than-optimal configurations in the first place).

One scan isn't enough. You have to keep after systems, and I generally find repeat scans turn up scores of new or repeat objects, many more harmful than simple cookies.

Rather more troubling are front-office systems whose browser homepages are set to grossly inappropriate sites (encountered at multiple worksites), systems which once scanned and cleaned fail to boot or function properly, and financial systems running multiple spyware and virus processes -- on repeated attempts to clean them. I was shocked a year or so back on scanning my parent's home PC to find that it had literally hundreds of instances of one of the then-prevalent MS Windows viruses on its hard drive, and who knows how many actively running. That was, as the saying goes, a little close to home.

Oh, and the results are pretty much independent of any particular Microsoft OS. That is, while Microsoft touts WinXP as the greatest thing since sliced bread (maybe they should exit software and enter the baking business...), I'm seeing XP systems get just as trashed as Win2K, Win98, or Win95 systems. On the 'Net, they're pretty much equals in inability to cope. In large part this is because of how legacy MS Windows systems are deployed. Even where true user-level security exists (NT/2K/XP), users are often granted administrator privileges, generally because it's easier to get things to work right that way. Some software flatly requires this, including some painfully popular small business financial software.

As I'm quoted in the article: you'd expect to be able to use computers in a safe and sane manner, but you can't, out of the box. You need a lot of expertise, a lot of time, and (ahem, comma) a healthy dose of paranoia.

As if to prove my point, three days after the article runs, I get a call "Hey, I think I have a problem". It's a WinME box. AdAware turns up 452 objects (including twenty or so programs and directories), an antivirus scan turns up over 1350 infestations, mostly Netsky.C, but a few other oddments thrown in for good measure. The system is now mostly clean, though one adware app showed up after reinstalling itself this morning, and another refuses to uninstall. But it is now far less reliable than it was previously, and freezes if set to run Ad-Aware at boot.

And again: in the kids' lab, I'm in the rare position of being able to call the shots. I've got an hour or so's free-and-clear time before the day starts in which I can (and do) perform maintenance tasks. If I deem the risk of allowing specific activities, software, or access too great, I can simply say "no". Few admins in a general work environment have this luxury, and it's a distinction that makes maintaining MS Windows systems (a very distasteful activity for me) palatable. I've repeatedly turned down the opportunity to take on the task elsewhere.

Some cultural observations

The short version of this section is: adware / spyware / malware is the logical outcome of the competitive, proprietary software market of the past several decades. The system has promoted cut-throat competition, and by gum, it's got it. This is in marked contrast to a more cooperative model adopted elsewhere. The rest being the longer story.

As I've noted several times, I use GNU/Linux largely, and while I'm conversant with more common operating systems, I really don't deal with them much day-to-day, and haven't for most of the past seven years.

As such, walking into the current world of MS Windows was something of an eye-opener -- as in, why do people stand for this level of pain? Sad truth is: mainstream computers have been awful for a long time; much as I love GNU/Linux, its unadulterated form may not be what the Public's ready for yet; and.... somehow I get the sense there's a bunch of Mac fanatics snickering in the corner (as usual). But they're really running Unix anyway. There's also the minor point that for an extant legacy MS Windows shop to switch to MacOS X entails a hardware change. There are a number of migration paths to GNU/Linux which allow both using current hardware and allowing for a gradual (and for most of the path, reversible) switch-over.

The first observation is simple the scale of the problem. It's a constant ongoing major concern for legacy MS Windows. What I fear most at the lab is that one or more systems starts going wonky, at which point it's more-or-less a death spiral. Diagnosing and recovering systems is a time-and-labor intensive process. I'd really rather not go there. Just the fear is like working with a constant, mid-grade headache.

On the GNU/Linux side, the problem simply doesn't exist. Full stop.

Sure, there are other hassles. Some of them significant. But the risk of random third-party software installing itself covertly on my system and taking it over is well down the threats-and-hassles list. In fact, the whole environment's a lot saner, calmer, cooler, and collecteder. Or something like that.

Cooperation vs. competition

Then there's the difference between software distribution environments which are intrinsically competitive in nature, and those which are cooperative.

One view is that AdWare is the logical extension of the proprietary consumer software market. Margins are razor thin (or inverted). Competition is fierce. Channels are tightly controlled, largely at the OS (one vendor) and HW (three major vendors) levels. With the "desktop" GUI shell metaphor, competition for icons on the desktop, and commensurate eyeballs, is brutal. The rise in significance of the browser makes it an attractive target, expressed as toolbars, icons, homepages, bookmarks, and the like. It's a dog-eat-dog race for getting in front of the public user, um, "consumer".

And if you can't get in fairly....

It's a maxim of incentive systems that you get the behavior you reward, and currently, that's aiming for the desktop or user's eyeballs, no matter what. And marketing execs (in this narrow scope at least) are rational animals.

There's another slight problem.

First: Debian.

Debian

This is a version, or "distribution", of the GNU/Linux operating system. It differs from most of the other major GNU/Linux versions -- Red Hat, SuSE, Mandrake -- in that it is a noncommercial, community-based, non-profit project. Which sounds all hippy-crunchy and all until you realize that it's also among the oldest distributions (organized in 1993), and among the highest ranking in popularity and installation (notoriously hard to estimate, but in the top two or three, with little space separating contenders). It also has a rock-solid technical reputation, and in several user groups I'm active with is the hands-down favorite choice. But enough boosterism.

Debian has several interesting characteristics, among them a constitution, a social contract, and a rather complex and contentious document called "debian-policy". It also has a packaging system ("dpkg", comprised of "Debs") and some 17,000+ software packages included in its archives, all FSF Free Software, which can be downloaded, free of charge, off the Net (or acquired by other means, but we're going to keep it simple. Honest).

You're probably wondering what this has to do with adware. Fair enough.

Well, for starters, the Debian Social Contract states as one of its five points, "Our priorities are our users and free software". The rather longer Debian Policy Manual codifies this (and other) directives as to specific behavior which may or may not be permitted in the act of installing or removing software from the system. It sets limits on the behavior of software "packages", and their "package maintainers".

To be sure: this applies only to software which is included within the Debian Project itself, and there is much software which isn't. However, there are some distinct advantages to inclusion:

The result is that there is more software available, within the distribution itself, for Debian, than for any other version of GNU/Linux. And there are good reasons for developers, so long as their code is free enough, to cooperate. Actually, there's a large base of code that isn't "free" (in Debian's terms) that is included in a sort of distaff section of the distribution called, appropriately enough, "non-free". But again, that's another essay.

For the quasi-technical readers I haven't lost yet, what Debian's packaging system provides, say, relative to the Windows Installer, is like the difference between an old style "cooperative multitasking", non-memory-protected operating system (OS) -- think MS Windows 95/98/ME -- and a "preemptive multitasking" operating system -- think NT, 2K, XP, or, um, GNU/Linux.

You're nodding your head, but if you don't really get it (don't worry, it's just the two of us here....), it's basically this. While "cooperative" sounds nicer than "preemptive", what it really means is that the OS doesn't have real control over what applications can do. In particular, what "resources" they can use, and especially, how processor time is allocated. In cooperative multitasking, programs are supposed to have breakpoints at which they give up control to another application to run. With preemptive multitasking, the program has no say in the matter: the operating system kernel itself allocates "timeslices", and can cut off a process, regardless of what that process wants.

Memory protection refers to what portions of a computer's memory programs use. The problem being that when two programs both try to change the same part of a computer's memory, things tend to go downhill fast. In a non-memory-protected environment, the OS is effectively saying "please don't do that", but doesn't have any effective way of enforcing the request.

By contrast, a memory-protected OS puts each each process in an iron cage with set limits. If the process tries to overstep its bounds, it's killed (the error, by the way, is called a "segmentation fault" -- memory being what's segmented, and the fault being not adhering to boundaries).

Both preemptive multitasking and memory protection give the OS more control over the system as a whole, and protect against negative effects of both poorly written, and intentionally malicious software. The result is a more stable, predictable, operating environment.

In the same way, a MS Windows installation is a set of polite expectations: software will go under C:\Program Files, it will create a set of Registry keys in appropriate locations, it will add itself to the Start menu, it may or may not create a mass of desktop icons (and these might or might not go under All Users or just the one user installing the app), and a slew of file associations might or might not change.... And when you removed the software, would it please clean up after itself (but not break the system in the process) because its mother doesn't work here. But there is no mechanism for enforcing these requests.

Under Debian, each of these relevant operations is governed by Policy.

Debian does for software installation what preemptive multitasking does for program execution.

Namely: it sets bounds on what's permissible. While the enforcement mechanism is a lot messier, the result is largely the same. If you overstep bounds, you're kicked out. The result for the user, after a few years of initial confusion in grokking the system, is something that's far easier to maintain, particularly as software is added, removed, and updated.

The mechanism isn't 100%, and the analogy isn't strictly parallel. Debian's policy has no influence over third-party applications, and the limitations aren't quite the same as memory protection and timeslice allocation. What is happening, though, is that limits are being imposed and enforced, in the name of system stability.

This also means that, within the context of the Debian package archive, activity manifested by MS Windows spyware is pretty much guaranteed not to happen. If the actions weren't prohibited by existing policy, odds are good that the document would be amended (Debian developers being pragmatic types) to disallow the behavior. This also leads to a positive feedback cycle: developers like the archive because it gives them a useful distribution channel, while users like it because it gives them software they can trust. Many Debian users will only go outside "the archive" for third party software with a large measure of reluctance. Experience shows that this software is more likely to be difficult to install and configure, doesn't behave properly or as expected, will have bugs, and (most importantly) is likely to be difficult to upgrade. My own system has only one non-Debian package on it.

There are even tools to take several other popular "package formats" (software distribution formats) and convert them to Debian form. Whatever it may reflect on Debian, this tool is called "alien".

While it would certainly be possible to create malware outside of the Debian archives which would be installable on systems, the prospects seem somewhat limiting. Not full proof against, but again, with the quantity and quality of software available within the archives, as well as the assurances offered by same, a would-be adware marketer's prospects would be limited. Numerous other factors ranging from Unix / Linux culture to default OS and user security and permissions levels make the prospect of "accidental" installations far lower. And any software within the archive which did allow such installations to happen without explicit approval would be considered gravely broken.

I'd also like to point out that while I've criticized aspects of competition, it's not a bad thing. Quite the contrary. In the GNU/Linux world, software is fiercely competitive -- in terms of features, quality, performance, and the like. Bad solutions, or even just "not as good as" solutions, fail. People stop using them. They switch to alternatives. The competition, however, isn't one for eyeballs and marketing contracts, it's sort of a lower-level, more Darwinian "fitness of code" model. A different arena, not a lack of competition at all. As with several other topics raised here, another essay in itself, though the topic's been discussed elsewhere as well.

So why not bring this model to the legacy MS Windows world?

Good question. It boils down to....

Trust

Debian's role in the GNU/Linux world is rather difficult to map onto the competitive, proprietary, MS Windows space.

Debian doesn't develop "the OS", that's the role of various Linux kernel developers from Linus on down. Nor does it develop libraries ("APIs" in MS Windows context), or end-user applications. There is some development of tools used to actually manage the system, thinks like "dpkg", "apt-get", "aptitude", and "synaptic", as well as utilities such as the aforementioned "alien". There's a documentation project. There are discussion lists, and the bug tracking system. In some regards, Debian is closer to a market or a bazaar than it is an operating system or software vendor.

Mostly, though, what Debian does is facilitate integration of independent pieces of code. There are several thousand "Debian developers", or 'dd's, each of whom manages one or more packages. These are assembled into Debian's packaging format, compiled for various architectures, tested, and uploaded to the distribution mirrors.

As a result, Debian has relatively little interest in any specific package. There's a small set which are afforded "core" or "base" status, which are required by the system. All others are just gravy.

One consequence is that the original (aka "upstream") developers of software pretty much trust Debian to do what's right. The project is notoriously open. Discussions on the user and developer mailing lists can be quite heated. There's one particularly warm corner of Hell on Earth called "debian-legal", which should serve as a caution to those who'd introduce computer geeks to law. And generally, Debian doesn't have a horse in a given race.

Compare this situation to, say, Microsoft.

Suppose that, say tomorrow, Microsoft were to announce it was going to implement a centralized repository of all MS Windows software, that it was going to create a set of rules that software had to follow in installation and removal.

There would be an industry-wide storm, Congressional inquiries, bad filks, DoJ investigations, and stern pronouncements from the EU. If the tech industry has learned one lesson over the years, it's that you don't trust Microsoft. Particularly when they've got a horse in the race, and moreso when they don't, because it means you haven't figured out which is theirs yet.

There are other problems with the concept as well, of course. Debian has developed its systems and policies over the course of a decade. Neither technical nor social structures fall into place overnight, and the current system is the result of countless small battles. It's a process that cannot be accelerated. The system is also based on a software architecture that lends itself highly to modularity and independence of components, almost the exact opposite of the MS Windows world. Which itself is the result of competitive and marketing pressures. I could go on, but the key is this:

Microsoft Windows, and Debian GNU/Linux, are independently the outcomes of development and competitive models which are different in fundamental ways, and which reflect those differences at every level of their existence. They simply cannot be mapped or transferred onto one another.

This isn't a problem with either a technical or short-term fix.

System transparency

Another issue worth noting is the level of transparency offered by GNU/Linux systems vs. MS Windows systems. Simply: it's far easier to figure out what's running, how, and where the executable came from on Linux than on Windows. Microsoft systems are in general amazingly reluctant to divulge information about themselves, and when it is possible, require some arcane ritual of menu poking to do so. Command-line interfaces are a far more direct route to inner workings, and allow far more powerful and useful manipulation of data.

Simple little things like the concept of a process tree, result of a very early decision in Unix called a fork(), make tracing the sequence of what commands invoke what far, far easier. Toss in a package database which knows the location of every file installed by every package on the hard drive, a 'locate' database which can track down pretty much everything else, uniform system scheduling and initialization scripts (cron and init.d), system state presented in file format under /proc and /sys directories (folders for the surviving MS Windows readers) and you have claim I made at top of this article: it's largely trivial to figure out who started what from where, and often when, why and how. And nonconformant processes stand out clearly.

Personal computing's future

I like working with computers because they're fun, powerful, interesting, effective, and in general, they help me get things done.

The problem is that there are ever larger amounts of pain that have to be incurred in reaching this stage. And that level of pain, on a number of fronts ranging from spam to adware to viruses to security issues to personal data theft, is reaching a point where the propsects for the average Joe or Jane to run a computer safely, sanely, and useful, are falling rapidly. I'll be the first to grant that I enjoy tinkering with my systems. But I also like getting out on a Saturday. I like tinkering to improve things, not just running as fast as I can, Red Queen style, to stay in one place. Stripped of my normal spam and mail processing systems for several weeks this summer, I got a small, but really bad taste of what the average user is up against going online right now, and it really sucks. You begin to wonder why people would put up with this, and in many cases, they don't: they avoid joining mailing lists, web discussions, and other interactions with people, out of fear for spam, viruses, and other online vermin. This isn't good.

The question is whether it's possible to recover some or all of this opportunity within the current dominant paradigm, or if a more fundamental change is needed.

Mac OS X

While my own use of Mac OS X is limited, its increasing popularity means several friends are being exposed to it. Rick Moen makes the following notes, specific to this essay's cultural observations:

I'm of necessity a part-time OSX admin and user: Not only does my G3 iBook have OS X 10.3.x (mostly unused; prefer Ubuntu/Debian), but also my mother is now the proud owner/user of a new G4 iBook, for which I was obliged to do a fair amount of software installation, to prepare it for her use.

As I've been time-constrained every time I've worked on Mom's iBook, I unfortunately didn't take good notes, but am reasonably certain of one thing I observed:

In the process of installing several third-party apps into OSX (Eudora, Firefox, Epson scanner-support software), I was not always presented by a sudo-driven request like "I need your permission as a member of the wheel group to do this" [actual prompt paraphrased and moved up-tech to improve its accuracy]. Thus, though I'd have to re-visit her house to verify this, I'm pretty sure that some OSX software eschews the multiuser framework and installs as user-owned files within the user's home directory (e.g., "/Users/Faye Dalton/") or elsewhere.

I get the strong impression that both Apple and the various VARs tend to be extremely lax with security standards (ownership / permssions) concerning files with which users will have direct contact, to better accomodate Mac users' infamously low tolerance for frustration and for any need to know technical details. Thus, the vital distinction between system and user files tends to be blurry, in practice.

This is going to bite them, at some point. Of course, being Mac people, they probably won't notice the bite marks, except perhaps to decry their lack of anti-aliasing.

Ironies

As a good friend says, dollar for dollar, still your best entertainment value....

Claria

Among the biggest, especially after the whole spiel above about cooperation and competition, is that Claria, f/k/a Gator, runs the Debian GNU/Linux distribution on its own internal systems, as evidenced by its careers page. This also turns up periodically searching for "Debian" on the San Francisco based Craigslist jobs listings page periodically. Notable as relatively few organizations do use it (though more seem to be climbing on board).

So, while Debian offers its users protections, this is a non-transitive relationship that doesn't extend to those influenced by the users.

In a further ironic twist, rumors emerged in July, 2005 that Microsoft are considering purchasing Claria, and that Microsoft's own spyware detection software downgrades Claria's own spyware threat.

Ease-of-use

Here's the paradox.

"Ease-of-use" features built into legacy MS Windows are resulting in a system which is anything but. In order to use the system with ease, a slew of software's got to be disabled, another slew installed, a third slew updated frequently (and rather inconveniently), reports scrutinized, much behavior consciously avoided, and a significant amount of functionality sacrificed. While the user is free not to do so, the consequence is a system which spirals ever further out of control, with random and undesirable behavior, generally resulting in the need for either major system repair or a clean wipe and fresh start.

This is ease of use?

GNU/Linux, by contrast, imposes a slightly less convenient environment (ever less so over time, and arguably par with legacy MS Windows), delivering a system on which 24/7 constant vigilance against your own system turning against you is not necessary.

I find the latter a much more satisfying experience.

The lesson appears to be that ease-of-use at the expense of security is not a net gain. One applies the security first, and works in the utility features within that construct.

Resources

A few of the resources and follow-on items concerning themes raised in this article.

Ben Edelman at Harvard University researches legal and technical issues concerning adware and spyware. In particular, he's uncovered Investors Supporting Spyware (January 12, 2005), showing the money trail behind those who are attacking your (or your friends', family's, and co-workers') computers. Ben updates his site frequently, it's worth visiting.

July 17, 2005 Update

There have been several articles addressing this topic at the Times. These include Terminating Spyware With Extreme Prejudice (Rachel Dodes, December 30, 2004, 2300 words), largely discussing the travails of backing up personal data, reformatting and rebuilding a legacy MS Windows system's hard drive, for which I can only ask "when was reformatting your hard drive national news material?".

More recently was a largely pathetic article by Matt Richtel and John Markoff, Corrupted PC's Find New Home in the Dumpster (July 17, 2005, 1200 words).

The latter is a gross disservice to Times readers for several reasons:

The article is accurate in one respect: legacy MS Windows is increasingly unsuited to use on the Internet, and typical users are having an increasingly difficult time coping with its shortcomings. As I noted above, being forced to interact with the Net for a few weeks last summer without my usual defenses, I found things rather tough going, and clearly, average users are being discouraged from participating on the Internet. It's going beyond the former issue of the hassle of spam discouraging participation on mailing lists and Usenet discussions, and now is an active fear of simply surfing the Web. This is a very troubling development.

Apologists suggest that it's simply enough to "act smart" and apply appropriate software countermeasures: antivirus, anti-spyware, anti-popup, firewall, .... Sorry, I disagree. As the quote above about aviation industry approaches to safety indicates: if everyone is having the same problems, and paricularly if other classes of people aren't having the same trouble, the problem isn't the people. It's the tools. The more so when many users are children, aged, and folks who very simply want to use their computer, not be enslaved to it.

The problem with the Times article is that while this situation is very much specific to software and operating systems built with very little eye to security, Richtel and Markoff, with 1244 words to work with, completely fail to make any mention of the fact. This, ladies and gentlemen, is not reportage. It's pointless whinging. And it's pointless whinging which further encourages ignorance of the technology that surrounds us in our daily lives, of readily accessible alternatives, and of an ever more "disposable goods" society.

At the very least, the article could have pointed those frustrated, more-money-than-wits types to several highly worthwhile computer recycling efforts:

In addition, many area schools are interested in not-overly-ancient computers, generally PII or better (typically 3-4 year old hardware). I've worked with Calvin Ross, ROP Instructor, Vintage High School & New Technology High School, Napa, CA. Donations welcomed.

Acknowledgments

Thanks to Ben Tilly for helpful criticism on the "Debian package management equivalent to preemptive multitasking" analogy. I like the basic point, but also recognize that it's something that can be a bit confusing. The basic principle: overt management wins over cooperative systems, especially where incentives to subvert the system exist.


Home
mail: kmself@ix.netcom.com
First published September 20, 2004
Last updated 2008/02/15 03:06:25