[conspire] Huge Nov. 5th Washington Post article on Linux security

[Rick Moen]

Four days ago, one Craig Timberg published a very, very long article in
the _Washington Post_:  'The Kernel of the Argument:  Fast, flexible,
and free, Linux is taking over the online world.  But there is growing
unease about security weaknesses.'

http://www.washingtonpost.com/sf/business/2015/11/05/net-of-insecurity-the-kernel-of-the-argument/


It's gotten a lot of notice, including quite a bit of deserved sarcasm,
starting with the awkward reference to 'kernel' and photo of Torvalds at
the top, even though the piece is almost entirely _not_ about the Linux
kernel.  The blunder is obvious and an easy target, but quite possibly
not even mostly of Timberg's making:  At most publications, article
authors have little or no input into the choice of title and
presentation.  OTOH, Timberg _is_ guilty throughout the piece of being
aware of 'Linux' as a kernel differing from 'Linux' as hundreds of
different suites (distros) of the kernel plus thousands of userspace
codebases, many of the latter cross-platform, and nonetheless blurring
the distinction.

This is important because if, say, you speak of a flaw on BIND9 or
Apache httpd as a problem with 'Linux', you are being either clueless or
dishonest or both unless you call it a problem with 'Linux, Windows,
MacOS, and everywhere else these programs run'.  Also, blaming the
kernel for failing to prevent the flaws in badly written userspace 
applications is a serious failure of perspective unless carefully
explained.

To jump ahead for a moment, yes, this is one of the many things Timberg
screws up -- especially the latter bit.


Herewith, a brief effort to sort out the wheat from the chaff, and
comment on both.  (I don't promise I won't cut short and bail, or 
at least switch to doing this in installments, for reasons that will be
apparent if you note the length and discursiveness of Timberg's piece.)

   critics warn that [Linux] has security weaknesses that could be
   fixed but haven't been

So nu?  This is true of every codebase ever written.  Is Timberg
ghostwriting some insider or insiders' peeve?  Next paragraph:

  Worse, as Internet security has surged as a subject of international
  concern, Torvalds has engaged in an occasionally profane standoff with
  experts on the subject. 

Torvalds has had occasionally profane standoffs with just about everyone
occasionally, but in this case I'm guessing that the people Timberg is
ghostwriting for are mainly the maintainers of the PaX and grsecurity 
out-of-tree security patchsets.

I'll mention in advance that I'm -- mostly -- a big fan of those
patchsets and the three people who maintain them.  (To my distress,
starting Sept. 2015, stable patchsets of grsecurity are available only
to paid customers and sponsors:  https://grsecurity.net/announce.php )

https://pax.grsecurity.net/  https://en.wikipedia.org/wiki/PaX
http://grsecurity.net/       https://en.wikipedia.org/wiki/Grsecurity

One cynic's take on this who article might be that it's all about
PaX Team / grsecurity people's sudden attempt to monetise their work.
Hmmm....


There is a history of Torvalds and other core kernel coders having a
difficult relationship with some dedicated security people, including
the PaX and grsecurity teams.  The latter tend to (IMO, justifiably) 
inflame this relationship by questioning the kernel's community's
handling of security bugs and the accuracy of the kernel's documentation
about security problems.  Torvalds has often expressed the view that
dedicated security people are oddballs, and he doesn't think the same
way, or want to.

And, seven short paragraphs in, Timberg switches without comment from
speaking of Linux as a kernel to Linux as an operating system.  Even
though he was talking a paragraph before about the problem being 'Linus
doesn't take security seriously', now it's:

  The rift between Torvalds and security experts is a particular source
  of worry for those who see Linux becoming the dominant operating system
  at a time when technology is blurring the borders between the online 
  and offline worlds....

Eh?  Why are the rift between Torvalds and _certain_ security experts
of concern outside the kernel?  But I need to ignore Timberg doing this,
or I'd be stuck on it for a long time.  Suffice it to say, he does this
sort of thing throughout.  Moving along:


   Even without a potential nuclear disaster, the stakes are high.
   Operating system kernels are the most essential code on any computer,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   allowing hardware to work smoothly with multiple pieces of software.

Meaningless gibberish.  

   Now, consider this: The Linux kernel runs on the New York Stock
   Exchange, every Android smartphone and nearly all of the world’s
   supercomputers. Much of the rapidly expanding universe of connected
   devices uses Linux, as do many of the world’s biggest companies,
   including Google, Facebook and Amazon.com.

Since the only thing all those uses have in common is specifically
the _kernel_ (e.g., Android uses its own libc, and even the kernel
is pretty seriously non-standard), the rest of the piece should be 
about the kernel, right?  It's not.


Timberg has a long digression where he chides Torvalds's habit of ranting
online at any kernel regular whose kernel patch breaks userspace.
Timberg's point, the connection to kernel security, was unclear until we
reach this bit:  

  The effect of Torvalds's approach to managing the kernel -- defensive,
  gradualist, sometimes cranky -- chilled debate about the security of
  Linux even as it became a bigger, richer target for hackers.  The
  result, critics argue, is that while Linux in its early days was widely
  considered a safer choice than Windows or other commercial operating
  systems, the edge has dwindled and perhaps disappeared.

Excuse me, but Torvalds yelling at his immediate kernel-maintenance 
lieutenants over patches that break userspace has 'chilled debate about
the security of Linux'?  Such total bullshit.  Someone, e.g., perhaps
a PaX team member, has fed Timberg a passive-aggressive line and he fell
for it.  No, Torvalds yelling at his trusted lieutenants for breaking
userspace has _not_ chilled debate about the security of the Linux
kernel.  E.g., the PaX and grsecurity maintainers cheerfully state their
views on the Linux kernel mailing list without reservation, and they 
don't even get yelled at for doing so.

The grsecurity and PaX patchsets have never had a prayer of a chance of
being merged into mainline kernels.  And why is that?  Because they
change kernel semantics and break some things.  As usual, tightening
security has serious costs.  Also, the patchsets can be applied cleanly
and functionally to only _certain_ kernel releases.

This addresses, in part, the enduring rift between the core kernel team
(not, as Timberg would suggest, just Torvalds) and dedicated security
people:  The core kernel team feel that the security people are 'crazy'
in the sense of having little appreciation of the many interests that
an OS kernel must serve and the style of governance required for a
rapidly changing codebase.  Torvalds and Co., with total justification, 
feel that if they ran the kernel the way the security people wished they
would, progress would slow to a snail's pace, and a lot of functionality
would simply break.

You can already have a kernel with painfully careful attention to
security.  It also has a glacial development process, abysmal scope of
supported hardware, and regrettably bad performance.  It's called
OpenBSD.


What Timberg _could_ have reasonably said about 'debate about the
security of Linux' is not that Torvalds 'chills' it -- which is 
utter nonsense -- but rather that it tends to be frustratingly
un-fruitful because the core kernel people aren't security professionals
and lack that mindset.  Hence, the two sides of that discussion have
little meeting of the minds, and tend to talk past each other.



[Article quotes Matthew Garrett, now principal security engineer for
CoreOS, as saying that cutting edge security work hasn't been applied to
the mainline kernel, and so isn't as secure as man people would wish it
to be, and then:]

  Versions of Linux have proved vulnerable to serious bugs in recent
  years. AshleyMadison.com, the Web site that facilitates extramarital
  affairs and suffered an embarrassing data breach in July, was reportedly
  running Linux on its servers, as do many companies.

  Those problems did not involve the kernel itself, but [...]

Hey, Mr. Timberg?  What did you just do there?  The kernel hasn't had as
good a security record as it might, says Garrett, and so just look at
the compromise of AshleyMadison.com!  Except, of course, that (very,
very probably) didn't involve the kernel.  (To the best of my ability to
tell, the compromise vector used for AshleyMadison.com was never
disclosed and might not even be known.  In my experience of such cases, 
it's astronomically more likely to be some stupid administrative error,
an inside job, or bad userspace code than a kernel security problem.)  
So, what are you saying, Mr. Timberg?  He goes on:

  [...] but experts say the kernel has become a popular target for
  hackers building 'botnets,' giant networks of computers that can be
  organized to initiate cyberattacks.

And your point is what, Timberg?

By the way, as someone who's done security for a long time, the Linux
kernel is, on balance, one of the components I worry _least_ about.
And for critical systems, there are hardening kits (including PaX and,
until just recently, grsecurity).  Kernels are almost never the problem.
The problem overwhelmingly is vulnerable, badly designed, and/or
ridiculously overfeatured applications.

It should also be noted carefully that the hardening strategies
typically concentrate _mostly_ (verging on overwhelmingly) on making the
Linux kernel better protect vulnerable userspace code from successful
attacks from outside -- mitigating application exploits.  Almost none of
the work is directed at fixing kernel correctness and mitigating kernel
exploit modes.  Ironically, the only clear example of the latter I can
point to came _from a mainline kernel coder_, Andrea Anrcangeli, and is 
a kernel feature (since 2.6.12, this past March) called 'secure
computing mode' or 'seccomp'.  It is -- finally -- a totally effective
sandboxing security for application code.  See:

https://en.wikipedia.org/wiki/Seccomp

This came from the Torvalds camp.  It did _not_ come from the security
malcontents who bitched to reporter Timberg.

In the former category, protecting vulnerable application code from its
own bugs being exploitable, the kernel already does quite a bit, such as
ASLR (randomizes the physical and virtual address at which the
kernel image is decompressed) and NX (No-eXecute bit marking stretches
of memory that shouldn't be executed as not valid to be executed by the
CPU).  Though neither of these is foolproof.

By the way, support for NX is one major advantage that's come painlessly
with x86_64, and is yet another reason to lose those ancient IA32 boxen
-- though the PaX patchset makes NX work on (some?) IA32 boxes.  


Kees Cook has been starting to coordinate information about both types
of kernel-based protective techniques:
http://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project
And for four years, there has been an active and influential
kernel-hardening mailing list:
http://www.openwall.com/lists/kernel-hardening/

Timberg stresses that Torvalds doesn't want to merge a number of such
out-of-tree security patchsets.  OK, and your point, Mr. Timberg?  Do 
you think everyone has to run a mainline kernel for security-sensitive
applications?  Because we don't.  (Users overwhelmingly don't run
kernel.org kernels _anyway_.  They run vendor kernels, which typically 
merge some non-mainline security patches but not others.)


Timberg's basic song is that kernel security could be better.  Well,
gosh, welcome to the real world, Mr. Timberg.  Security of any codebase
could always be better.  The question is cost, in time, money,
complexity, difficulty of debugging, adverse consequences on other code,
and similar things.  Security is always an economic calculation.  

Add hardening, add role-based access controls, reduce risky and unwanted
functionality, and you put a much bigger burden on users and
administrators to know what they're doing.  This is a reason why the
patchsets aren't default.  This is why even the biggest adopters of
SELinux (RHEL, Fedora) apply it only very tentatively ('targeted
policy').

Timberg quotss Kees Cook as describing the horrors of what an attacker
could do who 'gained control of the Linux kernel itself' without
bothering to say that this is like saying 'If I had wings and really low
gravity, I could fly.'

  [Grsecurity's Brad Spengler] soon demonstrated how nearly a dozen
  known Linux coding bugs could be exploited by malicious hackers to
  defeat external defenses and take control of the kernel. 

Yes?  Where?  I read LWN.net and that I seem to recall this is a serious
exaggeration.  Let's see, 2010 interview with Spengler:


'The only comments I make are on LWN (like when Ingo was spreading lies
about the perf_counter vulnerability) or in exploit headers, because
they seem to respond to that (or rather, the commercial influences care
about the media publicity generated by such things, which causes them to
improve security). It took 8(!) public exploits for null pointer
dereference bugs to be fixed as a privilege escalation vector. Granted,
after the first public exploit, mmap_min_addr was introduced some months
later, but the Linux developers who aren't really security people aren't
very good at introducing security features. mmap_min_addr has had at
least half a dozen different bypasses since its inception for such a
small, simple feature.'

https://slo-tech.com/clanki/10001en/

So, misrepresentation by Timberg (probably out of lack of comprehension)
of what Spengler showed.  Layman's explanation:
http://www.zdnet.com/article/linux-exploit-gets-around-security-barrier/

LWN.net coverage:  http://lwn.net/Articles/341773/

In short, privilege-escalation.  Privilege escalation is a Very Bad
Thing, but is not the same as 'coding bugs [that] could be exploited by
malicious hackers to defeat external defenses and take control of the
kernel'.  That is a gross exaggeration.

It should also be mentioned that Spengler's beef was primarily with Red
Hat's '2.6.18' vendor kernel (which the exploit sliced through without 
regard to SELinux), _not_ with kernel.org's tree, which had already been
patched for 11 days to fix that very problem, when he published his
exploit.  See:  http://lwn.net/Articles/341912/


   Torvalds also resisted suggestions that security deserved a special
   place in the hierarchy of concerns faced by software makers. All
   flaws, in his view, were equally serious. 

Bullshit.  This is not his view.

  This attitude was enshrined in a public posting in July 2008 that
  said: "I personally consider security bugs to be just ‘normal bugs.’ I
  don't cover them up, but I also don’t have any reason what-so-ever to
  think it's a good idea to track them and announce them as something
  special."

This is simply not the same as claiming that all flaws are equally
serious -- and Timberg either knew this when he made the claim or should
have.

  But in his recent interview with The Washington Post, Torvalds
  rejected the notion that bugs could be usefully sorted into categories.
  "I refuse to waste a second of my life -- or any other developer's life 
  -- trying to classify something that can’t be classified," he said.

Torvalds has some significant justification for his view, as Timberg
would know if he either studied security or consulted more people about
the story than just a couple of security folk with grudges.  I'm
guessing Torvalds's view is that attempting to highlight a specific class
of 'security' bugs would be futile because it's often very non-obvious 
at the time of discovery what bugs will turn out to have security
implications, and the time required to make that determination would be
better spent just fixing the bugs and moving on.  And there are decades
of history that tend (for the most part) to support Torvalds in this
view.

   Despite providing a steady supply of defensive innovations, Spengler
   did not become a popular figure within the upper reaches of the Linux
   community, where he was seen as extreme in his views and sometimes
   brittle in his manner.

Well, he is.  His comments pretty much everywhere come across as
sarcastic and bitter.

  There is no systemic mechanism for identifying and remedying problems
  [in the Linux kernel] before hackers discover them, or for 
  incorporating the latest advances in defensive technologies. And 
  there is no chief security officer for the Linux kernel.

Nor is there a Santa Claus.[1]

Chief security officers, where they exist, tend to be preplanned
scapegoats for whenever the next security catastrophe occurs.  Because,
honestly, what organisation is make the guy actually the project czar?
That just doesn't happen, so inevitably it's an advisory position
without significant decision authority -- and a scapegoat. 
  
  In the years since Spengler and others began warning about the
  security of Linux, it has triumphed in the marketplace. Google
  released its first version of the Android mobile operating system,
  which is based on Linux, in 2007, allowing Torvalds's work to reach
  hundreds of millions of smartphones each year. Google also made the
  kernel the basis of Chrome OS, which is used in an increasingly
  popular category of cloud-based computers called Chromebooks.

Of course, these are significantly modified _forks_ of Torvalds's work.
Now, watch for the hat-trick where Torvalds & Co. get slagged for 
not only the forked Google kernels but also for Google's notoriously lax 
userspace third-party ecosystem.  Cue it in one... two...

...Three!

  Hackers are more likely to prey upon Oracle's Java and Adobe's Flash
  and Acrobat. 

All of which are extremely deprecated in standard Linux -- because they
are security basket cases, and not safe to expose to arbitrary files on
the Internet.  (I don't know about Acrocrud on Android and ChromeOS, but
I doubt it.)

  More recently, in 2014, Linux devotees were unhappy to discover that
  an Italian surveillance company called Hacking Team had swiftly turned
  a Linux exploit known as "towelroot" into a skeleton key capable of
  gaining access to hundreds of millions of Android phones. 

Which involved Android's failure to patch a June 2014 Linux kernel bug
(CVE-2014-3153) in the futex subsystem (fast userspace mutex locking)
that could be exploited by a local process to escalate to root
privilege.  All distros already had fixed kernels out by the time
CVE-2014-3153 was published on June 5, 2014.  It was a ghastly coding
error, and a serious failure -- but the point is that standard Linux 
fixed it promptly.  

Why didn't Android?  The answer to that is easy and has everything to do
with the Google code ecosystem and nothing to do with Linux:  It
happened because the Android ecosystem has a horribly broken approach to 
code maintenance -- unlike standard Linux.  This is less Google's fault
than it is the institutional laxity built into Google's basic agreement
with the Open Hardware Alliance OEMs, in which the OEMs are in the
driver's seat and not Google.

A hardened variant of CyanogenMod (which itself is a totally open source
variant form of Android) now exists, called CopperheadOS:
https://copperhead.co/android/  I don't know whether their hardened
kernel would have evaded the rather ghastly CVE-2014-3153 bug, even
though the project boasts 'a PaX kernel, OpenBSD's hardened allocator
and compiler hardening features'.  But for damned sure, the project's 
far more likely to actively respond to serious problems than regular
Android in the field is, since the latter's functionally little
different from 'Use what got preloaded until the device dies.'


It should be mentioned in passing that kernel problems in Android don't
even rate:  The real problem in Android is the ubiquitous attitude of it
being OK to run any old code, proprietary or not, out of anywhere.  This
guarantees security problems that dwarf anything involving the kernel.


[skipping discussion of a August 2015 Linux Security Summit keynote
address about lurking security problems by Linux Foundation sysadmin
Konstantin Ryabitsev[0], and some reactions to that keynote]

  "We have some measures in place, although we are really not doing
  everything we can," wrote James Morris, maintainer of Linux's
  exterior defenses against attackers. 

That is _not_ James Morris's remit, by the way.  Timberg once again did
not bother to do basic fact-checking.   Morris maintains the LSM (Linux
Security Modules) subsystem, the kernel's interface for access control
software such as SELinux.[1]  LSM is sometimes inaccurately described as 
'Linux's security subsystem' -- as if security can be accomplished by 
containing it within a subsystem -- so Timberg probably picked up the
grossly inaccurate description from one of those.  But it's yet another
thing he flubbed by not doing fact-checking.

Good article by Morris, putting LSM in context:
https://www.linux.com/learn/docs/727873-overview-of-linux-kernel-security-features/


Timberg cites an encounter between Kees Cook and Torvalds at the 2015
Linux Summit, where Cook promoted some ideas from Morris's Linux
Security Summit keynote, including some specific items from PaX /
grsecurity, and:

  The ability to make the kernel execute code in user-space memory is
  exploitable. The best solution here can be hardware segmentation;
  Intel's "supervisor-mode execution prevention" and ARM's "privileged
  execute never" can both block execution from user-space memory.
  Instrumenting the compiler to set the high address bit on all kernel
  function calls can block calls into user-space memory (since the
  kernel's address space is at the upper end of the virtual address range,
  while user space is at the bottom). Kees also suggested emulating
  segmentation by using separate page tables for user mode and kernel
  mode; Linus jumped in at this point to say that this is the kind of idea
  that makes security people look crazy; such an approach would never
  perform well. He suggested avoiding talking about ideas that will
  clearly never make it into the mainline.

http://lwn.net/Articles/662219/

It's speculative _how_ big a performance hit this radical idea would
exact, but big, anyway.  And Torvalds has a point:  huge hits to
performance just so the kernel can compensate for egregious coding
errors in some applications would be wildly unpopular in 
_general-usage_ kernels.  Timberg portrays this as if it were a
questionable reaction:

  There was a revealing moment, however, when Cook raised the
  possibility of adding an especially intrusive feature long offered by
  Grsecurity. Torvalds immediately spoke up, saying this was "the kind of
  idea that makes security people look crazy," according to LWN.net, a
  site that follows Linux issues.

Some 'security' patches don't have big performance hits or have serious
functionality impact.  To my knowledge, they get mainlined just as often
as any other patches -- but part of the point of Torvalds's 'a bug is a
bug' (and therefore a bug labelled 'security' is per-se just a bug)
attitude that the security gang dislike is that just about _any_ bug
might turn out to have security implications, so why privilege some just
because of whom they arrive from, or what tag someone attaches, without
other reasons?

'Security' patches certainly do get mainlined.  They have an easier time
doing so if credibly introduced on LKML, and easier if they are modest
in scope rather than a huge stupendous change all at once, and easier if 
they don't break userspace.  If those conditions aren't met and the
security practitioner in question gets impatient, maintaining a patchset
out-of-tree still works.  Just ask the PaX / grsecurity people.  Who, 
by the way, aren't even proposing their patches on LKML because they
want to get paid mucho dinero for doing so (see:
https://lwn.net/Articles/538600/), so it's not surprising they don't get
mainlined.  So, what's the complaint, again?


Timberg closes with a comparison of the Linux ecosystem to the city of
Troy, and rhetorically asks who's protecting the Trojans?  

Well, Mr. Timberg, those who want the occasional serious inconveniences
and reduced performance of including the more-far-reaching security
patchsets are doing so, on their security-sensitive systems.  Those who
care less about hardening, which is _mostly_ protecting badly written
userspace applications from the consequences of those bugs, aren't.
It's called choice and diversity.  Some people aren't used to it.

It's nowhere near perfect, and there are bugs in all code, and death and
taxes also exist.  And no surviving citizens of Troy, either.

I'll let Spender have almost the last word, because even though he's a
cranky-pants gadfly with a questionable sense of perspective, I like him
anyway.  Slides:  https://grsecurity.net/spender_summit.pdf

And I'm going to be curious what security-sensitive people do to
compensate for the sudden restricted availability of grsecurity.


But in conclusion:

Sure, security in the Linux kernel is a huge, fat mess, there are 
problems that persist for rather too long, and there have been far too
few attempts at _systemic_ improvement as opposed to the patch-and-pray 
treadmill.  Same old.  Timberg seems to _want_ to imagine a magic world 
where everyone can be made simultaneously happy with a single
reformed/refactored kernel source tree arrived at without major delays
and expense.  Worse, he blames a bunch of things on Torvalds & Co. that
are not within their remit at all.  Worst of all, Timberg served mostly
as a stalking horse for a little gang of security people with grievances
but no real clue how to run a kernel project and why.  These are _my_
kind of malcontents, but I'm aware that they're out of their depth when
they suggest Torvalds & Co. are running the project wrong.  Timberg
isn't, and IMO got taken for a ride.

We needed a good article about the perennial standoff between the core
kernel people and security specialists.  This was about 1/3 of one.
There are real concerns.  This article at least puts them on the table,
though it does a poor job of discussing them.


[0] See also http://kernsec.org/files/lss2015/giant-bags-of-mostly-water.pdf
Highly recommended, and extremely funny & wise stuff.  I have lived this.


[1] It would be an excellent if antiquated gag to rename a village
within the Commonwealth of Virginia to be called 'Santa Claus,
Virginia'.  Except, you'd never be able to find it with a search engine.

[2] Morris is also maintainer of sVirt for virtualisation software and of
the kernel crytographic API, has done work on SELinux, and does other
security work in the kernel.  I don't want to denigrate his work, which
is important.  He's a Red Hat employee living in Sydney.