This is a personal blog. My other stuff: book | home page | Twitter | prepping | CNC robotics | electronics

September 27, 2014

Bash bug: apply Florian's patch now (CVE-2014-6277 and CVE-2014-6278)

OK, rebuild bash and deploy Florian's unofficial patch or its now-upstream version now. If you're a distro maintainer, please consider doing the same.

My previous post has more information about the original vulnerability (CVE-2014-6271). It also explains Tavis' and my original negative sentiment toward the original upstream patch. In short, the revised code did not stop bash from parsing the code seen in potentially attacker-controlled, remotely-originating environmental variables. Instead, the fix simply seeks to harden the parsing to prevent RCE. It relies on two risky assumptions:
  • That spare for this one bug we're fixing now, the process of parsing attacker-controlled functions is guaranteed to have no side effects on the subsequently executed trusted code.

  • That the underlying parser, despite probably not being designed to deal with attacker-supplied inputs, is free from the usual range of C language bugs.
From the very early hours, we have argued on the oss-security mailing list that a more reasonable approach would be to shield the parser from remotely-originating strings. I proposed putting the function export functionality behind a runtime flag or using a separate, prefixed namespace for the exported functions - so that variables such as HTTP_COOKIE do not go through this code path at all. Unfortunately, we made no real progress on that early in the game.

Soon thereafter, people started to bump into additional problems in the parser code. The first assumption behind the patch - the one about the parsing process not having other side effects - was quickly proved wrong by Tavis, who came up with a code construct that would get the parser in an inconsistent state, causing bash to create a bogus file and mangle any subsequent code that /bin/sh is supposed to execute.

This was assigned CVE-2014-7169 and led to a round of high-profile press reports claiming that we're still doomed, and people assigning the new bug CVSS scores all the way up to 11. The reality was a bit more nuanced: the glitch demonstrated by Tavis' code is a bit less concerning, because it does not translate into a universally exploitable RCE - at least not as far as we could figure it out. Some uses of /bin/sh would be at risk, but most would just break in a probably-non-exploitable way. The maintainer followed with another patch that locked down this specific hole.

The second assumption started showing cracks, too. First came a report from Todd Sabin, who identified an static array overflow error when parsing more than ten stacked redirects. The bug, assigned CVE-2014-7186, would cause a crash, but given the nature of the underlying assignment, immediate exploitability seemed fairly unlikely. Another probably non-security off-by-one issue with line counting in loops cropped up shortly thereafter (CVE-2014-7187).

The two latter issues do not have an officially released upstream patch at that point, but they prompted Florian Weimer of Red Hat to develop an unofficial patch that takes a seemingly more durable approach that we argued for earlier on: putting function exports in a separate namespace. Florian's fix effectively isolates the function parsing code from attacker-controlled strings in almost all the important use cases we can currently think of.

(One major outlier would be any solutions that rely on blacklisting environmental variables to run restricted shells or restricted commands as a privileged user - sudo-type stuff - but it's a much smaller attack surface and a a very dubious security boundary to begin with.)

Well... so, to get to the point: I've been fuzzing the underlying function parser on the side - and yesterday, bumped into a new parsing issue (CVE-2014-6277) that is almost certainly remotely exploitable and made easier to leverage due to the fact that bash is seldom compiled with ASLR. I'll share the technical details later on; for now, I sent the info to the maintainer of bash and to several key Linux distros. In general terms, it's an attempt to access uninitialized memory leading to reads from, and then subsequent writes to, a pointer that is fully within attacker's control. Here's a pretty telling crash: bash[3054]: segfault at 41414141 ip 00190d96 ... Soon after posting this entry, I also bumped in the sixth and most severe issue so far, essentially permitting very simple and straightforward remote code execution (CVE-2014-6278) on the systems that are patched against the first bug. It's a "put your commands here" type of a bug similar to the original report. I will post additional details in a couple of days to give people enough time to upgrade.

At this point, I very strongly recommend manually deploying Florian's patch unless your distro is already shipping it. (Florian's patch has been also finally included upstream shortly after I first posted this entry.)

From within the shell itself, the simplest way to check if you already have it installed would be: _x='() { echo vulnerable; }' bash -c '_x 2>/dev/null || echo not vulnerable' If the command shows "vulnerable", you don't have the patch and you are still vulnerable to a (currently non-public) RCE, even if you applied the original one (or the subsequent upstream patch that addressed the issue found by Tavis).

September 25, 2014

Quick notes about the bash bug, its impact, and the fixes so far

We spent a good chunk of the day investigating the now-famous bash bug (CVE-2014-6271), so I had no time to make too many jokes about it on Twitter - but I wanted to jot down several things that have been getting drowned out in the noise earlier in the day.

Let's start with the nature of the bug. At its core, the problem caused by an obscure and little-known feature that allows bash programs to export function definitions from a parent shell to children shells, similarly to how you can export normal environmental variables. The functionality in action looks like this: $ function foo { echo "hi mom"; } $ export -f foo $ bash -c 'foo' # Spawn nested shell, call 'foo' hi mom The behavior is implemented as a hack involving specially-formatted environmental variables: in essence, any variable starting with a literal "() {" will be dispatched to the parser just before executing the main program. You can see this in action here: $ foo='() { echo "hi mom"; }' bash -c 'foo' hi mom The concept of granting magical properties to certain values of environmental variables clashes with several ancient customs - most notably, with the tendency for web servers such as Apache to pass client-supplied strings in the environment to any subordinate binaries or scripts. Say, if I request a CGI or PHP script from your server, the env variables $HTTP_COOKIE and $HTTP_USER_AGENT will be probably initialized to the raw values seen in the original request. If the values happen to begin with "() {" and are ever seen by /bin/bash, events may end up taking an unusual turn.

And so, the bug we're dealing with stems from the observation that trying to parse function-like strings received in HTTP_* variables could have some unintended side effects in that shell - namely, it could easily lead to your server executing arbitrary commands trivially supplied in a HTTP header by random people on the Internet.

With that out of the way, it is important to note that the today's patch provided by the maintainer of bash does not stop the shell from trying to parse the code within headers that begin with "() {" - it merely tries to get rid of that particular RCE side effect, originally triggered by appending commands past the end of the actual function def. But even with all the current patches applied, you can still do this: Cookie: () { echo "Hello world"; } ...and witness a callable function dubbed HTTP_COOKIE() materialize in the context of subshells spawned by Apache; of course, the name will be always prefixed with HTTP_*, so it's unlikely to clash with anything or be called by incident - but intuitively, it's a pretty scary outcome.

In the same vein, doing this will also have an unexpected result: Cookie: () { oops If specified on a request to a bash-based CGI script, you will see a scary bash syntax error message in your error log.

All in all, the fix hinges on two risky assumptions:
  1. That the bash function parser invoked to deal with variable-originating function definitions is robust and does not suffer from the usual range of low-level C string parsing bugs that almost always haunt similar code - a topic that, when it comes to shells, hasn't been studied in much detail before now. (In fact, I am aware of a privately made now disclosed report of such errors in the parser - CVE-2014-7186 and CVE-2014-7187.)

    Update (Sep 26): I also bumped into what seems to be a separate and probably exploitable use of an uninitialized pointer in the parser code; shared the details privately upstream.

  2. That the parsing steps are guaranteed to have no global side effects within the child shell. As it happens, this assertion has been already proved wrong by Tavis (CVE-2014-7169); the side effect he found probably-maybe isn't devastating in the general use case (at least until the next stroke of brilliance), but it's certainly a good reason for concern.

    Update (Sep 26): Found a sixth and most severe issue that is essentially equivalent to the original RCE on all systems that only have the original, maintainer-provided patch.

Contrary to multiple high-profile reports, the original fix was not "broken" in the sense that there is no universal RCE exploit for it - but if I were a betting man, I would not bet on the patch holding up in the long haul (Update: as noted above, it did not hold up). A more reasonable solution would involve temporarily disabling function imports, putting them behind a runtime flag, or blacklisting some of the most dangerous variable patterns (e.g., HTTP_*); and later on, perhaps moving to a model where function exports use a distinct namespace while present in the environment.

What else? Oh, of course: the impact of this bug is an interesting story all in itself. At first sight, the potential for remote exploitation should be limited to CGI scripts that start with #!/bin/bash and to several other programs that explicitly request this particular shell. But there's a catch: on a good majority of modern Linux systems, /bin/sh is actually a symlink to /bin/bash!

This means that web apps written in languages such as PHP, Python, C++, or Java, are likely to be vulnerable if they ever use libcalls such as popen() or system(), all of which are backed by calls to /bin/sh -c '...'. There is also some added web-level exposure through #!/bin/sh CGI scripts, <!--#exec cmd="..."> calls in SSI, and possibly more exotic vectors such as mod_ext_filter.

For the same reason, userland DHCP clients that invoke configuration scripts and use variables to pass down config details are at risk when exposed to rogue servers (e.g., on open wifi). A handful of MTAs, MUAs, or FTP server architectures may be also of concern - in particular, there are third-party reports of qmail installations being at risk. Finally, there is some exposure for environments that use restricted SSH shells (possibly including Git) or restricted sudo commands, but the security of such approaches is typically fairly modest to begin with.

Exposure on other fronts is possible, but probably won't be as severe. The worries around PHP and other web scripting languages, along with the concern for userspace DHCP, are the most significant reasons to upgrade - and perhaps to roll out more paranoid patches, rather than relying solely on the two official ones. On the upside, you don't have to worry about non-bash shells - and that covers a good chunk of embedded systems out there. In particular, contrary to several claims, Busybox should be fine.

Update (Sep 28): the previously-unofficial namespace isolation patch from Florian has eventually made it upstream. You should deploy that patch ASAP.

PS. As for the inevitable "why hasn't this been noticed for 15 years" / "I bet the NSA knew about it" stuff - my take is that it's a very unusual bug in a very obscure feature of a program that researchers don't really look at, precisely because no reasonable person would expect it to fail this way. So, life goes on.

September 02, 2014

CVE-2014-1564: Uninitialized memory with truncated images in Firefox

The recent release of Firefox 32 fixes another interesting image parsing issue found by american fuzzy lop: following a refactoring of memory management code, the past few versions of the browser ended up using uninitialized memory for certain types of truncated images, which is easily measurable with a simple <canvas> + toDataURL() harness that examines all the fuzzer-generated test cases.

In general, problems like that may leak secrets across web origins, or more prosaically, may help attackers bypass security measures such as ASLR. For a slightly more detailed discussion, check out this post.

Here's a short proof-of-concept that should work if you haven't updated to 32 yet: This is tracked as CVE-2014-1564, Mozilla bug 1045977. Several more should be coming soon.

Some notes on web tracking and related mechanisms

Artur Janc and I put together a nice, in-depth overview of all the known fingerprinting and tracking vectors that appear to be present in modern browsers. This is an interesting, polarizing, and poorly-studied area; my main hope is that the doc will bring some structure to the discussions of privacy consequences of existing and proposed web APIs - and help vendors and standards bodies think about potential solutions in a more holistic way.

That's it - carry on!