This is a personal blog. My other stuff: book | home page | Twitter | prepping | CNC robotics | electronics

September 25, 2014

Quick notes about the bash bug, its impact, and the fixes so far

We spent a good chunk of the day investigating the now-famous bash bug (CVE-2014-6271), so I had no time to make too many jokes about it on Twitter - but I wanted to jot down several things that have been getting drowned out in the noise earlier in the day.

Let's start with the nature of the bug. At its core, the problem caused by an obscure and little-known feature that allows bash programs to export function definitions from a parent shell to children shells, similarly to how you can export normal environmental variables. The functionality in action looks like this: $ function foo { echo "hi mom"; } $ export -f foo $ bash -c 'foo' # Spawn nested shell, call 'foo' hi mom The behavior is implemented as a hack involving specially-formatted environmental variables: in essence, any variable starting with a literal "() {" will be dispatched to the parser just before executing the main program. You can see this in action here: $ foo='() { echo "hi mom"; }' bash -c 'foo' hi mom The concept of granting magical properties to certain values of environmental variables clashes with several ancient customs - most notably, with the tendency for web servers such as Apache to pass client-supplied strings in the environment to any subordinate binaries or scripts. Say, if I request a CGI or PHP script from your server, the env variables $HTTP_COOKIE and $HTTP_USER_AGENT will be probably initialized to the raw values seen in the original request. If the values happen to begin with "() {" and are ever seen by /bin/bash, events may end up taking an unusual turn.

And so, the bug we're dealing with stems from the observation that trying to parse function-like strings received in HTTP_* variables could have some unintended side effects in that shell - namely, it could easily lead to your server executing arbitrary commands trivially supplied in a HTTP header by random people on the Internet.

With that out of the way, it is important to note that the today's patch provided by the maintainer of bash does not stop the shell from trying to parse the code within headers that begin with "() {" - it merely tries to get rid of that particular RCE side effect, originally triggered by appending commands past the end of the actual function def. But even with all the current patches applied, you can still do this: Cookie: () { echo "Hello world"; } ...and witness a callable function dubbed HTTP_COOKIE() materialize in the context of subshells spawned by Apache; of course, the name will be always prefixed with HTTP_*, so it's unlikely to clash with anything or be called by incident - but intuitively, it's a pretty scary outcome.

In the same vein, doing this will also have an unexpected result: Cookie: () { oops If specified on a request to a bash-based CGI script, you will see a scary bash syntax error message in your error log.

All in all, the fix hinges on two risky assumptions:
  1. That the bash function parser invoked to deal with variable-originating function definitions is robust and does not suffer from the usual range of low-level C string parsing bugs that almost always haunt similar code - a topic that, when it comes to shells, hasn't been studied in much detail before now. (In fact, I am aware of a privately made now disclosed report of such errors in the parser - CVE-2014-7186 and CVE-2014-7187.)

    Update (Sep 26): I also bumped into what seems to be a separate and probably exploitable use of an uninitialized pointer in the parser code; shared the details privately upstream.

  2. That the parsing steps are guaranteed to have no global side effects within the child shell. As it happens, this assertion has been already proved wrong by Tavis (CVE-2014-7169); the side effect he found probably-maybe isn't devastating in the general use case (at least until the next stroke of brilliance), but it's certainly a good reason for concern.

    Update (Sep 26): Found a sixth and most severe issue that is essentially equivalent to the original RCE on all systems that only have the original, maintainer-provided patch.

Contrary to multiple high-profile reports, the original fix was not "broken" in the sense that there is no universal RCE exploit for it - but if I were a betting man, I would not bet on the patch holding up in the long haul (Update: as noted above, it did not hold up). A more reasonable solution would involve temporarily disabling function imports, putting them behind a runtime flag, or blacklisting some of the most dangerous variable patterns (e.g., HTTP_*); and later on, perhaps moving to a model where function exports use a distinct namespace while present in the environment.

What else? Oh, of course: the impact of this bug is an interesting story all in itself. At first sight, the potential for remote exploitation should be limited to CGI scripts that start with #!/bin/bash and to several other programs that explicitly request this particular shell. But there's a catch: on a good majority of modern Linux systems, /bin/sh is actually a symlink to /bin/bash!

This means that web apps written in languages such as PHP, Python, C++, or Java, are likely to be vulnerable if they ever use libcalls such as popen() or system(), all of which are backed by calls to /bin/sh -c '...'. There is also some added web-level exposure through #!/bin/sh CGI scripts, <!--#exec cmd="..."> calls in SSI, and possibly more exotic vectors such as mod_ext_filter.

For the same reason, userland DHCP clients that invoke configuration scripts and use variables to pass down config details are at risk when exposed to rogue servers (e.g., on open wifi). A handful of MTAs, MUAs, or FTP server architectures may be also of concern - in particular, there are third-party reports of qmail installations being at risk. Finally, there is some exposure for environments that use restricted SSH shells (possibly including Git) or restricted sudo commands, but the security of such approaches is typically fairly modest to begin with.

Exposure on other fronts is possible, but probably won't be as severe. The worries around PHP and other web scripting languages, along with the concern for userspace DHCP, are the most significant reasons to upgrade - and perhaps to roll out more paranoid patches, rather than relying solely on the two official ones. On the upside, you don't have to worry about non-bash shells - and that covers a good chunk of embedded systems out there. In particular, contrary to several claims, Busybox should be fine.

Update (Sep 28): the previously-unofficial namespace isolation patch from Florian has eventually made it upstream. You should deploy that patch ASAP.

PS. As for the inevitable "why hasn't this been noticed for 15 years" / "I bet the NSA knew about it" stuff - my take is that it's a very unusual bug in a very obscure feature of a program that researchers don't really look at, precisely because no reasonable person would expect it to fail this way. So, life goes on.

17 comments:

  1. This is actually a feature of GNU bash. One I could never understand, but, oh well.

    You can tighten your GNU/Linux-based systems to prevent this from leaking into popen/system.
    (No need to do this on Android/Linux because they use mksh as /system/bin/sh by default.)
    Install mksh, and use it as /bin/sh (on Debian-based systems, make the symlink to /bin/lksh
    which contains some quirks (enabled by being called as sh) which some legacy Debian
    maintainer scripts need).

    https://www.mirbsd.org/mksh.htm

    ReplyDelete
    Replies
    1. The later versions of debian have sh symlinked to dash, which does not suffer from this particular vulnerability.

      Delete
  2. Given enough eyeballs all bugs are shallow. Thanks for the write-up, it surely explains many doubts about this vulnerability.

    ReplyDelete
  3. I would assume the sensible defense here involves a patch to mod_cgi to ignore (or possibly, prefix with an extra space) any headers whose values begin with the four characters '() {'.

    That and an update to any web application firewalls to wipe out any headers beginning with that sequence. Also, a snort rule to search out HTTP headers beginning with '() {' seems to be in order.

    ReplyDelete
  4. I wonder why openBSD always said bash should not be used, and could be as any bloatware a security issue.
    I guess you are right no one could know, and that is the problem with bloatwares, no one cannot know if it is working «normally».
    But maybe stopping to use and make bloawares (like most of the tools we are designing nowadays) would expose us to less threats?
    Maybe Oauth2 is indeed the next harbringer of a new catastroph or diameter, or .... django or .... apache, mysql, emacs or vim...or dbus, systemd, gnome ...
    Maybe it is time to go back to basics: simpler tools that do one task and do it well. And taking time to code correctly

    ReplyDelete
  5. Could bash disable this "feature" when invoked as /bin/sh, but enable it when invoked as /bin/bash?

    ReplyDelete
  6. The vast majority of PHP pages are executed with mod_php, not CGI, and thus are not vulnerable to this bug (as they never launch a bash process in the first place).

    ReplyDelete
    Replies
    1. As noted in the post, the main concern here would be any call to functions that are backed by popen() / system(), which are somewhat prevalent in especially in apps written in Python, PHP, and other interpreted languages.

      The other prerequisite is that at the time of the call, the environment must contain attacker-controlled variable values; this is probably not a concern for in-process execution on Apache via mod_php, but may very well be for suphp, mod_fcgid, and so on; and for most non-PHP languages such as Python.

      Delete
    2. Michael, you are right 100%: http://imgur.com/lsBQ9DJ

      Delete
  7. This is a great little synopsis. Thanks for posting it!

    ReplyDelete
  8. You wrote: "This means that web apps written in languages such as PHP, Python, C++, or Java, are likely to be vulnerable if they ever use libcalls such as popen() or system()"

    For the life of me, I just cannot get to work… On my (still) unpatched test servers, I've got that PHP script "installed": https://gist.github.com/alexs77/1357baf1b084a365becb

    It does a system("/usr/bin/id") and a shell_exec("/bin/hostname") — but no testsite (eg. http://shellshock.brandonpotter.com/ and http://bashsmash.ccsir.org/) show this script as being exploitable.

    Could somebody maybe show the source of a PHP script, which can be exploited? I would like to "install" that on my servers and test again.

    ReplyDelete
    Replies
    1. Check the environment first. IIRC, some in-process ways of invoking PHP do not involve putting HTTP headers in env variables, in which case, you should be fine. I think that's the case for mod_php, but not for mod_fcgid, suphp, etc.

      Delete
    2. Have same issue while testing php script with system('ls -l') on Ubuntu VM with Apache + php-cgi.
      Solution for me - change symlink /bin/sh from /bin/dash to /bin/bash. After that I got working bash bug on this script.

      Delete
  9. Thank you for the synopsis. I wonder if Windows systems that have installed cygwin or other *nix emulators are vulnerable.

    ReplyDelete
  10. What is the CVE number of "the sixth and most severe issue"? CVE-2014-6277? CVE-2014-6278? other?

    ReplyDelete
    Replies
    1. 6278. Although, both '78 and '77 are bad (as in, remotely exploitable).

      Delete