This is a personal blog. My other stuff: book | home page | Twitter | prepping | CNC robotics | electronics

October 24, 2014

PSA: don't run 'strings' on untrusted files (CVE-2014-8485)

Many shell users, and certainly most of the people working in computer forensics or other fields of information security, have a habit of running /usr/bin/strings on binary files originating from the Internet. Their understanding is that the tool simply scans the file for runs of printable characters and dumps them to stdout - something that is very unlikely to put you at any risk.

It is much less known that the Linux version of strings is an integral part of GNU binutils, a suite of tools that specializes in the manipulation of several dozen executable formats using a bundled library called libbfd. Other well-known utilities in that suite include objdump and readelf.

Perhaps simply by the virtue of being a part of that bundle, the strings utility tries to leverage the common libbfd infrastructure to detect supported executable formats and "optimize" the process by extracting text only from specific sections of the file. Unfortunately, the underlying library can be hardly described as safe: a quick pass with afl (and probably with any other competent fuzzer) quickly reveals a range of troubling and likely exploitable out-of-bounds crashes due to very limited range checking, say:

$ wget
$ strings strings-bfd-badptr2
Segmentation fault
strings[24479]: segfault at 4141416d ip 0807a4e7 sp bf80ca60 error 4 in strings[8048000+9a000]
      while (--n_elt != 0)
        if ((++idx)->shdr->bfd_section)                                ← Read from an attacker-controlled pointer
          elf_sec_group (idx->shdr->bfd_section) = shdr->bfd_section;  ← Write to an attacker-controlled pointer
(gdb) p idx->shdr
$1 = (Elf_Internal_Shdr *) 0x41414141

The 0x41414141 pointer being read and written by the code comes directly from that proof-of-concept file and can be freely modified by the attacker to try overwriting program control structures. Many Linux distributions ship strings without ASLR, making potential attacks easier and more reliable - a situation reminiscent of one of the recent bugs in bash.

Interestingly, the problems with the utility aren't exactly new; Tavis spotted the first signs of trouble some nine years ago.

In any case: the bottom line is that if you are used to running strings on random files, or depend on any libbfd-based tools for forensic purposes, you should probably change your habits. For strings specifically, invoking it with the -a parameter seems to inhibit the use of libbfd. Distro vendors may want to consider making the -a mode default, too.

PS. I actually had the libbfd fuzzing job running on this thing!

October 14, 2014

Two more browser memory disclosure bugs (CVE-2014-1580 and #19611cz)

To add several more trophies to afl's pile of image parsing memory disclosure vulnerabilities:
  • MSFA 2014-78 (CVE-2014-1580) fixes another case of uninitialized memory disclosure in Firefox - this time, when rendering truncated GIF images on <canvas>. The bug was reported on September 5 and fixed today. For a convenient test case, check out this page. Rough timeline:

    • September 5: Initial, admittedly brief notification to vendor, including a simple PoC.
    • September 5: Michael Wu confirms the exposure and pinpoints the root cause. Discussion of fixes ensues.
    • September 9: Initial patch created.
    • September 12: Patch approved and landed.
    • October 2: Patch verified by QA.
    • October 13: Fixes ship with Firefox 33.

  • MSRC case #19611cz (MS14-085) is a conceptually similar bug related to JPEG DHT parsing, seemingly leaking bits of stack information in Internet Explorer. This was reported to MSRC on July 2 and hasn't been fixed to date. Test case here. Rough timeline:

    • July 2: Initial, admittedly brief notification to vendor, mentioning the disclosure of uninitialized memory and including a simple PoC.
    • July 3: MSRC request to provide "steps and necessary files to reproduce".
    • July 3: My response, pointing back to the original test case.
    • July 3: MSRC response, stating that they are "unable to determine the nature of what I am reporting".
    • July 3: My response, reiterating the suspected exposure in a more verbose way.
    • July 4: MSRC response from an analyst, confirming that they could reproduce, but also wondering if "his webserver is not loading up a different jpeg just to troll us".
    • July 4: My response stating that I'm not trolling MSRC.
    • July 4: MSRC opens case #19611cz.
    • July 29: MSRC response stating that they are "unable identify a way in which an attacker would be able to propagate the leaked stack data back to themselves".
    • July 29: My response pointing the existence of the canvas.toDataURL() API in Internet Explorer, and providing a new PoC that demonstrates the ability to read back data.
    • September 24: A notification from MSRC stating that the case has been transferred to a new case manager.
    • October 7: My response noting that we've crossed the 90-day mark with no apparent progress made, and that I plan to disclose the bug within a week.
    • October 9: Acknowledgment from MSRC.
Well, that's it. Enjoy!

Fuzzing random programs without execve()

The most common way to fuzz data parsing libraries is to find a simple binary that exercises the interesting functionality, and then simply keep executing it over and over again - of course, with slightly different, randomly mutated inputs in each run. In such a setup, testing for evident memory corruption bugs in the library can be as simple as doing waitpid() on the child process and checking if it ever dies with SIGSEGV, SIGABRT, or something equivalent.

This approach is favored by security researchers for two reasons. Firstly, it eliminates the need to dig into the documentation, understand the API offered by the underlying library, and then write custom code to stress-test the parser in a more direct way. Secondly, it makes the fuzzing process repeatable and robust: the program is running in a separate process and is restarted with every input file, so you do not have to worry about a random memory corruption bug in the library clobbering the state of the fuzzer itself, or having weird side effects on subsequent runs of the tested tool.

Unfortunately, there is also a problem: especially for simple libraries, you may end up spending most of the time waiting for execve(), the linker, and all the library initialization routines to do their job. I've been thinking of ways to minimize this overhead in american fuzzy lop, but most of the ideas I had were annoyingly complicated. For example, it is possible to write a custom ELF loader and execute the program in-process while using mprotect() to temporarily lock down the memory used by the fuzzer itself - but things such as signal handling would be a mess. Another option would be to execute in a single child process, make a snapshot of the child's process memory and then "rewind" to that image later on via /proc/pid/mem - but likewise, dealing with signals or file descriptors would require a ton of fragile hacks.

Luckily, Jann Horn figured a different, much simpler approach, and sent me a patch for afl out of the blue :-) It boils down to injecting a small piece of code into the fuzzed binary - a feat that can be achieved via LD_PRELOAD, via PTRACE_POKETEXT, via compile-time instrumentation, or simply by rewriting the ELF binary ahead of the time. The purpose of the injected shim is to let execve() happen, get past the linker (ideally with LD_BIND_NOW=1, so that all the hard work is done beforehand), and then stop early on in the actual program, before it gets to processing any inputs generated by the fuzzer or doing anything else of interest. In fact, in the simplest variant, we can simply stop at main().

Once the designated point in the program is reached, our shim simply waits for commands from the fuzzer; when it receives a "go" message, it calls fork() to create an identical clone of the already-loaded program; thanks to the powers of copy-on-write, the clone is created very quickly yet enjoys a robust level of isolation from its older twin. Within the child process, the injected code returns control to the original binary, letting it process the fuzzer-supplied input data (and suffer any consequences of doing so). Within the parent, the shim relays the PID of the newly-crated process to the fuzzer and goes back to the command-wait loop.

Of course, when you start dealing with process semantics on Unix, nothing is as easy as it appears at first sight; here are some of the gotchas we had to work around in the code:
  • File descriptor offsets are shared between processes created with fork(). This means that any descriptors that are open at the time that our shim is executed may need to be rewound to their original position; not a significant concern if we are stopping at main() - we can just as well rewind stdin by doing lseek() in the fuzzer itself, since that's where the descriptor originates - but it can become a hurdle if we ever aim at locations further down the line.

  • In the same vein, there are some types of file descriptors we can't fix up. The shim needs to be executed before any access to pipes, character devices, sockets, and similar non-resettable I/O. Again, not a big concern for main().

  • The task of duplicating threads is more complicated and would require the shim to keep track of them all. So, in simple implementations, the shim needs to be injected before any additional threads are spawned in the binary. (Of course, threads are rare in file parser libraries, but may be more common in more heavyweight tools.)

  • The fuzzer is no longer an immediate parent of the fuzzed process, and as a grandparent, it can't directly use waitpid(); there is also no other simple, portable API to get notified about the process' exit status. We fix that simply by having the shim do the waiting, then send the status code to the fuzzer. In theory, we should simply call the clone() syscall with the CLONE_PARENT flag, which would make the new process "inherit" the original PPID. Unfortunately, calling the syscall directly confuses glibc, because the library caches the result of getpid() when initializing - and without a way to make it reconsider, PID-dependent calls such as abort() or raise() will go astray. There is also a library wrapper for the clone() call that does update the cached PID - but the wrapper is unwieldy and insists on messing with the process' stack.

    (To be fair, PTRACE_ATTACH offers a way to temporarily adopt a process and be notified of its exit status, but it also changes process semantics in a couple of ways that need a fair amount of code to fully undo.)
Even with the gotchas taken into account, the shim isn't complicated and has very few moving parts - a welcome relief compared to the solutions I had in mind earlier on. It reads commands via a pipe at file descriptor 198, uses fd 199 to send messages back to parent, and does just the bare minimum to get things sorted out. A slightly abridged verion of the code is:

  /* Phone home and tell the parent that we're OK. */

  pushl $4          /* length    */
  pushl $__afl_temp /* data      */
  pushl $199        /* file desc */
  call  write
  addl  $12, %esp


  /* Wait for parent by reading from the pipe. This will block until
     the parent sends us something. Abort if read fails. */

  pushl $4          /* length    */
  pushl $__afl_temp /* data      */
  pushl $198        /* file desc */
  call  read
  addl  $12, %esp

  cmpl  $4, %eax
  jne   __afl_die

  /* Once woken up, create a clone of our process. */

  call fork

  cmpl $0, %eax
  jl   __afl_die
  je   __afl_fork_resume

  /* In parent process: write PID to pipe, then wait for child. 
     Parent will handle timeouts and SIGKILL the child as needed. */

  movl  %eax, __afl_fork_pid

  pushl $4              /* length    */
  pushl $__afl_fork_pid /* data      */
  pushl $199            /* file desc */
  call  write
  addl  $12, %esp

  pushl $2             /* WUNTRACED */
  pushl $__afl_temp    /* status    */
  pushl __afl_fork_pid /* PID       */
  call  waitpid
  addl  $12, %esp

  cmpl  $0, %eax
  jle   __afl_die

  /* Relay wait status to pipe, then loop back. */

  pushl $4          /* length    */
  pushl $__afl_temp /* data      */
  pushl $199        /* file desc */
  call  write
  addl  $12, %esp

  jmp __afl_fork_wait_loop


  /* In child process: close fds, resume execution. */

  pushl $198
  call  close

  pushl $199
  call  close

  addl  $8, %esp
But, was it worth it? The answer is a resounding "yes": the stop-at-main() logic, already shipping with afl 0.36b, can speed up the fuzzing of many common image libraries by a factor of two or more. It's actually almost unexpected, given that we still keep doing fork(), a syscall with a lingering reputation for being very slow.

The next challenge is devising a way to move the shim down the stream, so that we can also skip any common program initialization steps, such as reading config files - and stop just few instructions shy of the point where the application tries to read the mutated data we are messing with. Jann's original patch has a solution that relies on ptrace() to detect file access; but we've been brainstorming several other ways.

PS. On a related note, some readers might enjoy this.

October 01, 2014

Bash bug: the other two RCEs, or how we chipped away at the original fix (CVE-2014-6277 and '78)

The patch that implements a prefix-based way to mitigate vulnerabilities in bash function exports has been out since last week and has been already picked up by most Linux vendors (plus by Apple). So, here's a quick overview of the key developments along the way, including two really interesting things: proof-of-concept test cases for two serious, previously non-public RCE bugs tracked as CVE-2014-6277 and CVE-2014-6278.

NOTE: If you or your distro maintainers have already deployed Florian's patch, there is no reason for alarm - you are almost certainly not vulnerable to attacks. If you do not have this patch, and instead relied only on the original CVE-2014-6271 fix, you probably need to act now. See this entry for a convenient test case and other tips.

Still here? Good. If you need a refresher, the basic principles of the underlying function export functionality, and the impact of the original bash bug (CVE-2014-6271), are discussed in this blog post. If you have read the earlier post, the original attack disclosed by Stephane Chazelas should be very easy to understand:

HTTP_COOKIE='() { 0; }; echo hi mom;' bash -c :

In essence, the internal parser invoked by bash to process the specially encoded function definitions passed around in environmental variables had a small problem: it continued parsing the code past the end of the function definition itself - and at that point, flat out executed whatever instructions it came across, just as it would do in a normal bash script. Given that the value of certain environmental variables can be controlled by remote attackers in quite a few common settings, this opened up a good chunk of the Internet to attacks.

The original vulnerability was reported privately and kept under embargo for roughly two weeks to develop a fairly conservative fix that modified the parser to bail out in a timely manner and do not parse any trailing commands. As soon as the embargo was lifted, we all found out about the bug and scrambled to deploy fixes. At the same time, a good chunk of the security community reacted with surprise and disbelief that bash is keen to dispatch the contents of environmental variables to a fairly complex syntax parser - so we started poking around.

Tavis was the quickest: he found that you can convince the parser to keep looking for a file name for output redirection past the boundary between the untrusted string accepted from the environment and the actual body of the program that bash is being asked to execute (CVE-2014-7169). His original test case can be simplified at:

HTTP_COOKIE='() { function a a>\' bash -c echo

This example would create an empty file named "echo", instead of executing the requested command. Tavis' finding meant that you would be at risk of remote code execution in situations where attacker-controlled environmental variables are mixed with sanitized, attacker-controlled command-line parameters passed to calls such as system() or popen(). For example, you'd be in trouble if you were doing this in a web app:

system("echo '"+ sanitized_string_without_quotes + "' | /some/trusted/program");

...because the attacker could convince bash to skip over the "echo" command and execute the command given in the second parameter, which happens to be a sanitized string (albeit probably with no ability to specify parameters). On the flip side, this is a fairly specific if not entirely exotic coding pattern - and contrary to some of the initial reports, the bug probably wasn't exploitable in a much more general way.

Chet, the maintainer of bash, started working on a fix to close this specific parsing issue, and released it soon thereafter.

On the same day, Todd Sabin and Florian Weimer have independently bumped into a static array overflow in the parser (CVE-2014-7186). The bug manifested in what seemed to be a non-exploitable crash, trying to dereference a non-attacker-controlled pointer at an address that "by design" should fall well above the end of heap - but was enough to cast even more doubt on the robustness of the underlying code. The test for this problem was pretty simple - you just needed a sequence of here-documents that overflowed a static array, say:

HTTP_COOKIE='() { 0 <<a <<b <<c <<d <<e <<f <<g <<h <<i <<j <<k <<l <<m; }' bash -c :

Florian also bumped into an off-by-one issue with loop parsing (CVE-2014-7187); the proof-of-concept function definition for this is a trivial for loop nested 129 levels deep, but the effect can be only observed under memory access diagnostics tools, and its practical significance is probably low. Nevertheless, all these revelations prompted him to start working on an unofficial but far more comprehensive patch that would largely shield the parser from untrusted strings in normally encountered variables present in the environment.

In parallel to Tavis' and Florian's work, I set up a very straightforward fuzzing job with american fuzzy lop. I seeded it with a rudimentary function definition: () { foo() { foo; }; >bar; } ...and simply let it run with a minimalistic wrapper that took the test case generated by the fuzzer, put it in a variable, and then called execve() to invoke bash.

Although the fuzzer had no clue about the syntax of shell programs, it had the benefit of being able to identify and isolate interesting syntax based on coverage signals, deriving around 1,000 other distinctive test cases from the starting one while "instinctively" knowing not to mess with the essential "() {" prefix. For the first few hours, it kept hitting only the redirect issue originally reported by Todd and the file-creation issue discovered by Tavis - but soon thereafter, it spewed out a new crash illustrated by this snippet of code (CVE-2014-6277):

HTTP_COOKIE='() { x() { _; }; x() { _; } <<a; }' bash -c :

This proved to be a very straightforward use of uninitialized memory: it hit a code path in make_redirect() where one field in a newly-allocated REDIR struct - here_doc_eof - would not be set to any specific value, yet would be treated as a valid pointer later on (somewhere in copy_redirect()).

Now, if bash is compiled with both --enable-bash-malloc and --enable-mem-scramble, the memory returned to make_redirect() by xmalloc() will be set to 0xdf, making the pointer always resolve to 0xdfdfdfdf, and thus rendering the prospect of exploitation far more speculative (essentially depending on whether the stack or any other memory region can be grown by the attacker to overlap with this address). That said, on a good majority of Linux distros, these flags are disabled, and you can trivially get bash to dereference a pointer that is entirely within attacker's control:

HTTP_COOKIE="() { x() { _; }; x() { _; } <<`perl -e '{print "A"x1000}'`; }" bash -c :
bash[25662]: segfault at 41414141 ip 00190d96 sp bfbe6354 error 4 in[110000+191000]

The actual fault happens because of an attempt to copy here_doc_eof to a newly-allocated buffer using a C macro that expands to the following code:

strcpy(xmalloc(1 + strlen(redirect->here_doc_eof)), (redirect->here_doc_eof))

This appears to be exploitable in at least one way: if here_doc_eof is chosen by the attacker to point in the vicinity of the current stack pointer, the apparent contents of the string - and therefore its length - may change between stack-based calls to xmalloc() and strcpy() as a natural consequence of an attempt to pass parameters and create local variables. Such a mid-macro switch will result in an out-of-bounds write to the newly-allocated memory.

A simple conceptual illustration of this attack vector would be: char* result; int len_alloced; main(int argc, char** argv) { /* The offset will be system- and compiler-specific */; char* ptr = &ptr - 9; result = strcpy (malloc(100 + (len_alloced = strlen(ptr))), ptr); printf("requested memory = %d\n" "copied text = %d\n", len_alloced + 1, strlen(result) + 1); } When compiled with the -O2 flag used for bash, on one test system, this produces:

requested memory = 2
copied text = 28

Of course, the result will vary from system to system, but the general consequences of this should be fairly evident. The issue is also made worse by the fact that only relatively few distributions were building bash as a position-independent executable that could be fully protected by ASLR.

(In addition to this vector, there is also a location in dispose_cmd.c that calls free() on the pointer under some circumstances, but I haven't really really spent a lot of time trying to develop a functioning exploit for the '77 bug for reasons that should be evident in the text that follows... well, just about now.)

It has to be said that there is a bit less glamour to such a low-level issue that still requires you to go through some mental gymnastics to be exploited in a portable way. Luckily, the fuzzer kept going, and few hours later, isolated a test case that, after minimization, yielded this gem (CVE-2014-6278):

HTTP_COOKIE='() { _; } >_[$($())] { echo hi mom; id; }' bash -c :

I am... actually not entirely sure what happens here. A sequence of nested $... statements within a redirect appears to cause the parser to bail out without properly resetting its state, and puts it in the mood for executing whatever comes next. The test case works as-is with bash 4.2 and 4.3, but not with more ancient releases; this is probably related to changes introduced few years ago in bash 4.2 patch level 12 (xparse_dolparen()), but I have not investigated if earlier versions are patently not vulnerable or simply require different syntax.

The CVE-2014-6278 payload allows straightforward "put-your-commands-here" remote code execution on systems that are protected only with the original patch - something that we were worried about for a while, and what prompted us to ask people to update again over the past few days.

Well, that's it. I kept the technical details of the last two findings embargoed for a while to give people some time to incorporate Florian's patch and avoid the panic associated with the original bug - but at this point, given the scrutiny that the code is under, the ease of discovering the problems with off-the-shelf open-source tools, and the availability of adequate mitigations, the secrecy seems to have outlived its purpose.

Any closing thoughts? Well, I'm not sure there's a particular lesson to be learnt from the entire story. There's perhaps one thing - it would probably have been helpful if the questionable nature of the original patch was spotted by any of the notified vendors during the two-week embargo period. That said, I wasn't privy to these conversations - and hindsight is always 20/20.