This is a personal blog. My other stuff: book | home page | Twitter | prepping | CNC robotics | electronics

August 20, 2010

Commercial web scanners and the word "suck"

I had an interesting debate on a public mailing list today. It might have started with me casually claiming that commercial scanners are "sucky" - and escalated with both the former CTO of SPI Dynamics and the former director of Watchfire both expressing their displeasure. Their take is fair - but this particular argument is perhaps interesting enough to capture it here in a more coherent form.

Let's begin with a relatively uncontroversial observation: we simply don't know how to build decent web application scanners today - be it in the commercial or the open source world. How bad is the issue? Consider that the following topics essentially remain the Millenium Prize Problems in this field:

  • Deciding whether two HTTP responses seen at two different URLs functionally correspond to the same component. Many unrelated pages look very much alike, because of the heavy use of headers, footers, and other templates. Conversely, a single page may change appearance every time it is requested if the content is generated dynamically. Pretty much every single fuzzy matching algorithm used here has spectacular and common failure modes.

  • Deciding what constitutes "page not found" and "error occurred" responses for various parts of the website, a prerequisite for a crawl. This task gets particularly interesting when various URLs are mapped to several different back-end frameworks, a common scenario in enterprise settings; in some of these cases, HTTP response codes are not passed through to the end user, too.

  • Figuring out how the application encodes query parameters and interprets URL paths. Surprisingly, even the most respected and largest enterprise vendors - say, SAP or Oracle - often treat RFCs as a friendly suggestion at best, and invent their own, outlandish conventions for no apparent reason. Throw mod_rewrite into the mix, and things get seriously hairy.

  • Telling whether a particular security attack probe succeeded at all. Consider testing for a buffer overflow: both a successful attack on a vulnerable code, and tripping a security exception due to a well-implemented parameter length check, may result in an identical HTTP 500 response. What next?
The list goes on. In the end, automated scanners very often fare poorly when it comes to finding security bugs: they consistently fail to reach or properly recognize all the low-hanging fruit, spew out significant amounts of false positives - and just as often, simply catch fire when attempting to crawl a site in the first place.

This does not mean that web application scanners are worthless; but their primary value lies in equipping a skilled, human pentester with the initial view of the structure of a site; and automating certain cumbersome tasks, such as the execution of brute-force attacks. The reality is harsh: without the right person behind the wheel, the output of such a scanner, no matter how well tabulated and color-coded, tells you next to nothing about how secure your infrastructure is.

Regrettably, skilled pentesters with in-depth vulnerability hunting expertise, and excellent insight into how the web actually works, are exceptionally hard to get; heck, they are even difficult to recognize: there are no meaningful certifications, no particularly useful professional bodies... and even an impressive employment history or a history of conference presentations is a hit-and-miss indicator.

As a result, many commercial entities end up without the low-level security expertise needed to truly benefit from running a web security scanner - and in absence of this, they unrealistically expect the tool to give them some sort of a turn-key insight into the unknown. This never works as expected; and with nobody equipped to reliably evaluate the quality of the crawl, or the accuracy and ingenuity of the probes used, there is not even a proper feedback loop.

The problems the customers have here reflect negatively on the vendors, too: the company eventually accumulates a baggage of institutional customers who do not exert any pressure on improving the products to - let's say - always have cutting-edge SQL injection checks; and heavily focus on more easily verifiable, but peripheral functionality, instead: plug-and-play support for all sorts of internal SSO systems, elaborate reporting capabilities, compliance-related views that can be shown as a proof of due diligence, and so forth. All these features have some value, of course - but ultimately, they divert resources from the one pursuit that matters the most: helping a skilled pentester, and getting better at it.

In this sense, the commercial vulnerability scanner market is, by large, driven by pathological incentives - and will probably remain this way, at least until enough skilled professionals enter the workforce, or - less likely - until a major technological breakthrough is made.

Now, here's an interesting tidbit: have a look at the open-source vulnerability scanner world. Almost all the web application scanning tools that cropped up in the past two years are written by active bug hunters who use these tools in their day to day work, and were unhappy with what commercial tools have to offer. Not all of these alternatives are good, and not all will succeed - but they are developed with a very important goal in mind: to get really good at spotting bugs, and nothing more. Many of them already surpass some of the more stagnant commercial offerings in this department - and the improvement will continue.

I am very reluctant to make sweeping statements about the superiority of one software development model over another - but in this case, it seems to make a real difference, at least today. And, sure, I recognize that this argument goes both ways: the moment those open source developers start deriving most of their income from selling their products, rather than from consulting services, many of the tools will (as one person put it) quickly begin to suck.


  1. Well said. My experiences completely support all of your above statements too. In my opinion the issue comes down to the lack of available talented resources and the lack of budget for the security departments. The end result is that companies look for the (nonexistent) miracle scanning tool to just point at all sites and push "go".

    I also agree with your statement that automation should be used intelligently and to automate specific testing that yield low false positive results.

    Good post.

  2. It seems to me - admittedly knowing next to nothing about web security - that an external scanner working on its own has little hope of identifying security issues on a web site. I'd have thought you'd need a component on the web server(s) to watch and see what happens "on the inside" as it were.

    I don't recall the details offhand, but I've read about security software that is supposed to be able to detect (and perhaps block) exploitation of unknown vulnerabilities in monitored software in real-time, by spotting when the software being monitored starts behaving abnormally, having first built up some kind of database of normal behaviour. I suspect that combining this sort of functionality with a web scanner has the potential to be quite an effective way of identifying issues.

  3. Some scanners attempt to do this already; but as you can imagine, given the variety of architectures and technologies used to build web services, doing this in a portable manner is a very difficult challenge.

    Plus, in many types of assessments (e.g., external pentests of production systems, testing licensed systems with weird support agreements, etc), server-side instrumentation is often not viable for non-technical reasons.

  4. I've recently had an interview at a company that uses scan reports as a means to push their 3rd part security products.

    From what I can tell these "security experts" were no more than windows system engineers that were buying commercial tools to prove clients 'needed' to buy more equipment.

    For this reason I'm getting a local group together to talk about HIPPA, S&OX (buzzword bingo) but really to address the problem with snake-oil type security consultants that seem to be slapping a logo on a commercial tools scan results and calling it a day.

  5. Great post - I am a big fan and am looking forward to your next book, whatever it is going to be about. Silence on the Wire rocked.

    Anyway - The first coding conundrum that you mention -
    " Deciding whether two HTTP responses seen at two different URLs functionally correspond to the same component. " -
    Why is that important? If it is always the case that URLx may or may not be the same as URLy _and_ URLx may or may not be the same as URLx ten minutes later then why is that even something that matters? Is this simply to prove that the vuln is reproducible ? Is it needed in order to craft a payload? Or is it just some requirement that got thrown in without much thought to why it would be necessary?

    Sure - it is important if you are trying to get your pages ranked with google ;) - but why is it important from an attack or assessment standpoint? It seems to me that an attack would concentrate on something specific. The interesting parts of a web app are not the full page dumps from the CMS, but the constituent pieces that may yield footholds or injection points.
    Just curious as to why that high level categorization would be important.

  6. This matters chiefly because of the need to avoid endless crawling loops, and detect the impact of certain classes of attacks.

    For example, a reasonably common and completely boneheaded web framework design is to only pay attention to the last segment of the URL path. In essence, the servlet may recognize:

    ...or something along these lines. When doing directory brute-force, it's then very easy to endlessly recurse like this:

    You can account for this specific scenario, but other conceptually similar designs exist all over the web.

    You also need the ability to do fuzzy matching to, for example, detect 404 pages that contain search suggestions, etc.

  7. In a perfect world, it would be perfectly acceptable to report that the design of the site being investigated is too boneheaded for a security scan to be meaningful, and to assert that any site as badly designed as that almost certainly has security problems even if they can't be explicitly identified at present. The executive summary could perhaps include the phrase "call me back when you've had the site redesigned by someone competent".

    In the real world, of course, that's probably not going to go down well. :-)