This is a personal blog. My other stuff: book | home page | Twitter | prepping | CNC robotics | electronics

March 03, 2018

Setting up bug bounties for success

Bug bounties end up in the news with some regularity, usually for the wrong reasons. I've been itching to write about that for a while - but instead of dwelling on the mistakes of the bygone days, I figured it may be better to talk about some of the ways to get vulnerability rewards right.

What do you get out of bug bounties?

There's plenty of differing views, but I like to think of such programs simply as a bid on researchers' time. In the most basic sense, you get three benefits:

  • Improved ability to detect bugs in production before they become major incidents.
  • A comparatively unbiased feedback loop to help you prioritize and measure other security work.
  • A robust talent pipeline for when you need to hire.

What bug bounties don't offer?

You don't get anything resembling a comprehensive security program or a systematic assessment of your platforms. Researchers end up looking for bugs that offer favorable effort-to-payoff ratios for their skills and given the very imperfect information they have about your enterprise. In other words, you may end up with a hundred people looking for XSS and just one person looking for RCE.

Your reward structure can steer them toward the targets and bugs you care about, but it's difficult to fully eliminate this inherent skew. There's only so far you can jack up your top-tier rewards, and only so far you can go lowering the bottom-tier ones.

Don't you have to outcompete the black market to get all the "good" bugs?

There is a free market price discovery component to it all: if you're not getting the engagement you were hoping for, you should probably consider paying more.

That said, there are going to be researchers who'd rather hurt you than work for you, no matter how much you pay; you don't have to win them over, and you don't have to outspend every authoritarian government or every crime syndicate. A bug bounty is effective simply if it attracts enough eyeballs to make bugs statistically harder to find, and reduces the useful lifespan of any zero-days in black market trade. Plus, most researchers don't want their work to be used to crack down on dissidents in Egypt or Vietnam.

Another factor is that you're paying for different things: a black market buyer probably wants a reliable exploit capable of delivering payloads, and then demands silence for months or years to come; a vendor-run bug bounty program is usually perfectly happy with a reproducible crash and doesn't mind a researcher blogging about their work.

In fact, while money is important, you will probably find out that it's not enough to retain your top talent; many folks want bug bounties to be more than a business transaction, and find a lot of value in having a close relationship with your security team, comparing notes, and growing together. Fostering that partnership can be more important than adding another $10,000 to your top reward.

How do I prevent it all from going horribly wrong?

Bug bounties are an unfamiliar beast to most lawyers and PR folks, so it's a natural to be wary and try to plan for every eventuality with pages and pages of impenetrable rules and fine-print legalese.

This is generally unnecessary: there is a strong self-selection bias, and almost every participant in a vulnerability reward program will be coming to you in good faith. The more friendly, forthcoming, and approachable you seem, and the more you treat them like peers, the more likely it is for your relationship to stay positive. On the flip side, there is no faster way to make enemies than to make a security researcher feel that they are now talking to a lawyer or to the PR dept.

Most people have strong opinions on disclosure policies; instead of imposing your own views, strive to patch reported bugs reasonably quickly, and almost every reporter will play along. Demand researchers to cancel conference appearances, take down blog posts, or sign NDAs, and you will sooner or later end up in the news.

But what if that's not enough?

As with any business endeavor, mistakes will happen; total risk avoidance is seldom the answer. Learn to sincerely apologize for mishaps; it's not a sign of weakness to say "sorry, we messed up". And you will almost certainly not end up in the courtroom for doing so.

It's good to foster a healthy and productive relationship with the community, so that they come to your defense when something goes wrong. Encouraging people to disclose bugs and talk about their experiences is one way of accomplishing that.

What about extortion?

You should structure your program to naturally discourage bad behavior and make it stand out like a sore thumb. Require bona fide reports with complete technical details before any reward decision is made by a panel of named peers; and make it clear that you never demand non-disclosure as a condition of getting a reward.

To avoid researchers accidentally putting themselves in awkward situations, have clear rules around data exfiltration and lateral movement: assure them that you will always pay based on the worst-case impact of their findings; in exchange, ask them to stop as soon as they get a shell and never access any data that isn't their own.

So... are there any downsides?

Yep. Other than souring up your relationship with the community if you implement your program wrong, the other consideration is that bug bounties tend to generate a lot of noise from well-meaning but less-skilled researchers.

When this happens, do not get frustrated and do not penalize such participants; instead, help them grow. Consider publishing educational articles, giving advice on how to investigate and structure reports, or offering free workshops every now and then.

The other downside is cost; although bug bounties tend to offer far more bang for your buck than your average penetration test, they are more random. The annual expenses tend to be fairly predictable, but there is always some possibility of having to pay multiple top-tier rewards in rapid succession. This is the kind of uncertainty that many mid-level budget planners react badly to.

Finally, you need to be able to fix the bugs you receive. It would be nuts to prefer to not know about the vulnerabilities in the first place - but once you invite the research, the clock starts ticking and you need to ship fixes reasonably fast.

So... should I try it?

There are folks who enthusiastically advocate for bug bounties in every conceivable situation, and people who dislike them with fierce passion; both sentiments are usually strongly correlated with the line of business they are in.

In reality, bug bounties are not a cure-all, and there are some ways to make them ineffectual or even dangerous. But they are not as risky or expensive as most people suspect, and when done right, they can actually be fun for your team, too. You won't know for sure until you try.

February 24, 2018

Getting product security engineering right

Product security is an interesting animal: it is a uniquely cross-disciplinary endeavor that spans policy, consulting, process automation, in-depth software engineering, and cutting-edge vulnerability research. And in contrast to many other specializations in our field of expertise - say, incident response or network security - we have virtually no time-tested and coherent frameworks for setting it up within a company of any size.

In my previous post, I shared some thoughts on nurturing technical organizations and cultivating the right kind of leadership within. Today, I figured it would be fitting to follow up with several notes on what I learned about structuring product security work - and about actually making the effort count.

The "comfort zone" trap

For security engineers, knowing your limits is a sought-after quality: there is nothing more dangerous than a security expert who goes off script and starts dispensing authoritatively-sounding but bogus advice on a topic they know very little about. But that same quality can be destructive when it prevents us from growing beyond our most familiar role: that of a critic who pokes holes in other people's designs.

The role of a resident security critic lends itself all too easily to a sense of supremacy: the mistaken belief that our cognitive skills exceed the capabilities of the engineers and product managers who come to us for help - and that the cool bugs we file are the ultimate proof of our special gift. We start taking pride in the mere act of breaking somebody else's software - and then write scathing but ineffectual critiques addressed to executives, demanding that they either put a stop to a project or sign off on a risk. And hey, in the latter case, they better brace for our triumphant "I told you so" at some later date.

Of course, escalations of this type have their place, but they need to be a very rare sight; when practiced routinely, they are a telltale sign of a dysfunctional team. We might be failing to think up viable alternatives that are in tune with business or engineering needs; we might be very unpersuasive, failing to communicate with other rational people in a language they understand; or it might be that our tolerance for risk is badly out of whack with the rest of the company. Whatever the cause, I've seen high-level escalations where the security team spoke of valiant efforts to resist inexplicably awful design decisions or data sharing setups; and where product leads in turn talked about pressing business needs randomly blocked by obstinate security folks. Sometimes, simply having them compare their notes would be enough to arrive at a technical solution - such as sharing a less sensitive subset of the data at hand.

To be effective, any product security program must be rooted in a partnership with the rest of the company, focused on helping them get stuff done while eliminating or reducing security risks. To combat the toxic us-versus-them mentality, I found it helpful to have some team members with software engineering backgrounds, even if it's the ownership of a small open-source project or so. This can broaden our horizons, helping us see that we all make the same mistakes - and that not every solution that sounds good on paper is usable once we code it up.

Getting off the treadmill

All security programs involve a good chunk of operational work. For product security, this can be a combination of product launch reviews, design consulting requests, incoming bug reports, or compliance-driven assessments of some sort. And curiously, such reactive work also has the property of gradually expanding to consume all the available resources on a team: next year is bound to bring even more review requests, even more regulatory hurdles, and even more incoming bugs to triage and fix.

Being more tractable, such routine tasks are also more readily enshrined in SDLs, SLAs, and all kinds of other official documents that are often mistaken for a mission statement that justifies the existence of our teams. Soon, instead of explaining to a developer why they should fix a particular problem right away, we end up pointing them to page 17 in our severity classification guideline, which defines that "severity 2" vulnerabilities need to be resolved within a month. Meanwhile, another policy may be telling them that they need to run a fuzzer or a web application scanner for a particular number of CPU-hours - no matter whether it makes sense or whether the job is set up right.

To run a product security program that scales sublinearly, stays abreast of future threats, and doesn't erect bureaucratic speed bumps just for the sake of it, we need to recognize this inherent tendency for operational work to take over - and we need to reign it in. No matter what the last year's policy says, we usually don't need to be doing security reviews with a particular cadence or to a particular depth; if we need to scale them back 10% to staff a two-quarter project that fixes an important API and squashes an entire class of bugs, it's a short-term risk we should feel empowered to take.

As noted in my earlier post, I find contingency planning to be a valuable tool in this regard: why not ask ourselves how the team would cope if the workload went up another 30%, but bad financial results precluded any team growth? It's actually fun to think about such hypotheticals ahead of the time - and hey, if the ideas sound good, why not try them out today?

Living for a cause

It can be difficult to understand if our security efforts are structured and prioritized right; when faced with such uncertainty, it is natural to stick to the safe fundamentals - investing most of our resources into the very same things that everybody else in our industry appears to be focusing on today.

I think it's important to combat this mindset - and if so, we might as well tackle it head on. Rather than focusing on tactical objectives and policy documents, try to write down a concise mission statement explaining why you are a team in the first place, what specific business outcomes you are aiming for, how do you prioritize it, and how you want it all to change in a year or two. It should be a fluid narrative that reads right and that everybody on your team can take pride in; my favorite way of starting the conversation is telling folks that we could always have a new VP tomorrow - and that the VP's first order of business could be asking, "why do you have so many people here and how do I know they are doing the right thing?". It's a playful but realistic framing device that motivates people to get it done.

In general, a comprehensive product security program should probably start with the assumption that no matter how many resources we have at our disposal, we will never be able to stay in the loop on everything that's happening across the company - and even if we did, we're not going to be able to catch every single bug. It follows that one of our top priorities for the team should be making sure that bugs don't happen very often; a scalable way of getting there is equipping engineers with intuitive and usable tools that make it easy to perform common tasks without having to worry about security at all. Examples include standardized, managed containers for production jobs; safe-by-default APIs, such as strict contextual autoescaping for XSS or type safety for SQL; security-conscious style guidelines; or plug-and-play libraries that take care of common crypto or ACL enforcement tasks.

Of course, not all problems can be addressed on framework level, and not every engineer will always reach for the right tools. Because of this, the next principle that I found to be worth focusing on is containment and mitigation: making sure that bugs are difficult to exploit when they happen, or that the damage is kept in check. The solutions in this space can range from low-level enhancements (say, hardened allocators or seccomp-bpf sandboxes) to client-facing features such as browser origin isolation or Content Security Policy.

The usual consulting, review, and outreach tasks are an important facet of a product security program, but probably shouldn't be the sole focus of your team. It's also best to avoid undue emphasis on vulnerability showmanship: while valuable in some contexts, it creates a hypercompetitive environment that may be hostile to less experienced team members - not to mention, squashing individual bugs offers very limited value if the same issue is likely to be reintroduced into the codebase the next day. I like to think of security reviews as a teaching opportunity instead: it's a way to raise awareness, form partnerships with engineers, and help them develop lasting habits that reduce the incidence of bugs. Metrics to understand the impact of your work are important, too; if your engagements are seen mostly as a yet another layer of red tape, product teams will stop reaching out to you for advice.

The other tenet of a healthy product security effort requires us to recognize at a scale and given enough time, every defense mechanism is bound to fail - and so, we need ways to prevent bugs from turning into incidents. The efforts in this space may range from developing product-specific signals for the incident response and monitoring teams; to offering meaningful vulnerability reward programs and nourishing a healthy and respectful relationship with the research community; to organizing regular offensive exercises in hopes of spotting bugs before anybody else does.

Oh, one final note: an important feature of a healthy security program is the existence of multiple feedback loops that help you spot problems without the need to micromanage the organization and without being deathly afraid of taking chances. For example, the data coming from bug bounty programs, if analyzed correctly, offers a wonderful way to alert you to systemic problems in your codebase - and later on, to measure the impact of any remediation and hardening work.

February 02, 2018

Progressing from tech to leadership

I've been a technical person all my life. I started doing vulnerability research in the late 1990s - and even today, when I'm not fiddling with CNC-machined robots or making furniture, I'm probably clobbering together a fuzzer or writing a book about browser protocols and APIs. In other words, I'm a geek at heart.

My career is a different story. Over the past two decades and a change, I went from writing CGI scripts and setting up WAN routers for a chain of shopping malls, to doing pentests for institutional customers, to designing a series of network monitoring platforms and handling incident response for a big telco, to building and running the product security org for one of the largest companies in the world. It's been an interesting ride - and now that I'm on the hook for the well-being of about 100 folks across more than a dozen subteams around the world, I've been thinking a bit about the lessons learned along the way.

Of course, I'm a bit hesitant to write such a post: sometimes, your efforts pan out not because of your approach, but despite it - and it's possible to draw precisely the wrong conclusions from such anecdotes. Still, I'm very proud of the culture we've created and the caliber of folks working on our team. It happened through the work of quite a few talented tech leads and managers even before my time, but it did not happen by accident - so I figured that my observations may be useful for some, as long as they are taken with a grain of salt.

But first, let me start on a somewhat somber note: what nobody tells you is that one's level on the leadership ladder tends to be inversely correlated with several measures of happiness. The reason is fairly simple: as you get more senior, a growing number of people will come to you expecting you to solve increasingly fuzzy and challenging problems - and you will no longer be patted on the back for doing so. This should not scare you away from such opportunities, but it definitely calls for a particular mindset: your motivation must come from within. Look beyond the fight-of-the-day; find satisfaction in seeing how far your teams have come over the years.

With that out of the way, here's a collection of notes, loosely organized into three major themes.

The curse of a techie leader

Perhaps the most interesting observation I have is that for a person coming from a technical background, building a healthy team is first and foremost about the subtle art of letting go.

There is a natural urge to stay involved in any project you've started or helped improve; after all, it's your baby: you're familiar with all the nuts and bolts, and nobody else can do this job as well as you. But as your sphere of influence grows, this becomes a choke point: there are only so many things you could be doing at once. Just as importantly, the project-hoarding behavior robs more junior folks of the ability to take on new responsibilities and bring their own ideas to life. In other words, when done properly, delegation is not just about freeing up your plate; it's also about empowerment and about signalling trust.

Of course, when you hand your project over to somebody else, the new owner will initially be slower and more clumsy than you; but if you pick the new leads wisely, give them the right tools and the right incentives, and don't make them deathly afraid of messing up, they will soon excel at their new jobs - and be grateful for the opportunity.

A related affliction of many accomplished techies is the conviction that they know the answers to every question even tangentially related to their domain of expertise; that belief is coupled with a burning desire to have the last word in every debate. When practiced in moderation, this behavior is fine among peers - but for a leader, one of the most important skills to learn is knowing when to keep your mouth shut: people learn a lot better by experimenting and making small mistakes than by being schooled by their boss, and they often try to read into your passing remarks. Don't run an authoritarian camp focused on total risk aversion or perfectly efficient resource management; just set reasonable boundaries and exit conditions for experiments so that they don't spiral out of control - and be amazed by the results every now and then.

Death by planning

When nothing is on fire, it's easy to get preoccupied with maintaining the status quo. If your current headcount or budget request lists all the same projects as last year's, or if you ever find yourself ending an argument by deferring to a policy or a process document, it's probably a sign that you're getting complacent. In security, complacency usually ends in tears - and when it doesn't, it leads to burnout or boredom.

In my experience, your goal should be to develop a cadre of managers or tech leads capable of coming up with clever ideas, prioritizing them among themselves, and seeing them to completion without your day-to-day involvement. In your spare time, make it your mission to challenge them to stay ahead of the curve. Ask your vendor security lead how they'd streamline their work if they had a 40% jump in the number of vendors but no extra headcount; ask your product security folks what's the second line of defense or containment should your primary defenses fail. Help them get good ideas off the ground; set some mental success and failure criteria to be able to cut your losses if something does not pan out.

Of course, malfunctions happen even in the best-run teams; to spot trouble early on, instead of overzealous project tracking, I found it useful to encourage folks to run a data-driven org. I'd usually ask them to imagine that a brand new VP shows up in our office and, as his first order of business, asks "why do you have so many people here and how do I know they are doing the right things?". Not everything in security can be quantified, but hard data can validate many of your assumptions - and will alert you to unseen issues early on.

When focusing on data, it's important not to treat pie charts and spreadsheets as an art unto itself; if you run a security review process for your company, your CSAT scores are going to reach 100% if you just rubberstamp every launch request within ten minutes of receiving it. Make sure you're asking the right questions; instead of "how satisfied are you with our process", try "is your product better as a consequence of talking to us?"

Whenever things are not progressing as expected, it is a natural instinct to fall back to micromanagement, but it seldom truly cures the ill. It's probable that your team disagrees with your vision or its feasibility - and that you're either not listening to their feedback, or they don't think you'd care. It's good to assume that most of your employees are as smart or smarter than you; barking your orders at them more loudly or more frequently does not lead anyplace good. It's good to listen to them and either present new facts or work with them on a plan you can all get behind.

In some circumstances, all that's needed is honesty about the business trade-offs, so that your team feels like your "partner in crime", not a victim of circumstance. For example, we'd tell our folks that by not falling behind on basic, unglamorous work, we earn the trust of our VPs and SVPs - and that this translates into the independence and the resources we need to pursue more ambitious ideas without being told what to do; it's how we game the system, so to speak. Oh: leading by example is a pretty powerful tool at your disposal, too.

The human factor

I've come to appreciate that hiring decent folks who can get along with others is far more important than trying to recruit conference-circuit superstars. In fact, hiring superstars is a decidedly hit-and-miss affair: while certainly not a rule, there is a proportion of folks who put the maintenance of their celebrity status ahead of job responsibilities or the well-being of their peers.

For teams, one of the most powerful demotivators is a sense of unfairness and disempowerment. This is where tech-originating leaders can shine, because their teams usually feel that their bosses understand and can evaluate the merits of the work. But it also means you need to be decisive and actually solve problems for them, rather than just letting them vent. You will need to make unpopular decisions every now and then; in such cases, I think it's important to move quickly, rather than prolonging the uncertainty - but it's also important to sincerely listen to concerns, explain your reasoning, and be frank about the risks and trade-offs.

Whenever you see a clash of personalities on your team, you probably need to respond swiftly and decisively; being right should not justify being a bully. If you don't react to repeated scuffles, your best people will probably start looking for other opportunities: it's draining to put up with constant pie fights, no matter if the pies are thrown straight at you or if you just need to duck one every now and then.

More broadly, personality differences seem to be a much better predictor of conflict than any technical aspects underpinning a debate. As a boss, you need to identify such differences early on and come up with creative solutions. Sometimes, all you need is taking some badly-delivered but valid feedback and having a conversation with the other person, asking some questions that can help them reach the same conclusions without feeling that their worldview is under attack. Other times, the only path forward is making sure that some folks simply don't run into each for a while.

Finally, dealing with low performers is a notoriously hard but important part of the game. Especially within large companies, there is always the temptation to just let it slide: sideline a struggling person and wait for them to either get over their issues or leave. But this sends an awful message to the rest of the team; for better or worse, fairness is important to most. Simply firing the low performers is seldom the best solution, though; successful recovery cases are what sets great managers apart from the average ones.

Oh, one more thought: people in leadership roles have their allegiance divided between the company and the people who depend on them. The obligation to the company is more formal, but the impact you have on your team is longer-lasting and more intimate. When the obligations to the employer and to your team collide in some way, make sure you can make the right call; it might be one of the the most consequential decisions you'll ever make.

December 10, 2017

Weekend distractions, part deux: a bench, and stuff

Continuing the tradition of the previous post, here's a perfectly good bench:

The legs are 8/4 hard maple, cut into 2.3" (6 cm) strips and then glued together. The top is 4/4 domestic walnut, with an additional strip glued to the bottom to make it look thicker (because gosh darn, walnut is expensive).

Cut on a bandsaw, joined together with a biscuit joiner + glue, then sanded, that's about it. Still applying finish (nitrocellulose lacquer from a rattle can), but this was the last moment when I could snap a photo (about to get dark) and it basically looks like the final product anyway. Pretty simple but turned out nice.

Several additional, smaller woodworking projects here.

November 04, 2017

Weekend distractions: a perfectly good dining table

I've been a DIYer all my adult life. Some of my non-software projects still revolve around computers, especially when they deal with CNC machining or electronics. But I've been also dabbling in woodworking for quite a while. I have not put that much effort into documenting my projects (say, cutting boards) - but I figured it's time to change that. It may inspire some folks to give a new hobby a try - or help them overcome a problem or two.

So, without further ado, here's the build log for a dining table I put together over the past two weekends or so. I think I turned out pretty nice:

Have fun!

April 22, 2017

AFL experiments, or please eat your brötli

When messing around with AFL, you sometimes stumble upon something unexpected or amusing. Say, having the fuzzer spontaneously synthesize JPEG files, come up with non-trivial XML syntax, or discover SQL semantics.

It is also fun to challenge yourself to employ fuzzers in non-conventional ways. Two canonical examples are having your fuzzing target call abort() whenever two libraries that are supposed to implement the same algorithm produce different outputs when given identical input data; or when a library produces different outputs when asked to encode or decode the same data several times in a row.

Such tricks may sound fanciful, but they actually find interesting bugs. In one case, AFL-based equivalence fuzzing revealed a bunch of fairly rudimentary flaws in common bignum libraries, with some theoretical implications for crypto apps. Another time, output stability checks revealed long-lived issues in IJG jpeg and other widely-used image processing libraries, leaking data across web origins.

In one of my recent experiments, I decided to fuzz brotli, an innovative compression library used in Chrome. But since it's been already fuzzed for many CPU-years, I wanted to do it with a twist: stress-test the compression routines, rather than the usually targeted decompression side. The latter is a far more fruitful target for security research, because decompression normally involves dealing with well-formed inputs, whereas compression code is meant to accept arbitrary data and not think about it too hard. That said, the low likelihood of flaws also means that the compression bits are a relatively unexplored surface that may be worth poking with a stick every now and then.

In this case, the library held up admirably - spare for a handful of computationally intensive plaintext inputs (that are now easy to spot due to the recent improvements to AFL). But the output corpus synthesized by AFL, after being seeded just with a single file containing just "0", featured quite a few peculiar finds:

  • Strings that looked like viable bits of HTML or XML: <META HTTP-AAA IDEAAAA, DATA="IIA DATA="IIA DATA="IIADATA="IIA, </TD>.

  • Non-trivial numerical constants: 1000,1000,0000000e+000000, 0,000 0,000 0,0000 0x600, 0000,$000: 0000,$000:00000000000000.

  • Nonsensical but undeniably English sentences: them with them m with them with themselves, in the fix the in the pin th in the tin, amassize the the in the in the inhe@massive in, he the themes where there the where there, size at size at the tie.

  • Bogus but semi-legible URLs: CcCdc.com/.com/m/ /00.com/.com/m/ /00(0(000000CcCdc.com/.com/.com

  • Snippets of Lisp code: )))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))).

The results are quite unexpected, given that they are just a product of randomly mutating a single-byte input file and observing the code coverage in a simple compression tool. The explanation is that brotli, in addition to more familiar binary coding methods, uses a static dictionary constructed by analyzing common types of web content. Somehow, by observing the behavior of the program, AFL was able to incrementally reconstruct quite a few of these hardcoded keywords - and then put them together in various semi-interesting ways. Not bad.

August 26, 2016

So you want to work in security (but are too lazy to read Parisa's excellent essay)

If you have not seen it yet, Parisa Tabriz penned a lengthy and insightful post about her experiences on what it takes to succeed in the field of information security.

My own experiences align pretty closely with Parisa's take, so if you are making your first steps down this path, I strongly urge you to give her post a good read. But if I had to sum up my lessons from close to two decades in the industry, I would probably boil them down to four simple rules:

  1. Infosec is all about the mismatch between our intuition and the actual behavior of the systems we build. That makes it harmful to study the field as an abstract, isolated domain. To truly master it, dive into how computers work, then make a habit of asking yourself "okay, but what if assumption X does not hold true?" every step along the way.

  2. Security is a protoscience. Think of chemistry in the early 19th century: a glorious and messy thing, chock-full of colorful personalities, unsolved mysteries, and snake oil salesmen. You need passion and humility to survive. Those who think they have all the answers are a danger to themselves and to people who put their faith in them.

  3. People will trust you with their livelihoods, but will have no way to truly measure the quality of your work. Don't let them down: be painfully honest with yourself and work every single day to address your weaknesses. If you are not embarrassed by the views you held two years ago, you are getting complacent - and complacency kills.

  4. It will feel that way, but you are not smarter than software engineers. Walk in their shoes for a while: write your own code, show it to the world, and be humiliated by all the horrible mistakes you will inevitably make. It will make you better at your job - and will turn you into a better person, too.

August 04, 2016

CSS mix-blend-mode is bad for your browsing history

Up until mid-2010, any rogue website could get a good sense of your browsing habits by specifying a distinctive :visited CSS pseudo-class for any links on the page, rendering thousands of interesting URLs off-screen, and then calling the getComputedStyle API to figure out which pages appear in your browser's history.

After some deliberation, browser vendors have closed this loophole by disallowing almost all attributes in :visited selectors, spare for the fairly indispensable ability to alter foreground and background colors for such links. The APIs have been also redesigned to prevent the disclosure of this color information via getComputedStyle.

This workaround did not fully eliminate the ability to probe your browsing history, but limited it to scenarios where the user can be tricked into unwittingly feeding the style information back to the website one URL at a time. Several fairly convincing attacks have been demonstrated against patched browsers - my own 2013 entry can be found here - but they generally depended on the ability to solicit one click or one keypress per every URL tested. In other words, the whole thing did not scale particularly well.

Or at least, it wasn't supposed to. In 2014, I described a neat trick that exploited normally imperceptible color quantization errors within the browser, amplified by stacking elements hundreds of times, to implement an n-to-2n decoder circuit using just the background-color and opacity properties on overlaid <a href=...> elements to easily probe the browsing history of multiple URLs with a single click. To explain the basic principle, imagine wanting to test two links, and dividing the screen into four regions, like so:

  • Region #1 is lit only when both links are not visited (¬ link_a ∧ ¬ link_b),
  • Region #2 is lit only when link A is not visited but link B is visited (¬ link_a ∧ link_b),
  • Region #3 is lit only when link A is visited but link B is not (link_a ∧ ¬ link_b),
  • Region #4 is lit only when both links are visited (link_a ∧ link_b).

While the page couldn't directly query the visibility of the segments, we just had to convince the user to click the visible segment once to get the browsing history for both links, for example under the guise of dismissing a pop-up ad. (Of course, the attack could be scaled to far more than just 2 URLs.)

This problem was eventually addressed by browser vendors by simply improving the accuracy of color quantization when overlaying HTML elements; while this did not eliminate the risk, it made the attack far more computationally intensive, requiring the evil page to stack millions of elements to get practical results. Gave over? Well, not entirely. In the footnote of my 2014 article, I mentioned this:

"There is an upcoming CSS feature called mix-blend-mode, which permits non-linear mixing with operators such as multiply, lighten, darken, and a couple more. These operators make Boolean algebra much simpler and if they ship in their current shape, they will remove the need for all the fun with quantization errors, successive overlays, and such. That said, mix-blend-mode is not available in any browser today."

As you might have guessed, patience is a virtue! As of mid-2016, mix-blend-mode - a feature to allow advanced compositing of bitmaps, very similar to the layer blending modes available in photo-editing tools such as Photoshop and GIMP - is shipping in Chrome and Firefox. And as it happens, in addition to their intended purpose, these non-linear blending operators permit us to implement arbitrary Boolean algebra. For example, to implement AND, all we need to do is use multiply:

  • black (0) x black (0) = black (0)
  • black (0) x white (1) = black (0)
  • white (1) x black (0) = black (0)
  • white (1) x white (1) = white (1)

For a practical demo, click here. A single click in that whack-a-mole game will reveal the state of 9 visited links to the JavaScript executing on the page. If this was an actual game and if it continued for a bit longer, probing the state of hundreds or thousands of URLs would not be particularly hard to pull off.

May 11, 2016

Clearing up some misconceptions around the "ImageTragick" bug

The recent, highly publicized "ImageTragick" vulnerability had countless web developers scrambling to fix a remote code execution vector in ImageMagick - a popular bitmap manipulation tool commonly used to resize, transcode, or annotate user-supplied images on the Web. Whatever your take on "branded" vulnerabilities may be, the flaw certainly is notable for its ease of exploitation: it is an embarrassingly simple shell command injection bug reminiscent of the security weaknesses prevalent in the 1990s, and nearly extinct in core tools today. The issue also bears some parallels to the more far-reaching but equally striking Shellshock bug.

That said, I believe that the publicity that surrounded the flaw was squandered by failing to make one very important point: even with this particular RCE vector fixed, anyone using ImageMagick to process attacker-controlled images is likely putting themselves at a serious risk.

The problem is fairly simple: for all its virtues, ImageMagick does not appear to be designed with malicious inputs in mind - and has a long and colorful history of lesser-known but equally serious security flaws. For a single data point, look no further than the work done several months ago by Jodie Cunningham. Jodie fuzzed IM with a vanilla setup of afl-fuzz - and quickly identified about two dozen possibly exploitable security holes, along with countless denial of service flaws. A small sample of Jodie's findings can be found here.

Jodie's efforts probably just scratched the surface; after "ImageTragick", a more recent effort by Hanno Boeck uncovered even more bugs; from what I understand, Hanno's work also went only as far as using off-the-shelf fuzzing tools. You can bet that, short of a major push to redesign the entire IM codebase, the trickle won't stop any time soon.

And so, the advice sorely missing from the "ImageTragick" webpage is this:

  • If all you need to do is simple transcoding or thumbnailing of potentially untrusted images, don't use ImageMagick. Make a direct use of libpng, libjpeg-turbo, and giflib; for a robust way to use these libraries, have a look at the source code of Chromium or Firefox. The resulting implementation will be considerably faster, too.

  • If you have to use ImageMagick on untrusted inputs, consider sandboxing the code with seccomp-bpf or an equivalent mechanism that robustly restricts access to all user space artifacts and to the kernel attack surface. Rudimentary sandboxing technologies, such as chroot() or UID separation, are probably not enough.

  • If all other options fail, be zealous about limiting the set of image formats you actually pass down to IM. The bare minimum is to thoroughly examine the headers of the received files. It is also helpful to explicitly specify the input format when calling the utility, as to preempt auto-detection code. For command-line invocations, this can be done like so:

    convert [...other params...] -- jpg:input-file.jpg jpg:output-file.jpg

    The JPEG, PNG, and GIF handling code in ImageMagick is considerably more robust than the code that supports PCX, TGA, SVG, PSD, and the likes.

February 09, 2016

Automatically inferring file syntax with afl-analyze

The nice thing about the control flow instrumentation used by American Fuzzy Lop is that it allows you to do much more than just, well, fuzzing stuff. For example, the suite has long shipped with a standalone tool called afl-tmin, capable of automatically shrinking test cases while still making sure that they exercise the same functionality in the targeted binary (or that they trigger the same crash). Another similar tool, afl-cmin, employed a similar trick to eliminate redundant files in any large testing corpora.

The latest release of AFL features another nifty new addition along these lines: afl-analyze. The tool takes an input file, sequentially flips bytes in this data stream, and then observes the behavior of the targeted binary after every flip. From this information, it can infer several things:

  • Classify some content as no-op blocks that do not elicit any changes to control flow (say, comments, pixel data, etc).
  • Checksums, magic values, and other short, atomically compared tokens where any bit flip causes the same change to program execution.
  • Longer blobs exhibiting this property - almost certainly corresponding to checksummed or encrypted data.
  • "Pure" data sections, where analyzer-injected changes consistently elicit differing changes to control flow.

This gives us some remarkable and quick insights into the syntax of the file and the behavior of the underlying parser. It may sound too good to be true, but actually seems to work in practice. For a quick demo, let's see what afl-analyze has to say about running cut -d ' ' -f1 on a text file:

We see that cut really only cares about spaces and newlines. Interestingly, it also appears that the tool always tokenizes the entire line, even if it's just asked to return the first token. Neat, right?

Of course, the value of afl-analyze is greater for incomprehensible binary formats than for simple text utilities; perhaps even more so when dealing with black-box parsers (which can be analyzed thanks to the runtime QEMU instrumentation supported in AFL). To try out the tool's ability to deal with binaries, let's check out libpng:

This looks pretty damn good: we have two four-byte signatures, followed by chunk length, four-byte chunk name, chunk length, some image metadata, and then a comment section. Neat, right? All in a matter of seconds: no configuration needed and no knobs to turn.

Of course, the tool shipped just moments ago and is still very much experimental; expect some kinks. Field testing and feedback welcome!

January 14, 2016

Show and tell: doomsday planning for less crazy folk

Yup. I've been quiet of recent, but that's in part because I've been working on this piece:

http://lcamtuf.coredump.cx/prep/

It's a fairly systematic and level-headed approach to threat modeling and risk management, except not for computer systems - and instead, for real life. There's not much I can add on top of what's already said on the linked page; have a look, you will probably find it to be an interesting read.

July 15, 2015

Poland and the United States: all that begins must end

With my previous entry, I wrapped up an impromptu series of articles that chronicled my childhood experiences in Poland and compared the culture I grew up with to the American society that I'm living in today. For the readers who want to be able to navigate the series without scrolling endlessly, I wanted to put together a quick table of contents. Here it goes.

The entry that started it all:

  • "On journeys" - a personal story recounting my travels from Poland to the US.

Oh, the places you won't go:

Poland (and Europe) vs the United States:

And now, back to the regularly scheduled programming...

Poland vs the United States: American exceptionalism

This is the fourteenth article talking about Poland, Europe, and the United States. To explore the entire collection, start here.

This is destined to be the final entry in the series that opened with a chronicle of my journey from Poland to the United States, only to veer into some of the most interesting social differences between America and the old continent. There are many other topics I could still write about - anything from the school system, to religion, to the driving culture - but with my parental leave coming to an end, I decided to draw a line. I'm sure that this decision will come as a relief for those who read the blog for technical insights, rather than political commentary :-)

The final topic I wanted to talk about is something that truly irks some of my European friends: the belief, held deeply by many Americans, that their country is the proverbial "city upon a hill" - a shining beacon of liberty and righteousness, blessed by the maker with the moral right to shape the world - be it by flexing its economic and diplomatic muscles, or with its sheer military might.

It is an interesting phenomenon, and one that certainly isn't exclusive to the United States. In fact, expansive exceptionalism used to be a very strong theme in the European doctrine long before it emerged in other parts of the Western world. For one, it underpinned many of the British, French, Spanish, and Dutch colonial conquests over the past 500 years. The romanticized notion of Sonderweg played a menacing role in German political discourse, too - eventually culminating in the rise of the Nazi ideology and the onset of World War II. It wasn't until the defeat of the Third Reich when Europe, faced with unspeakable destruction and unprecedented loss of life, made a concerted effort to root out many of its nationalist sentiments and embrace a more harmonious, collective path as a single European community.

America, in a way, experienced the opposite: although it has always celebrated its own rejection of feudalism and monarchism - and in that sense, it had a robust claim to being a pretty unique corner of the world - the country largely shied away from global politics, participating only very reluctantly in World War I, then hoping to wait out World War II up until being attacked by Japan. Its conviction about its special role on the world stage has solidified only after it paid a tremendous price to help defeat the Germans, to stop the march of the Red Army through the continent, and to build a prosperous and peaceful Europe; given the remarkable significance of this feat, the post-war sentiments in America may be not hard to understand. In that way, the roots of American exceptionalism differed from its European predecessors, being fueled by a fairly pure sense of righteousness - and not by anger, by a sense of injury, or by territorial demands.

Of course, the new superpower has also learned that its military might has its limits, facing humiliating defeats in some of the proxy wars with the Soviets and seeing an endless spiral of violence in the Middle East. The voices predicting its imminent demise, invariably present from the earliest days of the republic, have grown stronger and more confident over the past 50 years. But the country remains a military and economic powerhouse; and in some ways, its trigger-happy politicians provide a counterbalance to the other superpowers' greater propensity to turn a blind eye to humanitarian crises and to genocide. It's quite possible that without the United States arming its allies and tempering the appetites of Russia, North Korea, or China, the world would have been a less happy place. It's just as likely that the Middle East would have been a happier one.

Some Europeans show indignation that Americans, with their seemingly know-it-all attitudes toward the rest of the world, still struggle to pinpoint Austria or Belgium on the map. It is certainly true that the media in the US pays little attention to the old continent. But deep down inside, European outlets don't necessarily fare a lot better, often focusing its international coverage on the silly and the formulaic: when in Europe, you are far more likely to hear about a daring rescue of a cat stuck on a tree in Wyoming, or about the Creation Museum in Kentucky, than you are to learn anything substantive about Obamacare. (And speaking of Wyoming and Kentucky, pinpointing these places on the map probably wouldn't be the European viewer's strongest feat). In the end, Europeans who think they understand the intricacies of US politics are probably about as wrong as the average American making sweeping generalizations about Europe.

And on that intentionally self-deprecating note, it's time to wrap the series up.

Poland vs the United States: work and entitlements

This is the thirteenth article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

In one of my earlier posts, I alluded to the pervasive faith in the American Dream: the national ethos of opportunity, self-sufficiency, and free enterprise that influences the political discourse in the United States. The egalitarian promise of the American Dream is simple: no matter who you are, hard work and ingenuity will surely allow you to achieve your dreams. From that, it follows that on your journey, you are not entitled to much; the government will be there to protect your freedom, but it will not give you a head start.

Unlike many of my peers, I suspect that there is truth to the cliche; the United States is a remarkably industrious nation and the home to many of the world's most innovative and fastest-growing businesses. It certainly treads ahead of European economies, still dominated by pre-war industrial conglomerates and former state monopolists, and weighed down by aging populations, highly regulated markets, and inflexible, out-of-control costs. America's mostly-self-made magnates, the likes of Elon Musk, Bill Gates, and Warren Buffett, are also far more likable and seemingly more human than Europe's stereotypical caste of aristocratic families and shadowy oligarchs.

On the flip side, the striking upward mobility of rags-to-riches icons such as Steve Jobs or Oprah Winfrey tends to be an exception, not a rule. Many scholars point out that parents' incomes are highly predictive of the incomes of their children - and that in the US, this effect is more pronounced than in some of the European states. Such studies can be misleading, because in less unequal EU societies, moving to a higher income quantile may confer no substantial change in the quality of life - but ultimately, there is no denying that people who are born into poor families will usually remain poor for the rest of their lives. And with the contemporary trends in outsourcing and industrial automation, the opportunities for unskilled blue collar labor - once a key stepping stone in the story of the American Dream - are shrinking fast.

In contrast with the United States, many in Europe reject Milton Friedman's views on consensual capitalism and hold that it is a basic human right to be able to live a good life or to have an honest and respectable job. This starts with the labor law: in much of the United States, firing an employee can happen in the blink of an eye, for almost any reason - or without giving a reason at all. In Europe, the employer will need a just cause and will go through a lengthy severance period; depending on the circumstances, the company may be also barred from hiring another person to do the same job. Employment benefits follow the same pattern; in the US, paid leave is largely up to employers to decide, with skilled workers being lured with packages that would make Europeans jealous - but many unskilled laborers, especially in the retail and restaurant business, getting the short end of that stick.

In Europe, enabling the disadvantaged to contribute to the society and to live fulfilling lives is also a matter of government policy, often implemented through sweeping wealth redistribution - or through public-sector employment orchestrated at a scale that rivals that of quasi-communist China and other authoritarian countries (for example, in France and Greece, about one in three jobs is run by the state). Such efforts tend to be more successful in small and wealthy Scandinavian countries, where the society can be engineered with more finesse. In many other parts of the continent, systemic, long-term poverty is still rampant, with the government being able to do little more than providing people with a lifetime of subsidized basic sustenance and squalor living conditions. Ultimately, when it comes to combating multi-generational poverty, financial aid administered by sprawling national bureaucracies is not always a cure-all.

Perhaps interestingly, the benefits that are most frequently described as inadequate in the US are not as strikingly different from what one would be entitled to in the EU. For example, the minimal wage is quite comparable; it is around $2.60 per hour in Poland, about $3.70 in Greece, some $9.30 in Germany, and in the ballpark of $10.00 in the UK. In the US, the national average hovers somewhere around $8.00, with some of the states with higher costs of living on track to raise it to $10.00 within a year or two; in fact, some progressive municipalities are aiming for $15.

Unemployment and retirement benefits, although certainly not lavish, also follow the same pattern. When it comes to unemployment in particular, in the States, workers are entitled to about half of their previous salary for up to six months - although that period has been routinely extended in times of economic calamity. In Europe, the figures are roughly comparable, with payments in the ballpark of 50-70% of your previous salary, typically extending for somewhere between 6 and 12 months. The main difference is that the upper limit for monthly benefits tends to be significantly lower in the US than in Europe, often putting far greater strain on single-income families in places with high cost of living. In France, the ceiling seems to be around $8,000 a month; in the US, you will probably see no more than $2,000.

Another overlooked dimension of this debate is the unique tradition of charitable giving in the United States - a phenomenon that allows private charities to provide extensive assistance to people in need. Such giving happens on a staggering scale, with citizens donating more than $350 billion a year - more than twenty times the amount donated in the UK. The bulk of that money goes to organization that provide food, shelter, and counseling to the poor. It is an interesting model, with its own share of benefits and trade-offs: private charities operate on a more local scale and have a far stronger incentive to spend money wisely and provide meaningful aid. On the flip side, their reach is not as universal - and the benefits are not guaranteed.

Many of the conservatives who preach the virtues of the American Dream vastly underestimate the pervasive and lasting consequences of being born into poverty or falling onto hard times; they also underestimate the role that unearned privilege and luck played in their own lives. The progressives often do no better, seeing European social democracies as a flawless role model, even in the midst of the enduring sovereign debt crisis in the eurozone; breathlessly reciting knock-off Marxist slogans; and portraying the rich as Mr. Burns-esque villains of unfathomable wealth, motivated by just two goals: to exploit the working class and to avoid paying taxes at any cost. In the end, helping the disadvantaged is a moral imperative - but many ideas sound better on a banner than when implemented as a government policy.

For the next and final article in the series, click here.

July 14, 2015

Poland vs the United States: governance

This is the twelfth article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

The American model of government is a complex beast. To a visitor from continental Europe, accustomed to the Napoleonic traditions of civil law and to the political realities of unitary states, the sight can be also a bit perplexing: after all, how does a country of this size prosper with a bitterly partisan, gridlocked Congress that repeatedly fails to even pass the budget on time? And how is it possible that, with an approval rating of 15%, the elected officials are not facing a wave of widespread social unrest?

I suspect that the key to solving this riddle lies in the fact that the United States is still very much a federation of self-governing states - and that most of the decisions that affect the lives of ordinary citizens are not made in Washington. Each and every state establishes its own criminal and civil law, levies its own taxes, runs its own welfare systems, and appoints its own judges - sometimes by popular vote. In fact, the states routinely confer far-reaching powers onto individual municipalities: for example, most towns and counties operate their own, completely autonomous police departments that respond to local officials, not to a career politician on the East Coast.

All this makes the government feel quite different from what you are likely to experience in Europe. Let's stick to law enforcement: in Poland and in some other European states, where the police are a part of a sprawling national bureaucracy, the citizens may have very few options for addressing concerns that do not rise to the level of national debate. In the US, dismantling the entire police force may seem trivial in comparison: the concerned citizens may need to get a local newspaper interested in their cause, then band together to recall the local official who is ultimately on the hook. Of course, the independence comes at a price: small, self-funded police departments can be quicker to adopt questionable practices that would not stand to broader scrutiny, such as racial profiling or the rash application of civil forfeiture.

When it comes to the role of the federal government, the picture is complicated. In principle, the constitution gives it only a couple of duties; for example, the feds control various aspects of interstate commerce, print money, maintain armed forces, and handle foreign affairs. Of course, over the years, their responsibilities have expanded considerably, with the legislators exploiting the vagueness of the concept of "interstate commerce" in all sorts of creative ways. Today, the ongoing debate about the appropriate boundaries of this practice fuels the partisan gridlock in Washington. Modern-day Republicans, swayed by the conservative Tea Party movement, argue that the feds should honor the vision of the Founding Fathers and not meddle in the affairs of the states. The Democratic party, taking notes from the vaguely leftist Occupy campaign, increasingly sees the federal government as a flexible tool for establishing country-wide standards of environmental protection, labor rights, welfare, gun control, education, and other progressive causes historically associated with European social democrats.

On that matter, the voters themselves seem to be split. In polls, a robust majority of Americans declare that their government regulates too many aspects of their lives, tries to solve too many problems, wields too much control, and is inherently less efficient and less fair than private enterprises; about two-thirds of respondents see the feds as more of a problem than a solution, and a shocking 50% believe that the apparatus poses an immediate and serious threat to civil liberties. Yet, despite holding views that would make Milton Friedman proud, when asked about specific programs and entitlements - be it defense spending or Medicare - most voters oppose budget cuts. Ultimately, the equally powerful distrust of big corporations, coupled with the allure of European-style welfare systems, often sends the public into the embrace of big-government progressives who promise to solve a growing range of societal ills using federal-level income redistribution and overarching legislative frameworks.

Either way, owing to the parties' newly-found tendency to pander to populist fringes and their inability to compromise, the dysfunctional Congress gets very little love from the average voter; but somewhat paradoxically, the representatives from each and every district are usually well-liked by their own constituents and get reelected with ease. Some blame gerrymandering, but a simpler explanation exists: most of the candidates have strong ties to the districts they represent, many of them having a track record as local politicians or successful businessmen. As a result, they understand what matters to their constituents and often meaningfully work to advance that agenda. They also live and die at the mercy of local newspapers, sometimes lending a hand to the voters who write or call them to resolve bureaucratic hurdles and address other everyday grievances. The practice of getting your representatives involved in such matters is almost unthinkable in Poland, where the slots on local ballots are traded by party officials - and are routinely handed out to people with little or no connection to the region they are supposed to represent.

With American political campaigns financed from private funds, it is often argued that the representatives in Congress are disproportionately influenced by the wealthy few and by a variety of organized lobby groups. This is likely true, although the disparity is at least partly offset by the public's fascination with human interest stories and the tendency to root for the common folk. Ultimately, even the most cynical congresspeople can afford to be persuaded by money only when it comes to the topics that their constituents are fairly indifferent to.

Beyond the legislative and executive branches of the government, some distinct undertones of self-governance are present in the US judicial system, too. The country borrows from the traditions of British common law, rather than the civil law system utilized in much of continental Europe. It embraces the significance of legal precedent and emphasizes humanist values over the strict application of legal codes, with remarkably broad powers vested in the judges and in the juries of peers - up to the notion of jury nullification. Ultimately, the system seeks to limit the consequences of the fallibility of legislators, who often struggle to properly consider all the implications of the laws they pass; it trades it for the increased risk of fallible courts - who bring in their own subconscious biases into the mix.

For the next article in the series, click here.

July 06, 2015

Poland vs the United States: immigration

This is the eleventh article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

There are quite a few corners of the world where the ratio of immigrants to native-born citizens is remarkably high. Many of these places are small or rapidly growing countries - say, Monaco or Qatar. Some others, including several European states, just happen to be on the receiving end of transient, regional demographic shifts; for example, in the past decade, over 500,000 people moved from Poland to the UK. But on the list of foreigner-friendly destinations, the US deserves a special spot: it is an enduring home to by far the largest, most diverse, and quite possibly best-assimilated migrant population in the world.

The inner workings of the American immigration system are a fascinating mess - a tangle of complex regulation, of multiple overlapping bureaucracies, and of quite a few unique social norms. The bureaucratic machine itself is ruthlessly efficient, issuing several million non-tourist visas and processing over 700,000 naturalization applications every year. But the system is also marred by puzzling dysfunction: for example, it allows highly skilled foreign students to attend US universities, sometimes granting them scholarships - only to show many of them the door the day they graduate. It runs a restrictive H-1B visa program that ties foreign workers to their petitioning employers, preventing them from seeking better wages - thus artificially depressing the salaries of some citizen and permanent resident employees who now have to compete with H-1B captives. It also neglects the countless illegal immigrants who, with the tacit approval of legislators and business owners, prop up many facets of the economy - but are denied the ability to join the society even after decades of staying out of trouble and doing honest work.

Despite being fairly picky about the people it admits into its borders, in many ways, the United States is still an exceptionally welcoming country: very few other developed nations unconditionally bestow citizenship onto all children born on their soil, run immigration lotteries, or allow newly-naturalized citizens to invite their parents, siblings, and adult children over, no questions asked. At the same time, the US immigration system has a shameful history of giving credence to populist fears about alien cultures - and of implementing exclusionary policies that, at one time or another, targeted anyone from the Irish, to Poles, to Arabs, to people from many parts of Asia or Africa. Some pundits still find this sort of scaremongering fashionable, now seeing Mexico as the new threat to the national identity and to the American way of life. The claim made very little sense 15 years ago - and makes even less of it today, as the migration from the region has dropped precipitously and has been eclipsed by the inflow from other parts of the world.

The contradictions, the dysfunction, and the occasional prejudice aside, what always struck me about the United States is that immigration is simply a part of the nation's identity; the principle of welcoming people from all over the world and giving them a fair chance is an axiom that is seldom questioned in any serious way. When surveyed, around 80% Americans can identify their own foreign ancestry - and they often do this with enthusiasm and pride. Europe is very different, with national identity being a more binary affair; I always felt that over there, accepting foreigners is seen as a humanitarian duty, not an act of nation-building - and that this attitude makes it harder for the newcomers to truly integrate into the society.

In the US, as a consequence of treating contemporary immigrants as equals, many newcomers face a strong social pressure to make it on their own, to accept American values, and to adopt the American way of life; it is a powerful, implicit social contract that very few dare to willingly renege on. In contrast to this, post-war Europe approaches the matter differently, seeing greater moral value in letting the immigrants preserve their cultural identity and customs, with the state stepping in to help them jumpstart their new lives through a variety of education programs and financial benefits. It is a noble concept, although I'm not sure if the compassionate European approach always worked better than the more ruthless and pragmatic American method: in France and in the United Kingdom, massive migrant populations have been condemned to a life of exclusion and hopelessness, giving rise to social unrest and - in response - to powerful anti-immigrant sentiments and policies. I think this hasn't happened to nearly the same extent in the US, perhaps simply because the social contract is structured in a different way - but then, I know eminently reasonable folks who would disagree.

As for my own country of origin, it occupies an interesting spot. Historically a cosmopolitan nation, Poland has lost much of its foreign population and ethnic minorities to the horrors of World War II and to the policies implemented within the Soviet Bloc - eventually becoming one of the most culturally and ethnically homogeneous nations on the continent. Today, migrants comprise less than 1% of its populace, and most of them come from the neighboring, culturally similar Slavic states. Various flavors of xenophobia run deep in the society, playing right into the recent pan-European anti-immigration sentiments. As I'm writing this, Poland is fighting the European Commission tooth and nail not to take three thousand asylum seekers from Syria; many politicians and pundits want to first make sure that all the refugees are of Christian faith. For many Poles, reasonable concerns over non-assimilation and extremism blend with a wholesale distrust of foreign cultures.

For the next article in the series, click here.

July 05, 2015

Poland vs the United States: crime and punishment

This is the tenth article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

Throughout much of its history, the United States has been a comparatively violent nation. From the famed lawlessness of the western frontier, to the brawling biker gangs, to the iconic Italian Mafia and the fearsome Mexican drug cartels, the thirst for blood has left a mark on the American psyche - and profoundly influenced many of the country's most cherished works of literary and cinematic art.

But sooner or later, a line gets drawn. And so, when a tidal wave of violent crime swept the nation in the late 80s, the legislators and the executive branch felt obliged to act. Many wanted to send a message to the criminal underworld by going after it with relentless and uncompromising zeal - kicking off the multi-decade War on Drugs and rolling out policies such as the three strikes law in California or stop-and-frisk in New York City. Others saw the root of all evil in the pervasive gun culture of the United States - successfully outlawing the possession or carry of certain classes of firearms and establishing a nation-wide system of background checks.

And then, in the midst of these policy changes, something very interesting started to unfold: the crime rate plunged like a rock, dropping almost 50% over the course of twenty years. But why? Well, the funny thing is, nobody could really tell. The proponents of tough policing and the War on Drugs tooted their own horns; but less vindictive municipalities that adopted programs of community engagement and proactive policing heralded broadly comparable results. Gun control advocates claimed that getting AR-15s and handguns off the streets made a difference; gun rights activists found little or no crime gap between the gun-friendly and the gun-hostile states. Economists pointed out that people were living better, happier, and longer lives. Epidemiologists called out the elimination of lead - an insidious developmental neurotoxin - from paints and gasoline. Some scholars have gone as far as claiming that easy access to contraception and abortion caused fewer children to be born into multi-generational poverty and to choose the life of crime.

Europe certainly provided an interesting contrast; the old continent, having emerged from two unspeakably devastating and self-inflicted wars, celebrated its newly-found pacifist streak. Its modern-day penal systems reflected the philosophy of reconciliation - abolishing the death penalty and placing greater faith in community relationships, alternative sentencing, and the rehabilitation of criminals. A person who served a sentence was seen as having paid the dues: in Poland and many other European countries, his or hers prospective employers would be barred from inquiring about the criminal record, and the right to privacy would keep the indictments and court records from public view.

It's hard to say if the European model worked better when it comes to combating villainy; in the UK, crime trends followed the US trajectory; in Sweden, they did the opposite. But the utilitarian aspect of the correctional system aside, the US approach certainly carries a heavy humanitarian toll: the country maintains a truly astronomical prison population, disproportionately comprised of ethnic minorities and the poor; recidivism rates are high and overcrowding in some penitentiary systems borders on the inhumane.

Untangling this mess is not easy; most Americans seriously worry about crime and see it as a growing epidemic, even if their beliefs are not substantiated by government-published stats. Perhaps because of this, they favor tough policing; reports of potential prosecutorial oversight - such as the recent case of a tragic homicide in San Francisco - tend to provoke broader outrage than any comparable claims of overreach. Similarly, police brutality or prison rape are widely acknowledged and even joked about - but are seen as something that only ever happens to the bad folks.

For the next article in the series, click here.

July 04, 2015

Poland vs the United States: the cutting edge of technology

This is the ninth article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

No matter what's your take on the United States, there is no denying that the country has been on the forefront of scientific and industrial progress for much of the past century. In that time frame alone, the nation's research institutions and corporations have made countless fundamental contributions to almost every single aspect of contemporary technology - from polymer science, to computing, to aviation, to medicine, to nuclear power, to space exploration, to communications, to modern warfare.

Given the country's track record of relentless innovation, one would expect its residents to be quick to embrace technological novelties and futuristic design trends. But when it comes to everyday living, I find that the opposite is often more true. Let's take banking: many of my Polish friends recoil in terror when they find out that the world's most sophisticated financial system still settles many private transactions by writing checks; that in stores, you usually swipe the magnetic strip and scribble your name on a piece of paper; or that sending a wire transfer usually involves a trip to your bank, a hefty fee, and waiting a couple of days.

For many of them, it must be equally perplexing to visit a typical well-off American home. Kitchens are a good example: in much of continental Europe, the standard of upscale kitchen architecture tends to revolve around sleek, sterile looks constructed out of flat panes of glass, steel, plastic, and concrete; the drawers and cabinets will cleverly blend in to reveal space-age appliances hidden inside. The kitchen is, in essence, the embodiment of technological progress and of modern design aesthetics.

In the US, the European school of design has gained some foothold in pricey downtown apartments targeted at the wealthy youth - but the dominant, all-American archetype looks nothing like it. Many of the newly-built houses will feature old-fashioned, bulky granite countertops and ornate but functionally basic colonial-style wooden carpentry; most of the fancy small appliances will feel like they were pulled straight out of the 30s, too. Decorative details, such as crown moldings, vaulted ceilings, and marble columns are thrown in to differentiate luxury developments from the housing available to the middle class. Elsewhere in the house, featureless top-loading washing machines and clunky upright vacuums are a common sight.

The contrast is interesting and difficult to explain; it's certainly not that Americans are Luddites: they are quick to take lead with many types of utilitarian technologies. The country pioneered and popularized everything from refrigerators, to air conditioning, to dishwashers, to automatic transmission, to smartphones, to microwaves. It's also not that the residents show special reverence to the traditions of the bygone days. Perhaps the utilitarian principle is key: it may be that consumers judge many of their purchases based the utility and lasting value of the durable goods, more than their novelty or the image said goods may project.

If so, the observation would fly in the face of the country's reputation for rampant consumerism, a stereotype frequently contrasted with the meditated sophistry of Europe. But then, the conclusion may be overly broad: even within the United States, there are many interesting differences in how tangible goods are used to signal personal wealth. In Los Angeles or Miami, just like in much of Europe, luxury sports vehicles are a widely accepted symbol of affluence. In Silicon Valley, the practice is frowned upon, with many of the dot-com millionaires living in unassuming homes and driving fuel-efficient cars. Perhaps this is a matter of social conscience; perhaps of having different priorities; and perhaps simply of fearing that they would be vilified by the society.

For the next article in the series, click here.

June 29, 2015

Poland vs the United States: suburban sprawl

This is the eighth article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

If you live in any other western country, your perception of the United States is bound to be profoundly influenced by Hollywood. You may think you're immune to it, but you are not: sure, you can sneer at the ridiculous plot holes or the gratuitous patriotism in American blockbusters - but the establishing shots of high-rise cityscapes of Manhattan or Los Angeles will be seared into your mind. These images will color your expectations and your understanding of the country in more ways than you may expect.

Because of this phenomenon, urban dwellers from Europe who come to visit the US may be in for a surprise: the country will probably feel a lot more rural than they would have thought. They will get to marvel the grand cities and the iconic skyscrapers; but chances are, this scenery will quickly morph not into the familiar urban jungle of massive apartment blocks seen throughout much of Europe, but into the endless suburban sprawl of single-family homes and strip malls.

For most Americans, this vast, low-density suburban landscape is the backdrop of their everyday lives. Take San Francisco: just 800,000 people live in the city proper. The San Francisco Bay Area, the home to 8 million residents and the location of the largest and most influential tech hub in the world, is nothing more than an enormous stretch of greenery peppered with detached homes, unassuming two-story office buildings, and roadside car dealerships. Heck, even New York City, by far the largest urban conglomeration in America, is just a blip on the radar compared to the colossal suburban sprawl that engulfs the region - stretching all the way from Massachusetts to Washington D.C.

The raw numbers paint a similar picture: in Poland, the average population density is around 125 people per square kilometer; in the more densely populated Germany, the figure is closer to 220. In comparison, with fewer than 35 people per km2, the United States comes out looking like a barren wasteland. The country has many expanses of untouched wilderness - and quite a few rural regions where the residents get by without as little as a postal address, a nearby fire station, a police department, or a hospital.

Awareness of the predominantly suburban and rural character of much of the US is vital to understanding some the national stereotypes that may seem bizarre or archaic to urban-dwelling Europeans. It certainly helps explain the limited availability of public transportation, or the residents' love for rifles and gas-guzzling pickup trucks. The survivalist "prepper" culture, focused on self-sufficiency in the face of disaster, is another cultural phenomenon that although seemingly odd, is not just pure lunacy; in the past few decades, millions of Americans had to evacuate or dig in in response to hurricanes, wildfires, earthquakes, or floods.

The stark difference between urban and rural living can also make it easier to grasp some of the ideological clashes between the big-city liberal progressives and the traditionally conservative dwellers of the so-called "flyover states". Sometimes, the conservatives are simply on the wrong side of history; but on some other occasions, the city-raised politicians, scholars, and journalists are too eager to paint the whole nation with the same brush. Take something as trivial as car efficiency standards: they will rub you one way if you take subway to the office and drive your compact car to the grocery store; and another if you ever needed to haul firewood or construction materials on the back of your Ford F-150.

For the next article in the series, click here.

June 28, 2015

Poland vs the United States: friends & acquaintances

This is the seventh article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

Cultural stereotypes are a dangerous and corrosive thing. They teach us that Poles are a tribe of thieving simpletons; or that Americans are arrogant, violent, and obese. And that's just the ethnicities that get off easy: the perception of blacks, Muslims, or European Jews can be far more vicious, often serving a pretext for violent hate crime.

At the same time, there is no denying that certain unique archetypes are etched into the fabric of every society. I'd also posit that when cultures come into contact with each other, there is an uncanny valley effect at play: the more similar the nations are, the easier it is for travelers to instinctively pick up the subtle variations - and to misread them as the personality quirks of the people they interact with.

For Poles who settle in the United States, the most striking contrast of this sort must be the persistence with which Americans want to engage in oddly personal small talk: you will be always greeted with "how are you?", be it by the cashier at a grocery store, by your mailman, by the park ranger met at a trail, or by the waiter serving your food at a restaurant. The social expectation is to share short pleasantries or announce a brief piece of good news. But if your answer is overly specific or focuses on a negative event, you may be given quizzical looks and the conversation will stall.

To many of my compatriots, the exchange - lacking any apparent purpose - feels uncomfortable and insincere. I try not to look at it in a cynical way: the upbeat chit-chat, repeated over and over again, can probably make your day a bit better and a tad more fun. This constrained form of communication also provides something to build on the next time you see that person, even if every individual interaction is necessarily non-committal and brief.

Another explanation for the forced positivity may have to do with the pervasive can-do spirit at the core of the American culture. The national ethos of self-determination and unconstrained social mobility flies in the face of the daily struggles of disadvantaged citizens - but it remains a fundamental part of the cultural identity of the United States. The American Dream manifests itself everywhere, from the country songs of the Midwest to the high-tech entrepreneurship of the Silicon Valley. Your friends, coworkers, neighbors, and even complete strangers are there to support you when true calamity strikes - but dwelling on everyday mishaps is almost universally seen as a weakness that one needs to overcome in order to succeed in life.

In this regard, the Polish culture is strikingly different. After hundreds of years of political repression and foreign control, Poles have developed a colorful tradition of sarcastic humor and idle lamentation. This coping mechanism functions to this day: to a Pole, being asked about your day is seen as an invitation to air all the petty grievances; you wouldn't expect a friend to smile, exclaim "I'm doing great!", and move on. Complaining about politics or work is how you build rapport with your peers. In fact, being overly upbeat or talking about professional success or accomplishment is likely to be met with suspicion or scorn. If you're a successful entrepreneur, you will probably open by complaining about your dealings with the Polish equivalent of the IRS.

In many ways, the Polish approach to chit-chat is more genuine and less rigid. At the same time, I feel that the negativity comes at a price; meeting a cranky clerk at a store sets the tone for the remainder of your day. The constant pessimism can also dampen some altruistic instincts: relatively few people in Poland get engaged in their communities or dedicate themselves to other forms of civic service. It is more accepted to just complain about the ways things are.

Interestingly, in the United States, the boundaries that govern the conversations with complete strangers also extend into the workplace. When interacting with casual acquaintances, sarcasm is seen as jarring, while petty grumbling is perceived as an off-putting and unproductive personality trait. Off-color humor, widely tolerated in Poland, is usually inappropriate in white collar environments; doubly so if it comes at the expense of women, immigrants, or other disadvantaged social groups.

Some Europeans characterize the workplace etiquette in the US as political correctness run amok. There are situations where political correctness can stifle free speech, but I don't think it's one of them; for most part, not hearing political rants or jokes about blondes or Jews just makes the world a bit better, even if the comments are uttered with no ill intent. Violating these rules will not necessarily get you in trouble, but in a culturally diverse society, it can make it harder to find new friends.

For the next article in the series, click here.