Zen and the Art of Facebook Application Load Testing (Part 1)

Zen_motorcycle Many years ago when I was an undergrad in college, I had lots of pre-med friends who took Biochemistry. One of their assigned texts was Robert Pirsig’s Zen and the Art of Motorcycle Maintenance, something that didn’t make any sense to me at the time. It was only several years later after I read the book for myself that I understood why this book should be assigned to budding scientists.

Zen tells the story of a motorcycle trip, but at its core, is really about the search for the meaning of quality. It also contains some great discussions about the Scientific Method. You remember the Scientific Method, that process you learned in the seventh grade that you didn’t see any use for it. Well, it turns out that this process has loads of meaning for software testing, and it’s worth looking into.

For me, the most insightful passage of the whole book is this:

The TV scientist who mutters sadly, “The experiment is a failure; we have failed to achieve what we had hoped for,” is suffering mainly from a bad scriptwriter. An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question, when the data it produces don’t prove anything one way or another.

What Pirsig is really getting at here can be applied to software testing, where each test that you run is (or really should be) an experiment. When viewed this way, Pirsig’s words have real meaning. Every test that you run should have a purpose, which is, to test a hypothesis. You should be able to predict the results of the test before you run it, and most importantly, you should always learn something from the outcome of a test regardless of the results.

Software testing, especially load testing is pretty resource intensive. Each test usually requires someone to run the test, someone to run the computing infrastructure, perhaps a DBA to watch the database (often the first thing to break), a developer or two, and perhaps a few others depending on the application. You need a deployment-quality infrastructure to run the application under test, and a much larger infrastructure to run the load testing tool. Beforehand, you need plenty of planning, and afterwards you need some time to analyze the data. All of this needs coordination, and most importantly time. In short, running a single load test is a fairly complex thing to do, and is pretty expensive. Expensive, not in terms of money, but in time, schedule and focus.

And that’s where Zen comes in. Before you invest that amount of time and effort into running your test, it’s critically important to ensure that you don’t end up like that TV scientist. A test where you don’t learn anything afterwards is a waste. Since load testing is often the last thing done before deployment, it is really important not to be wasteful, and to always complete a test cycle knowing more than when you started.

In the next post, we’ll discuss some concrete strategies you can take to plan your own load tests.