Zen and the Art of Facebook Application Load Testing (Part 4)

25th We’ve finally reached the promised land – this is the final post about Robert Pirsig’s book Zen and the Art of Motorcycle Maintenance . Thanks for sticking it out to those of you who’ve been following so far.

In the last post, we ended with an application that we can call nominally scalable. To get to this point we proved that the application scales well when going from one to two servers. In the process, we’ve figured out some important information such as how many virtual users a single instance of the application server can handle. Time now for the big test.

Run Test #3

This is pretty easy. Set up a deployment-quality infrastructure. Try and run SimulGoal users through it (remember that SimulGoal is the number of simultaneous users that the application must be able to support). See what happens.

Now some astute readers will recognize this as the same test that we warned about not doing back in Part 2. Aha, but things have changed now since we’ve followed a methodical plan where every step can either pass or fail, but will always teach us something about the system so that we can continue and be productive. We know a lot about the system at this point, so even if this test fails, it shouldn’t be too hard to hypothesize why, and then run another test to prove/disprove the hypothesis. In a short time you should be able to locate any problems still remaining, fix them and enjoy success.

And that’s the real point that Pirsig and Zen try to teach us – never finish a test where you scratch your head not knowing what happened. Always be making some progress. To do that, plan your tests out carefully, know what hypotheses you’re tryng to prove, and know which direction to go if the test passes/fails.

Conclusion

It’s taken us four posts to fully explain this methodology that we’ve developed while testing real Facebook applications. We’d love to hear your feedback about what you think of this, both good and bad. How would you modify this to make it better?