Before Testing

Before running any tests, you’ll need to set some goals. The first and most important goal is the number of simultaneous users the application must be able to service at any time. We’ll call this SimulGoal. If your application is already deployed, you can get a sense of this number from the current statistics. Otherwise, you need to do a little magic mixed with some marketing projections. Every application is going to be different, and there are going to be loads of variables to determine the right number. This is an interesting topic in itself that we hope to cover sometime in the future, but one very rough rule of thumb that you may use is that for an application with a million Monthly Active Users (MAU) you’ll need to be able to service a couple thousand simultaneously (YMMV!!).

The next goal you’ll need to define is the number of simultaneous users that can be serviced by a single application server, we’ll call this ServerGoal. Coming up with ServerGoal can, and has been be done lots of different ways:

• Bottom up: Figure out the computational power of a single application server. Then, test and determine how much computational power is necessary to process a single transaction. Do some arithmetic, and you’ve got the number.
• Top down: Find out your budget for server equipment, and determine the number of servers you’ll be able to run for this application. Divide SimulGoal by this number to determine ServerGoal. This may not have any basis in reality, but it’s a start.
• Typical: Pull a number out of one of your body’s orifices. Note: this is the method that most people use, so don’t be ashamed to admit it.

Note that ServerGoal is a “soft” goal that can and should change as more is learned about how the application responds under load.

Okay, now that you’ve set some goals, let’s start testing:

Test #1

The first thing you’ll need to decide for your first test is how many virtual users it should contain. Of course your instinct is going to tell you to try and run SimulGoal users through a full deployment infrastructure. Resist this instinct as it is a very bad idea. Why? Well, let’s look at the situation through a Zen lens. You’re not really testing any hypothesis. When you run the test, it is sure to fail (because the first one always fails). When it does fail, what have you learned? Nothing – the test has ended as a true failure and you’ve wasted your time.

A much better approach is this: Define a “Minimal Viable Server Infrastructure” (MVSI) which is the smallest infrastructure you can run and still service requests. Often this simply means a single application server with an optional database server if necessary. If you typically use a load balancer, throw one into the mix.

Now, run a test against this MVSI with ServerGoal+(a few extra) virtual users. Our hypothesis will be that when the test reaches ServerGoal users, the application infrastructure will begin to fail. When the test completes, you’ll definitely have learned something, namely ServerReal which is the actual number of simultaneous users a single server can support. What to do next will be determined by the state you’re in:

ServerReal < ServerGoal

This is probably where you’re going to be. You’ll need to rethink how you came up with ServerGoal (e.g. did you use the orifice method?) and perhaps modify it. Maybe ServerReal really is the limit that your application can handle. If you’re okay with that, set ServerGoal = ServerReal and continue onto Test #2.

However, if you’re thinking “Something’s wrong – this server must be able to handle more users than that”, then you’re in for some work. Look through the server logs and try to find some evidence of something the application is doing that isn’t efficient. Look through the code and see if you can find it there. Look at how the servers that run the application are configured, and see if they can be tweaked to get better performance. In short. come up with some idea for how to squeeze more performance out of the system.

Once you do all of that, you now have a hypothesis for how to improve performance. Test this hypothesis by re-running Test #1 and see if anything gets better. Continue this iteration cycle until you get to a value of ServerGoal that just can’t get any better.

ServerReal > ServerGoal

Wow, you’re in really good shape. First, go talk to your developers and thank them for writing such a great application. Next, reset ServerGoal = ServerReal, and if you’re satisfied with that, continue to Test #2, otherwise iterate and try to get even better.

In the next post, we’ll talk about what happens in Test #2 where we’ll determine just how scalable the application is.

Zen tells the story of a motorcycle trip, but at its core, is really about the search for the meaning of quality. It also contains some great discussions about the Scientific Method. You remember the Scientific Method, that process you learned in the seventh grade that you didn’t see any use for it. Well, it turns out that this process has loads of meaning for software testing, and it’s worth looking into.

For me, the most insightful passage of the whole book is this:

The TV scientist who mutters sadly, “The experiment is a failure; we have failed to achieve what we had hoped for,” is suffering mainly from a bad scriptwriter. An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question, when the data it produces don’t prove anything one way or another.

What Pirsig is really getting at here can be applied to software testing, where each test that you run is (or really should be) an experiment. When viewed this way, Pirsig’s words have real meaning. Every test that you run should have a purpose, which is, to test a hypothesis. You should be able to predict the results of the test before you run it, and most importantly, you should always learn something  from the outcome of a test regardless of the results.

Software testing, especially load testing is pretty resource intensive. Each test usually requires someone to run the test, someone to run the computing infrastructure, perhaps a DBA to watch the database (often the first thing to break), a developer or two, and perhaps a few others depending on the application. You need a deployment-quality infrastructure to run the application under test, and a much larger infrastructure to run the load testing tool. Beforehand, you need plenty of planning, and afterwards you need some time to analyze the data. All of this needs coordination, and most importantly time. In short, running a single load test is a fairly complex thing to do, and is pretty expensive. Expensive, not in terms of money, but in time, schedule and focus.

And that’s where Zen comes in. Before you invest that amount of time and effort into running your test, it’s critically important to ensure that you don’t end up like that TV scientist. A test where you don’t learn anything afterwards is a waste. Since load testing is often the last thing done before deployment, it is really important not to be wasteful, and to always complete a test cycle knowing more than when you started.

In the next post, we’ll discuss some concrete strategies you can take to plan your own load tests.

In this case, the application was Facebook’s newly released Facebook Stories application. This application allows users to tell the story about how they got onto Facebook and how it changed their life. The application became overwhelmed and eventually crashed due to the load placed on it by …

… rabid Justin Bieber fans

That’s right, I just wrote the most unlikely string of words that can show up in a blog about technology. The crash was reported by Justin himself, ironically on Twitter.

 Kids, don’t let this happen to your application. Before deployment, load test it with a tool like FriendRunner.

Update

Ahhh geez! Okay, this will teach me to trust Justin Bieber. All Facebook is now reporting that the rabid Justin Bieber fans never crashed the Facebook Stories app. Instead, he was asked to remove the tab for the application from his page because he was using it to solicit stories about himself instead of using it for the purpose it was intended for (How has Facebook changed your life?).

Okay, I was fooled, but the original story did have a believable provenance. I originally heard it from Randi Zuckerberg, who after all is the sister of …. She got it from All Facebook, which is a reliable source of great FB information.

I apologize to you dear reader, and will now make the commitment that this blog will never ever disseminate any more news about Justin Bieber, regardless of what it is or where it comes from.

Today marks the day when I would have been inducted into the IBM Quarter Century Club had I stuck it out in IBM all of these years, and I would have earned the right to put this cool logo on my business cards. But the truth is, I left back in 1996 to join a small startup where I learned that I’d be much happier as a small fish in a small pond rather than a small fish in a big pond. Twenty five years is a long time, and I learned a few things along the way:

• IBM was a great place to start out.
  • I got to work with some really great people, and learned about developing software professionally in a great environment, something that they don’t (and really can’t) teach you in school. IBM allowed me to work on lots of different and interesting projects, paid for my Master’s degree, moved me halfway across the world, and actually paid me a salary the whole time. 
•  Leaving IBM was a good idea.
  • Towards the end of my tenure at IBM, the company wasn’t doing so well, so to turn things around management created this mass hysteria known as Market Driven Quality (MDQ). We were told that it “wouldn’t be one of those Quality programs where you get a coffee mug, and that’s the end of it”. Big mistake. It shouldn’t have existed past the coffee mug stage, but it ruled our lives for a few years. You don’t hear about it anymore, and there’s not even much on the Internet about MDQ, but an interesting description can be found here. One thing it did do was to start to convince me that IBM may not be the place for me. Fortunately, after John Akers was removed as CEO, much of this nonsense was gone and the company really did recover. Except …. without me.
• It used to be really hard to bring new ideas into IBM.
  • Back in 1992 I worked on an IBM product called BookManager, an early attempt at softcopy books (think Adobe Reader, but  a lot less sexy). It looked terrible, but had a very sophisticated automatic indexing and search facility so you could search through large stacks of IBM manuals. A friend of mine and I wrote a proposal to apply this indexing/search technology to the nascent Internet, so you could search for information there (although we weren’t sure why anyone would want to do this …). This is pre-Google, pre-Yahoo, pre-Mosaic! The management we spoke with asked compelling questions such as “What IBM platform does this ‘Internet’ run on – MVS, VM, or OS/2?”. That was the end of that, although I did read somewhere that those Yahoo and Google guys did fairly well for themselves with “Internet search”.
• There’s always something new to learn.
  • Over the past 25 years, I can’t even recall the number of languages, operating systems, scripting languages, IDEs, development tools, and protocols I’ve had to learn and have since forgotten. But that’s good as it keeps you fresh. Last week I learned Jersey, a cool system for writing RESTful Web services in Java. But I still refuse to learn Rails.
• Being a software developer is a cool thing to do.
  • I’ve been able to do it professionally in three different countries. I’ve started my own company. I get to work on something I like doing, and feel like I’m making a contribution to the world. I even wrote a compiler despite the fact that I refused to take Compiler Theory in university because I thought “How many compilers does the world really need anyway?”.
• We’ve come a long way.
  • My first job in IBM was working on this monstrosity for the US Air Force with hundreds of other developers. The team I worked with was involved with something called “packet switching”, kind of cutting edge back then but so ubiquitous today that you never even hear the term used. Sort of like turning on the tap and having clean water semi-miraculously pouring out, packet switching does the same with bytes. The fact that it’s so invisible and pervasive reminds me of how far we’ve come.

I have loads more stories to tell, but also lots of real work to do, so I better get on that. Let’s hope that the next 25 years is as good as the first.

To learn more about their experience with Friendrunner, the only tool available to properly load test Facebook applications, click here.

That’s right. Have you ever thought about how Facebook tests its own product? Well wonder no more as the good people at Quora have already asked that question, and Steven Grimm, Facebook’s test engineering tech lead has provided quite a good answer.

A few interesting take-aways:

• For a company that’s so grounded in PHP, it’s interesting to see that they’re so heavily invested in Watir, a tool from the Ruby world.
• “Facebook has no dedicated QA team.” Now for many people, this will come as no surprise. For others from a more traditional development background, no dedicated QA for something as large and mission critical as Facebook seems like a really bad idea.
• They don’t talk about load testing. From what we can tell, they probably don’t even do it. In a lot of respects, a proper load test of Facebook is probably far more trouble than it’s worth.

All Facebook reports that Power Ventures, creator of the social network aggregator Power.com is being sued by Facebook for violating their Terms Of Service. Power.com allows users to log into multiple social networking sites simultaneously – you log into their service and they log into the sites automatically. The problem is that Facebook has a very clear prohibition in their TOS against accessing their service by automated means, which is exactly what Power.com is doing. How Facebook sees this as a criminal act seems kind of crazy, but that’s exactly what’s happening.

So what does this have to do with load testing? Everything! The Facebook TOS (as well as a few other technical hurdles) prevent you from hooking a standard load testing tool up to Facebook to load test your newly created Facebook app. Kids, don’t try this at home, lest you get arrested. Better stick with a dedicated tool like FriendRunner that allows you to load test your Facebook app outside of the Facebook infrastructure.

Facebook Canvas applications

To understand why PLT is a bad idea for Canvas applications,  we must go back and understand the Canvas architecture. Users interact with Canvas applications by using a browser connected to the Facebook server which acts as a middleman and requests data from the app. The app may call APIs that interact with the Facebook server, and then ultimately return an FBML file to Facebook which renders it into HTML for return to the users bowser. The problem is that if we only look at PLT by measuring response time at the browser, we have all of this extra stuff being counted in the timing:

• Network latency between the browser and Facebook
• Time required by Facebook to figure out which application must be called, authenticating the request, etc.
• Time to render the FBML into HTML
• Time spent at the Facebook server servicing API calls

These “extraneous” functions will take an unpredictable amount of time depending on, among other things, the current load at the Facebook server. This will make it hard, if not impossible, to positively correlate the timings of two different test runs. Since one of the prime motivators of load testing is to see the effects of changes to the code on performance, metrics like these will ultimatly prove to be of little use.

Facebook Connect applications

Using PLT for Facebook Connect applications is a little bit better. Users interact with Facebook Connect applications directly, and do not need the Facebook server to act as the middleman. The only extraneous functionality being counted in the timing is the time spent at the Facebook Server responding to API calls. If there are many API calls being made, this will cause a bit of randomness in the metrics.

Application Response Times

In contrast to PLT, creating metrics based on the Application Response Time is much more meaningful and useful, although harder to calculate. Application Response Time should calculate the amount of time spent at the application and only the time spent at the application. Unfortunately, in contrast to PLT there’s no simple way to get the Application Response Time – you’ll need to do a little bit of work to get at it.

To calculate this, you’ll need to create a stopwatch function which starts when your application is called, and stops when the application completes the request and returns an FBML file to the caller. In between, the stopwatch needs to be paused when a Facebook API is called. Finally, those of you using PHP as your development platform should check out the microtime() function as it’s much more accurate than the standard time() function. If your application is able to keep up with all of the bookkeeping, you’ll have metrics that can accurately show you the effects on application performance from any changes you make.

For those of you who don’t want to be bothered with keeping track of all of these timing calculations, please note that FriendRunner does all of this for you automatically.

Disclosure Agreement

That’s right – what I’d really like is to have an agreement with everyone we work with that says that we’re legally bound to talk about each other. Something that forces us to tell at least 100 people about our  partner in the hopes that at least 1 or 2 of them will be interested enough to investigate further. Write blog posts, tweet, put links on the website, do whatever else is necessary to tell the world that we believe that those guys are doing something so innovative, valuable, and cool,  that you should also check them out. Isn’t that what all of this social media stuff is all about anyway?

So what do you think about this? Is this a good idea or just really goofy? Are there any lawyers out there forward thinking enough to even try to draft such an agreement? If so, let us know at admin@testfacebook.com.

Until then, I guess the “Like” button will have to do.

Postscript

Two days after publishing this I received an email containing yet another NDA to sign. On the one hand, it’s very positive that other companies wish to talk to us. However, it would be even nicer if they also talked about us to others.

While I was in San Francisco for f8, I had the good fortune to attend an informal meeting of Facebook developers to discuss the changes to the Platform that were announced at f8. It was graciously hosted at the offices of one of the leading Facebook development companies in the city. They were beautiful offices high up in an historical building with great views, and more Red Bull than a developer could drink.

While there, I noticed a copy of the iconic Kernighan and Ritchie (K&R) book on the C language sitting on a desk. What transpired was a short conversation between myself and one of the employees of the company:

Me  : Very impressive, someone still reads K&R.
Him : <Blank Stare>
Me  : The Kernighan and Ritchie book on C – it’s a classic
Him : <Blank Stare>
Me  : You should read this book, it’s so beautifully written. Look how thin it is, and the whole language is thoroughly described with examples and exercises.
Him : Yeah, the Rails documentation is much thicker
Me  : <Blank Stare>
Him : That’s our CTO’s desk – it’s probably his book

Okay, so the point here is that I’m realizing that you can actually get a degree in Computer Science without ever reading K&R. Without learning C. Without really understanding what goes on at all levels inside the computer without it all getting abstracted away. I was starting to feel bad that this whole drag-and-drop Rubyization of the world was creating developers who weren’t grounded in the basics. Who didn’t deeply understand how to properly analyze an algorithm. And who probably didn’t read stuff from guys like Joel Spolsky who kvetched about it all.

But then something else crossed my mind. The reason I love C and K&R so much is because it works on so many levels. You can use it to develop (practically) on the bare metal. You can use it to write customer-facing applications. You can use it to build a server. However, it’s becoming true that the higher off the metal you get, the harder it is to use C (or even C++). Web apps, Application Servers, Multi-tiered architectures, virtualization – all of these things are so much easier to do with more “modern” languages. And you know what, I’m okay with that.

So here’s my wish: Go get a copy of K&R and read it. I used to own 3 copies: one for work, one for home, and one for lending out. Even if you never plan to use C, please read it. It’s the best technical documentation you’ll ever read, and something you should aspire to. And the reason this is true is that the language is robust and, most importantly, simple.

Which brings me to the new Facebook Graph API and the way it simplifies everything. I think we should all say Hurray. K&R reminds us that simple does not have to mean not powerful. And what the Facebook guys did with this API reminds me of what Dr. Kernighan and Dr. Ritchie had done previously. Go read the book so you too can be inspired to continue building things that are both powerful and simple.

Postscript

For sheer entertainment value, you can’t beat the first version of the Java Language spec. In particular, check out the index entry for index entries to see all of the great references they included, and see how many you can understand. Their description of multiple inheritance (Section 20.3.5) is inspired, obscure, and absolutely brilliant.