Load Test your Facebook Application!!

Once again, the need to load test your Facebook application before deployment has been proven out in the wild. Another company has seen their application crash and burn under a load that it either wasn’t expecting, or unable to deal with. The interesting part of this story, which seems to get told over and over, is the company that was the victim of the crash and the source of the load.

In this case, the application was Facebook’s newly released Facebook Stories application. This application allows users to tell the story about how they got onto Facebook and how it changed their life. The application became overwhelmed and eventually crashed due to the load placed on it by …

… rabid Justin Bieber fans

That’s right, I just wrote the most unlikely string of words that can show up in a blog about technology. The crash was reported by Justin himself, ironically on Twitter.

 Kids, don’t let this happen to your application. Before deployment, load test it with a tool like FriendRunner.

Update

Ahhh geez! Okay, this will teach me to trust Justin Bieber. All Facebook is now reporting that the rabid Justin Bieber fans never crashed the Facebook Stories app. Instead, he was asked to remove the tab for the application from his page because he was using it to solicit stories about himself instead of using it for the purpose it was intended for (How has Facebook changed your life?).

Okay, I was fooled, but the original story did have a believable provenance. I originally heard it from Randi Zuckerberg, who after all is the sister of …. She got it from All Facebook, which is a reliable source of great FB information.

I apologize to you dear reader, and will now make the commitment that this blog will never ever disseminate any more news about Justin Bieber, regardless of what it is or where it comes from.

Arrested for load testing your Facebook app??

Yes, it sounds crazy that you could get arrested for load testing a Facebook application, but that’s the implication for what’s been reported on All Facebook.

All Facebook reports that Power Ventures, creator of the social network aggregator Power.com is being sued by Facebook for violating their Terms Of Service. Power.com allows users to log into multiple social networking sites simultaneously – you log into their service and they log into the sites automatically. The problem is that Facebook has a very clear prohibition in their TOS against accessing their service by automated means, which is exactly what Power.com is doing. How Facebook sees this as a criminal act seems kind of crazy, but that’s exactly what’s happening.

So what does this have to do with load testing? Everything! The Facebook TOS (as well as a few other technical hurdles) prevent you from hooking a standard load testing tool up to Facebook to load test your newly created Facebook app. Kids, don’t try this at home, lest you get arrested. Better stick with a dedicated tool like FriendRunner that allows you to load test your Facebook app outside of the Facebook infrastructure.

Facebook Testing: Which metric is important?

Many people ask us which metric is best to use when performing a load test of a Facebook application. Common wisdom says that Page Load Time (PLT) works for web applications: it’s easy to calculate, seemingly applicable, and everyone understands what it means. However, it isn’t so useful in the Facebook world since it contains so much extraneous information that makes it hard to analyze anything with it.

Facebook Canvas applications

To understand why PLT is a bad idea for Canvas applications,  we must go back and understand the Canvas architecture. Users interact with Canvas applications by using a browser connected to the Facebook server which acts as a middleman and requests data from the app. The app may call APIs that interact with the Facebook server, and then ultimately return an FBML file to Facebook which renders it into HTML for return to the users bowser. The problem is that if we only look at PLT by measuring response time at the browser, we have all of this extra stuff being counted in the timing:

• Network latency between the browser and Facebook
• Time required by Facebook to figure out which application must be called, authenticating the request, etc.
• Time to render the FBML into HTML
• Time spent at the Facebook server servicing API calls

These “extraneous” functions will take an unpredictable amount of time depending on, among other things, the current load at the Facebook server. This will make it hard, if not impossible, to positively correlate the timings of two different test runs. Since one of the prime motivators of load testing is to see the effects of changes to the code on performance, metrics like these will ultimatly prove to be of little use.

Facebook Connect applications

Using PLT for Facebook Connect applications is a little bit better. Users interact with Facebook Connect applications directly, and do not need the Facebook server to act as the middleman. The only extraneous functionality being counted in the timing is the time spent at the Facebook Server responding to API calls. If there are many API calls being made, this will cause a bit of randomness in the metrics.

Application Response Times

In contrast to PLT, creating metrics based on the Application Response Time is much more meaningful and useful, although harder to calculate. Application Response Time should calculate the amount of time spent at the application and only the time spent at the application. Unfortunately, in contrast to PLT there’s no simple way to get the Application Response Time – you’ll need to do a little bit of work to get at it.

To calculate this, you’ll need to create a stopwatch function which starts when your application is called, and stops when the application completes the request and returns an FBML file to the caller. In between, the stopwatch needs to be paused when a Facebook API is called. Finally, those of you using PHP as your development platform should check out the microtime() function as it’s much more accurate than the standard time() function. If your application is able to keep up with all of the bookkeeping, you’ll have metrics that can accurately show you the effects on application performance from any changes you make.

For those of you who don’t want to be bothered with keeping track of all of these timing calculations, please note that FriendRunner does all of this for you automatically.

Load testing Facebook apps (Part 2)

In Part 1 of this article, we discussed that one of the reasons load testing Facebook apps was difficult was because Facebook prohibits the use of automated tools. An obvious question is “Why does Facebook do this?”.

The simple answer: It protects you, the developer

Think for a moment what would happen if Facebook did not prohibit the use of automated tools to access their servers. Well for one, we’d be able to easily carry out load tests of our applications. This is good and builds value for your application as well as indirectly building value for the entire Facebook Platform.

However, it also opens up a world of possibilities to the vast armies of hackers and spammers who’d use this to their advantage. What could they do? Let’s think:

• They could launch Denial-of-Service attacks and crash your servers. Does your application have competition? Would your competition benefit if your (obviously superior) app was seen as unstable? Let the attacks begin!
• Mischief. Facebook is now viewed as a prime target for all sorts of miscreants. Hardly a week goes by that we don’t hear about another type of attack launched at Facebook users, applications, or Facebook itself. Of course the methods by which most attacks are typically launched is by the use of automation.
• Building up your own numbers. Actually not really attack – you could use a botnet to artificially inflate your own active user numbers. MAUs are used for a variety of reasons (which all ultimately relate to money). If MAU counts also included MAARs (Monthly Active Automated Robots), then all counts become meaningless. the entire system unwinds, and everyone loses.

So let’s say that as a good developer you’d like to protect your application against a Denial-of-Service attack. You install an Intrusion Detection System, but wait - this won’t work in a Facebook environment! Remember that the Facebook Platform architecture is built so that all of the user traffic coming into your application is funneled through one place – the Facebook servers. Any attempts you make to try to distinguish good access from bad access will be foiled by the fact that the bad guys are on the other side of the Facebook servers from you. Your app has no way to distinguish good users from botnet users, it must rely on Facebook to do it for you. This simply pushes the detection of good vs. bad access out to the Facebook servers, so we can ask: “How can Facebook detect a DOS attack on my app?”

Facebook avoids Denial-of-Service attacks by using the same techniques as any other web site. This, of course, will preclude you from running an automated load test since it will simply look like a very basic (not even distributed) Denial-of-Service attack. Instead of trying to distinguish the good load tests from the bad attacks, Facebook simply says “Do not access our servers with automated tools or we’ll cut you off”. They aren’t trying to stifle our ability to get things done, but rather, they’re protecting us from a whole lot of other pain.

In the next part of this article, we’ll discuss some ideas that can be used to load test your Facebook app.

Load testing Facebook apps (Part 1)

Calvin's dad on load testing

Load testing Facebook applications is really important. Usually the whole point of deploying onto Facebook is to leverage the social graph and go viral. Developers want their applications to go viral, but there’s a nagging concern about what will happen if they do. Is the app as scalable as you think? Are the hosting choices correct? What’s the cost of deploying and crashing?

The way to alleviate these concerns is to load test your app, but how do you do that in a Facebook environment? The simple answer is that it’s very hard due to two issues central to Facebook:

Issue1: You must control enough Facebook users

Testing your application with one or two simultaneous users is easy: simply use your own Facebook account and call up some friends and have them do the same. But running a test with hundreds or even thousands of simultaneous users is much more difficult. Where do you get the users from? That answer isn’t so simple. You can say “I’ll just go into Facebook and define a few hundred ‘phony’ people that I can control”. While this may seem like a good idea, it isn’t:

• It violates the Facebook Terms of Use (section 4.1) which prohibits you from creating phony accounts. When Facebook finds your phony account, they will delete it.
• Real users have friends, become fans of pages, post updates, and hundreds of other things that Facebook allows you to do. This creates the richness of the social graph, and applications often need to take these things into account. Unless you’re willing to spend an inordinate amount of time “humanizing” your phony user army, your tests with them will not be realistic.
• While Facebook does provide the concept of a Test Account to developers to test with, these accounts are very limited in what they can do so are not of very much use for load testing.

Issue 2: Load testing must be automated

Load testing web applications is hard. Traditionally, to test an application you’d deploy it into a “staging area” and then simulate many people using it with a tool such as HP’s LoadRunner or a cloud-based solution such as BrowserMob.

Load testing by it’s very nature must be an automated process. But this is a problem since Facebook applications must be accessed through the Facebook servers (for API support, FBML-to-HTML rendering, etc.)

 

Load testing the traditional way

This is a problem because it violates Facebook’s Terms of Use (section 3.2) which prohibits the use of accessing the Facebook servers with automated tools. This will force us to rethink our Facebook application load testing strategy.

In Part 2 of article, we’ll discuss why Facebook has a prohibition against automated tools, and why that is ultimately good for you, the developer.