Facebook Testing: Which metric is important?

fbmetrics Many people ask us which metric is best to use when performing a load test of a Facebook application. Common wisdom says that Page Load Time (PLT) works for web applications: it’s easy to calculate, seemingly applicable, and everyone understands what it means. However, it isn’t so useful in the Facebook world since it contains so much extraneous information that makes it hard to analyze anything with it.

Facebook Canvas applications

To understand why PLT is a bad idea for Canvas applications, we must go back and understand the Canvas architecture. Users interact with Canvas applications by using a browser connected to the Facebook server which acts as a middleman and requests data from the app. The app may call APIs that interact with the Facebook server, and then ultimately return an FBML file to Facebook which renders it into HTML for return to the users bowser. The problem is that if we only look at PLT by measuring response time at the browser, we have all of this extra stuff being counted in the timing:

CanvasApps

Network latency between the browser and Facebook
Time required by Facebook to figure out which application must be called, authenticating the request, etc.
Time to render the FBML into HTML
Time spent at the Facebook server servicing API calls

These “extraneous” functions will take an unpredictable amount of time depending on, among other things, the current load at the Facebook server. This will make it hard, if not impossible, to positively correlate the timings of two different test runs. Since one of the prime motivators of load testing is to see the effects of changes to the code on performance, metrics like these will ultimatly prove to be of little use.

Facebook Connect applications

Using PLT for Facebook Connect applications is a little bit better. Users interact with Facebook Connect applications directly, and do not need the Facebook server to act as the middleman. The only extraneous functionality being counted in the timing is the time spent at the Facebook Server responding to API calls. If there are many API calls being made, this will cause a bit of randomness in the metrics.

Application Response Times

In contrast to PLT, creating metrics based on the Application Response Time is much more meaningful and useful, although harder to calculate. Application Response Time should calculate the amount of time spent at the application and only the time spent at the application. Unfortunately, in contrast to PLT there’s no simple way to get the Application Response Time – you’ll need to do a little bit of work to get at it.

To calculate this, you’ll need to create a stopwatch function which starts when your application is called, and stops when the application completes the request and returns an FBML file to the caller. In between, the stopwatch needs to be paused when a Facebook API is called. Finally, those of you using PHP as your development platform should check out the microtime() function as it’s much more accurate than the standard time() function. If your application is able to keep up with all of the bookkeeping, you’ll have metrics that can accurately show you the effects on application performance from any changes you make.

For those of you who don’t want to be bothered with keeping track of all of these timing calculations, please note that FriendRunner does all of this for you automatically.