FriendRunner – a Facebook testing tool

FriendRunner, the first testing platform specifically designed for Facebook applications has opened its beta program to public registration. FriendRunner allows developers to test their applications outside of the Facebook infrastructure, and has the ability to run large load tests in a realistic environment. Developers can use FriendRunner to help optimize their code by examining the interaction between their applications and the Facebook server, including information such as transaction timings and API call/parameter analysis. FriendRunner is cloud-based, so it does not need to be installed or require any hardware. It works for both Facebook Canvas as well as Facebook Connect applications.

Developers who wish to participate in this beta program can register for it at www.friendrunner.com.

What’s up with deprecated functions?

Facebook has recently made some significant changes to the Platform, and in doing so has deprecated some functions from the API. Deprecation is a developer’s way of getting a do-over. A way to create something better than what’s there currently. However, if developers have applications that depend on those APIs, you can’t just yank them away. You need to to give some warning first, perhaps a transitional grace period, and then you can yank them away. Since the Facebook Platform is so volatile, it’s worth thinking about how to protect your applications in such an environment, and what we should be testing for.

Functions are deprecated for several reasons

There are generally 3 types of things that can have their features deprecated.

• Libraries – such as Java and Windows
• Standards – such as HTML
• REST-oriented APIs – such as Facebook

The owners of these things will typically deprecate their features for one of three different reasons:

There’s a better way to do it

Once upon a time, if you wanted to have something  centered on your page, you’d use a <center> tag to do it. With the advent of CSS, this is now considered to be very bad form, and so the tag has been deprecated.

The underlying implementation has changed which forces a redesign

The Java java.security.Certificate interface represents an abstraction to things like SSL and PGP certificates. As the world moves on and those objects change, holes start to form in this underlying abstraction until someone decides to create a whole new abstraction and deprecate the old one.

OMG! What were we thinking??

If you’ve been around long enough, you’ll be all too aware of the abomination known as the <blink> tag. Now the real truth is, this tag was actually never deprecated, but if any one deserves to be it’s this (well, and marquee too). Deprecated things like this allows developers to sweep their embarrassing stuff under the rug.

Enter Facebook

When a Java function is deprecated, life is pretty straight-forward for developers. The code will stop compiling, and alternate ways of doing things will need to be found (typically specified by the developer of the function). Deprecating an HTML tag is much more complex because it ultimately impacts the servers, clients, as well as the large amount of content that depends on the tag. Things will get broken, and there’s not much we can do about it – that’s the nature of things.

Facebook sits in the middle of this. Since Facebook APIs are ultimately just REST calls, there’s nothing to force your compilation to fail – hey, most Facebook apps aren’t even compiled to begin with. On the other hand, you don’t need to worry about every version of every browser out there. Your application deals with one Facebook server, and when that server’s interface changes, it’s important for you to take notice.

So what happens when you call a deprecated Facebook API?

Given that there’s a good possibility of apps out there that call deprecated functions, it’s important to ask what Facebook does when that happens. The short answer is that they return an error code – API_EC_DEPRECATED (= 11 for those keeping score). It is very important to check for this code and act on it if it’s ever returned. Remember, it’s Facebook’s sandbox, you just get to play in it so you better follow their rules. Calling a deprecated REST function is simply unnecessary, wastes resources, and ultimately will piss off the people who run the servers that your app relies on. Receiving an API_EC_DEPRECATED is akin to being told to “cut it out”. Of course, applications that don’t feel a need to follow the rules are always subject to the Developer Principles and Policies as well as the Terms of Use, things that we’re told are going to be increasingly enforced.

Protect your app

To protect your app, the correct attitude to take is to never be in a position where you receive an API_EC_DEPRECATED when you call a function. Here’s our 3 step plan for how to stay on Facebook’s good side:

1. Keep yourself informed. Know which functions are being deprecated by following all of the Facebook developer resources, and acting on the information.
2. Check your error codes. If there’s one thing that developers hate to do it’s checking the error codes. Make sure to do this, and if you ever get this, make sure the application does something to notify you.
3. Know your code. You may have written this app a year ago, but if it’s still available on Facebook, you need to check it. Get a list of all of the deprecated Facebook functions here. Use grep, or something similar to look through your code to see if you’re at risk.

Facebook Application Verification Program: We Hardly Knew Ya

Hello Goodbye

Today marks the deprecation of the Application Verification Program  (AVP) as announced during the big Platform Roadmap announcement back in October. Why is Facebook doing this, and what does this mean to developers and testers. Let’s have a look:

The AVP was announced a year ago as a way for developers to differentiate their apps. Verification was supposed to show that the apps met a set of criteria to reassure users that the apps provided a good user experience. The theory was that users would flock to these verified apps, Facebook would feel comfortable that these apps provided a consistent user experience, and developers would be rewarded for playing according to the rules with things like prominent placement in the app directory and an increase on notification allocations. So why was this program axed less than a year after announcement? The truth is, it didn’t work as well as planned. When the first wave of verified apps became available in May 2009, bugs prevented some of the promised developer benefits from actually working (e.g. the special green checkmark to denote a verified app did not always show up). Developers also weren’t thrilled with the $375 price that Facebook charged to verify their app, something that really hurt small developers. And finally, did any users really care whether the apps they were using were verified or not, and moreover, did they recognize when they were?

And so today, this program is being put to rest. But that doesn’t mean Facebook is giving up on consistent user experience. They’ve simply taken the criteria from the AVP, strengthened them, and reinvented them as the Developer Principles and Policies . This is not a voluntary program as the AVP was, everyone is supposed to live by these rules. Facebook is reserving the right to censure or cut off developers who don’t. We don’t know the circumstances under which Facebook will do an audit of an application, but one thing is for sure – it won’t be so easy for them.

Understanding why auditing for consistent user experience isn’t easy gets to the central issue that Facebook is trying to solve. Consistent user experience is important. Those of us old enough to remember that awkward time between command line DOS and the advent of Windows will remember the many applications that provided quasi-GUI interfaces on top of a DOS base. None of these applications looked or felt like any other, which made the process of learning to use one long and tedious. Say what you will about Microsoft, but they did users a great service by providing a Framework for consistency with Windows (okay, borrowed from Apple, borrowed from Xerox). Contrast this with my 8-year old son’s Super Mario games which he plays expertly on 3 different platforms with different sets of controls. Not only does he not read a manual, but mocks the idea that a manual even exists. This is the epitome of consistent user experience that Facebook would like all Platform apps to have.

The trouble is, consistency for Facebook apps is elusive. After all, they’re really just web apps, and developers can do pretty much anything they like (a la the pre-Windows GUI days). To compound problems, an application audited today may change how it works tomorrow. Even if the application code doesn’t explicitly change, any changes to its software stack (e.g. the libraries it uses, the application server, the operating system) may alter the way it works.

Developer Principles and Policies: How to stay on Facebook’s good side

Facebook breaks down proper application behavior into two Principles: Be trustworthy, and Create a great user experience. These principles are expanded into ten policies that Facebook says it will enforce. Upon reading the policies, you’ll find that most of them fall into the Be trustworthy category, or as I like to think about it, the “Don’t be a jerk category”. Most of these rules are simply an extension of civilized behavior for any application: Don’t misuse people’s information. Don’t SPAM the network. Don’t engage in illegal activity. Respect copyright. … and lots more. Facebook unfortunately has to list all of these things since when it shuts someone down who engages in one, they’ll need to point to the policy that was violated. Be trustworthy is a necessary evil that Facebook must delineate, and spend lots of time, effort, and money to police. After all, hate speech is really subjective, and cannot be determined through automated processes. It requires real people to make that determination.

On the other hand, Create a great user experience is a much more positive set of policies, mainly focused on Application Integration Points and Application Responses to User Actions. Whereas Windows codified consistent behavior like where the “File” menu was (and what the typical submenus should be) these policies outline the types of things a user should expect (or better yet, not expect) when using your application. Reading the list, one can remember various apps that used some of the techniques outlined before they were outlawed, and the user backlash that they caused. Some of the policies are:

• You must not prompt users to send invitations, requests, generate notifications, or use other Facebook communication channels immediately after a user allows access or returns to your application.

Invitations or notifications should really only be used once you use and like an application. This is a good policy since it spares your friends the application SPAM that was once common. This also signals to users who receive such invitations that their friends who sent it probably really do like the application and there may be some merit in trying it out themselves.

• You must only use one Facebook communication channel in response to a user’s single action.

When a user clicks on some choice that will generate some action, that user should not be surprised when additional actions are also generated.

• You must not prompt users to bookmark your application (e.g., by using a modal window or pop-up dialog). Instead, users must explicitly invoke any bookmark option you provide.

Nothing is worse than a modal dialog that forces you to take some action that you don’t necessarily agree with, but you must do because hey, it’s modal. Sure, you can go back and delete it, but this policy puts to rest this kind of nonsense that some applications used to pull.

• You must use discretion when publishing Stream stories and must not misuse the Stream by publishing an excessive amount of stories on a user’s behalf.

Forcing your friends to confront too much information about you and some application isn’t nice. Having an application use some discretion is good.

• Stream stories must be consistent with our design and be user-focused based on the user’s action that triggered the story. In no case should a Stream story serve primarily as a means to promote or advertise your application.

Time was when any sort of notification was simply an advertisement to your friends to come and check out this cool application. Well, maybe things stayed the same, but don’t be so blatant about it.

Going through the entire list, we begin to notice a common theme. Application consistency in the Facebook world is not about where the “File” menu is, it’s about the behavior of the application when dealing with the generation of privately and publically available information. But what it’s REALLY about is trying to ensure that you don’t look stupid in front of your friends (probably one of the greatest fears out there). It’s about allaying your fears to try out an application, lest that application begins to SPAM your friends who will get annoyed at you. Once we can all try out any application without worrying that bad things will happen, we all (especially Facebook) win.

One other interesting aspect of application consistency is that many of the rules of this principle can be tested in an automated way. Does Facebook have a way to do this automatically, it would sure save them a lot of legwork. How about your development organization? Would an automated tool to test compliance with Developer Principles be of use to you? Leave a comment and let us know!

Load testing Facebook apps (Part 2)

In Part 1 of this article, we discussed that one of the reasons load testing Facebook apps was difficult was because Facebook prohibits the use of automated tools. An obvious question is “Why does Facebook do this?”.

The simple answer: It protects you, the developer

Think for a moment what would happen if Facebook did not prohibit the use of automated tools to access their servers. Well for one, we’d be able to easily carry out load tests of our applications. This is good and builds value for your application as well as indirectly building value for the entire Facebook Platform.

However, it also opens up a world of possibilities to the vast armies of hackers and spammers who’d use this to their advantage. What could they do? Let’s think:

• They could launch Denial-of-Service attacks and crash your servers. Does your application have competition? Would your competition benefit if your (obviously superior) app was seen as unstable? Let the attacks begin!
• Mischief. Facebook is now viewed as a prime target for all sorts of miscreants. Hardly a week goes by that we don’t hear about another type of attack launched at Facebook users, applications, or Facebook itself. Of course the methods by which most attacks are typically launched is by the use of automation.
• Building up your own numbers. Actually not really attack – you could use a botnet to artificially inflate your own active user numbers. MAUs are used for a variety of reasons (which all ultimately relate to money). If MAU counts also included MAARs (Monthly Active Automated Robots), then all counts become meaningless. the entire system unwinds, and everyone loses.

So let’s say that as a good developer you’d like to protect your application against a Denial-of-Service attack. You install an Intrusion Detection System, but wait - this won’t work in a Facebook environment! Remember that the Facebook Platform architecture is built so that all of the user traffic coming into your application is funneled through one place – the Facebook servers. Any attempts you make to try to distinguish good access from bad access will be foiled by the fact that the bad guys are on the other side of the Facebook servers from you. Your app has no way to distinguish good users from botnet users, it must rely on Facebook to do it for you. This simply pushes the detection of good vs. bad access out to the Facebook servers, so we can ask: “How can Facebook detect a DOS attack on my app?”

Facebook avoids Denial-of-Service attacks by using the same techniques as any other web site. This, of course, will preclude you from running an automated load test since it will simply look like a very basic (not even distributed) Denial-of-Service attack. Instead of trying to distinguish the good load tests from the bad attacks, Facebook simply says “Do not access our servers with automated tools or we’ll cut you off”. They aren’t trying to stifle our ability to get things done, but rather, they’re protecting us from a whole lot of other pain.

In the next part of this article, we’ll discuss some ideas that can be used to load test your Facebook app.

Load testing Facebook apps (Part 1)

Calvin's dad on load testing

Load testing Facebook applications is really important. Usually the whole point of deploying onto Facebook is to leverage the social graph and go viral. Developers want their applications to go viral, but there’s a nagging concern about what will happen if they do. Is the app as scalable as you think? Are the hosting choices correct? What’s the cost of deploying and crashing?

The way to alleviate these concerns is to load test your app, but how do you do that in a Facebook environment? The simple answer is that it’s very hard due to two issues central to Facebook:

Issue1: You must control enough Facebook users

Testing your application with one or two simultaneous users is easy: simply use your own Facebook account and call up some friends and have them do the same. But running a test with hundreds or even thousands of simultaneous users is much more difficult. Where do you get the users from? That answer isn’t so simple. You can say “I’ll just go into Facebook and define a few hundred ‘phony’ people that I can control”. While this may seem like a good idea, it isn’t:

• It violates the Facebook Terms of Use (section 4.1) which prohibits you from creating phony accounts. When Facebook finds your phony account, they will delete it.
• Real users have friends, become fans of pages, post updates, and hundreds of other things that Facebook allows you to do. This creates the richness of the social graph, and applications often need to take these things into account. Unless you’re willing to spend an inordinate amount of time “humanizing” your phony user army, your tests with them will not be realistic.
• While Facebook does provide the concept of a Test Account to developers to test with, these accounts are very limited in what they can do so are not of very much use for load testing.

Issue 2: Load testing must be automated

Load testing web applications is hard. Traditionally, to test an application you’d deploy it into a “staging area” and then simulate many people using it with a tool such as HP’s LoadRunner or a cloud-based solution such as BrowserMob.

Load testing by it’s very nature must be an automated process. But this is a problem since Facebook applications must be accessed through the Facebook servers (for API support, FBML-to-HTML rendering, etc.)

 

Load testing the traditional way

This is a problem because it violates Facebook’s Terms of Use (section 3.2) which prohibits the use of accessing the Facebook servers with automated tools. This will force us to rethink our Facebook application load testing strategy.

In Part 2 of article, we’ll discuss why Facebook has a prohibition against automated tools, and why that is ultimately good for you, the developer.

 

Welcome to Test Facebook

Welcome to Test Facebook, a resource for Facebook application developers who are interested in testing their apps.

You may be wondering… since Facebook apps are Web apps, why do I need a special place to discuss testing them? Can’t I apply the same tools and techniques for developing and testing web apps to my Facebook apps?

Well, as with many things in life, the answer is yes and no. Sure you can use PHP and your favorite tools to build your app. However, because of the uniqueness of the Facebook Platform, there are special considerations that you need to think about when testing a Facebook app. Perhaps you’ve already encountered some of these special issues:

• How can I perform load testing on my app without defining hundreds, if not thousands of phony Facebook users, and how do I control them all? (…and BTW, defining phony users violates the Facebook Terms of Use Section 4.1)
• If I’m doing performance testing, how do I factor out the time spent at the Facebook servers during API calls and FBML-to-HTML rendering?
• Are there special security issues for social media applications, and how do I test for these?
• How do I tell if if my application is usable?
• How do I test my application when our development schedule is already behind?
• Are there any special issues when testing applications written for Facebook Connect?
• How do I do proper load testing when Facebook doesn’t allow me to hook up automated test tools? (Again, check the Facebook Terms of Use Section 3.2)
• Now that Facebook will proactively apply their Developer Principles, how can I be sure that my app doesn’t violate them?
• How do I test in an environment where the Platform is constantly changing?
• How can I simulate some of the error conditions that Facebook returns when I call API functions so I’ll know my app is robust?

We plan to discuss these, and many other similar topics here. But mostly, we want this to be a resource for you. If you have a question or topic that you’d like discussed, or want to be a part of our ongoing knowledge base of Facebook testing, please let us know at admin@testfacebook.com