Monitoring your Facebook application for regression – Part 3

In our last post, we discussed why it’s so important to monitor your Facebook application for regression since Facebook is continuously updating the Platform. Facebook provides a valuable service by running the Beta Tier for you to test against to catch potential problems before they get out to your users.

Here’s how you can test against the Beta Tier:

Run a second copy of your application

You’ll need to run a second copy of your application as a “regression test” in parallel with your “live” deployment copy. This is because you’ll need to make some fundamental changes to it that cannot be made to your deployment copy. In particular, any API calls that are made to *.facebook.com should be made to *.beta.facebook.com instead. This will allow you to use Facebook’s latest beta code, which is what you really want to test regression against. To do this, you’ll need to edit the libraries that you’re using to interface with Facebook and make this modification. Of course, this assumes that you have access to the source code of the library which interfaces with Facebook. If you don’t, then you’ll need to come up with something clever.

Running a second copy of the app will obviously mean that it must be hosted in a second location which means that it also must be registered with Facebook. Registering a second copy means that you get new Application IDs and secrets, which of course means modification to your sorce code.

At this point you’re going to need to be a bit careful because every time you make an update to your code, you should also update the regression test copy.  However, you’ll need to be careful that you don’t overwrite your changes that you’ve made to the ID/Secret or to the library. Perhaps you should consider stripping that code out into its own file so that it’s clear what should be overwritten and what shouldn’t. It’s also probably a good idea to copy things over automatically as part of your build/deploy process.

Choose an automation solution

Clearly you want the process of monitoring to be automated, so you’ll need to choose a solution to automate the monitoring. This can be a commercial solution or something you build yourself. The benefit of using a commercial solution will be that it can be done very economically and will be easy to use. Alternatively, if you want much more control, you can build your own solution around a system such as Selenium.

At this point you’ll also need to create a test user.  This is easy enough to do, but remember that you’ll need to add the user as a developer if the application is not public.

Create a test script

Creating a test script should be easy to do. How you do it, of course, will depend on the automation solution you’ve chosen. The difficult and more important thing to do here is to determine the list of actions that the script should contain. You should try to get coverage of all of the possible functions. If this isn’t possible, you should at least get coverage of all of the API calls that the applications makes, as well as all of the FBML tags used.

Execute, monitor and act

Now that you’ve got everything set up, sit back and let the monitoring begin. You’ll want to run your script whenever the Beta Tier is updated, so make sure you’re aware of when it’s going to happen.

Also remember that any changes you make to your app must be deployed to the beta test version also. Most importantly, when the system identifies some regression, stop whatever you’re doing and investigate.  Understand the reason for the regression and determine whether it’s real regression or simply a change in design and act accordingly.

Finally, watch the Facebook forums to see what’s happening and if other developers are experiencing any regression. If you find that your app has regressed, be a good community member and make sure you let others know so they can check their apps too.

Monitoring your Facebook application for regression – Part 2

In our previous post, we talked about the Facebook Beta Tier, and why it was important to test your application against it, lest your application break due to Facebook’s changes. In this post we’ll look at things a little bit deeper and see what Facebook can do that could break your application.

Facebook changes that can break your application

Deprecated Functionality

Facebook does occasionally deprecate features in the system such as

• Methods (profile.getFBML)
• FBML tags (fb:add-section-button)
• FQL tables (The Metrics FQL table)
• Entire Functions (Datastore API)

Why does Facebook deprecate functionality? Well, we’ve already discussed that topic once, but the important thing to take away is that if they didn’t, the Platform would be a complete and utter mess.

It’s important to note that Facebook does give developers loads of warning when something is getting deprecated so that they’ll be able to fix it. It’s also true that when Facebook announces that they’re going to deprecate something, a grassroots outcry from developers can be successful in changing Facebook’s minds (e.g. admin.banUsers).

When Facebook does get around to formally removing a feature from the Platform it’s important that your application no longer uses it because it will break.

Modified Functionality

Something that typically flies under the radar is the fact that Facebook is constantly modifying Platform functionality in small ways every week. These modifications tend to be small: a changed default here, an added parameter there. Not huge changes, but changes nonetheless. Facebook even publishes this information in their Change Log.

So can these types of changes be breaking changes? Sure they can, depending on what your application does.

Bugs

Yes, it’s uncomfortable to think about it, but occasionally the Facebook developers introduce bugs into the Facebook Platform. This happens for many reasons, the most easily understood though is that the developers are people, and people make mistakes. Software development happens at Facebook at near breakneck speed, and occasionally bugs do enter into the code. Facebook also famously has no formal Quality Assurance team, relying on a system of peer code reviews instead. Depending on your development experience, you’ll probably have strong positive or negative feelings about doing things this way, but hey, it works for them.

Can something really break?

Uhhh… Yes. If you spend a little bit of time on the Facebook Forums or Googling for more information, you’ll easily find people talking about changes that Facebook made that broke their applications.

For examples, here are two that we easily found:

Facebook modifies a workflow that’s been in place for a while

Facebook introduces a bug in OAuth processing

The bottom line to all of this is that Facebook’s changes can cause regression and break your application even when you do nothing. In our next post we’ll look at strategies for using the Beta Tier to try to contain this problem.

Monitoring your Facebook application for regression – Part 1

A few days ago there was a post in the Facebook Developers Blog entitled Testing Using The Beta Tier, which is important to read and to understand its implications. The gist of the post is that the developers at Facebook are pretty busy and are constantly modifying the Facebook Platform. They push out updates on a weekly basis and all of us in the Facebook Platform ecosystem are affected by this whether we like it or not (and whether we know it or not). Fortunately, as this post explains, you can get access to the beta version of these updates before they go live to ensure that your application plays well with it. This is a valuable service that Facebook is providing, and it’s important to understand why.

Let’s contrast a Facebook app with a plain old Web Application. Imagine you’ve just finished designing and building your great new Web App. Before deployment you test every possible thing that could go wrong and you feel really confident about its stability and robustness. You’re so confident that it won’t break that you decide to relax and spend the next two weeks in Bora Bora without even a laptop. Things may go wrong, but if your app is really well-written, then all of the problems will be operational problems. Someone tripped over a power cord and knocked out a router, or spilled coffee into a server.  Things that are certainly problems, but they don’t need you the programmer to fix them. Go get yourself another drink and don’t worry about your app.

But Facebook development isn’t like that. You still need a good operations team to make sure your app stays online, but the stability and reliability of your Facebook app also depends on the invisible hand of Facebook, something that you have no control over. Before booking your flight to Bora Bora you’ll need to come to grips with the possibility that Facebook could very easily push out an update that changes an API in a way that breaks your application. If/when that happens to you, then you the programmer will need to do something about it to get your app back up and running again, even if it means getting up from the beach. And that’s the important point here: Facebook applications can and will experience regression even when they aren’t being changed because Facebook is always changing.

The first step to dealing with a problem like this is knowing that there is a problem. If you have a general Web App, knowing that it isn’t working isn’t so hard to do. You can simply set up a monitoring system to periodically hit the app and make sure that it responds properly. There are many inexpensive commercial services that make this really easy to do.

But what about Facebook applications? You can monitor the application to ensure it is up and running, but that won’t tell you if Facebook has done something to break it. That’s why Facebook’s Beta Tier is so valuable – you can find and fix potential problems before they get seen by users.

But how can we do this automatically? It would be great if we could set up an automated way to know that the next Facebook update will break your application. In our next post we’ll look at some strategies for how we can do this. In the meantime, leave a comment if you think this would be a good idea, and any thoughts you have about it.