Assignment 11: Beta-test results and Test Plan due; Launch!

Grade Value: 75 points

Due: Thursday November 20, in class

brief

This assignment has two parts: the learnings from your user testing [using your beta release], and creating a test plan (which will be executed over Thanksgiving Break).

You should feature freeze sometime this week; you should be focusing most of your energy on testing, to find out if you're going in the right direction.

assignment part I - beta-test results

Give people outside your group access to your application. (Invite codes are fine, rather than making it completely open.)

Similarly to Part II of this assignment, we suggest you log a lot of data as your users use the system. In this case, however, you'll be able to talk directly to your users (since there are so few of them in this beta). After they've used the system, talk to them. The interviews don't need to be of any specific length, and can be via email, via SurveyMonkey, in person, via IRC, or however else you can get in contact with them.

Suggestions on what to find out in your interviews: what they don't understand about the application; what they like; what they don't like; what they think your application should be used for; who they think would use your application.

In short, you're an expert on this application. You want to see it from a beginner's point of view.

assignment part II - test plan

In the studio environment, we are able to look over the shoulder of a user and observe how they use your application. With usage logs, we have the opportunity of looking over the shoulder of thousands of users and observing how they use your application. Usage logs are a very valuable and often underutilized source of information, particularly when we ask the right questions from our logs.

For this assignment, you'll create at least one A/B test within your application. You should change some aspect of the interface, randomly assign each user to either the A or B group, log which group they are in, and then log if they took the action you wanted.

For example, you could test two different pieces of copytext for a Register link: "click here to sign up" and "sign up now!". When a user came to the page, you would randomly choose one of them, show it to the user, and then log that you showed it. Then if they follow the link, you would log that they followed the link.

The tests will start executing during Thanksgiving Break, though you're welcome to start them sooner if you like. If you're not clear on exactly what A/B testing means, or how it should be done, email Greg. He can explain the concept more thoroughly, and has some example PHP code.

Notes:

• if you create more than one A/B test, make sure they don't have any interaction! If they do, you'll actually end up with 4 (or more) groups of users, and will require more users to obtain interesting conclusions.
• Groups doing iPhone applications will have to write it themselves; we don't know of any suitable libraries. If you're working on a web application, you could also write the code yourself, or you could use Google Website Optimizer, or another A/B testing suite.
• Make sure you write to your logs at the right times! In the example above, you need to log showing it to them, and then log after they click the link. If you only logged the latter, you'd only get positive results. Negative results can be very instructive.

in studio

Your slides should explain what you've learned from your beta testing, as well show what you're changing in your A/B test.

Groups will present in this order: TourSafari, Jumpin, Bumpr, RunMonster, Inherit Health