First, create a user study plan and pilot it with at least one person. Then, test your functional prototype with at least three different people, generate user feedback and insights, and use this to inform revisions to your prototype your final project.
First, develop a user test plan. Base this plan on what you would like to learn. Then, prepare an outline or agenda of how the user study will be run, written introductions that you will read to participants, and any other materials that will be used during the study (e.g. post-test questionnaires and interview questions). This will be similar to what you did for the paper prototype, but with greater depth.
Your high-level goal is to find problems with your interface. Look for breakdowns and pain points in your interface (you might want to revisit Nielson's 10 Usability Heuristics), and try to understand what the problems are and how you might fix them. If possible, try modifying or updating your prototype before continuing to the next user.
Consider the different types of user tests that one can run, and choose the type of user test that you believe makes the most sense for your project. Make sure you can justify this decision! A great way to do this is to begin by writing down exactly what you hope to learn from the study and then use this list to inform your study planning. You may want to revisit the Martin reading for guidance as you develop your test plan.
As mentioned in the previous assignment, the more realistic your prototype is, the more likely you are to uncover unanticipated design errors as the application can be used more naturally.
When actually running the user study, each member of your team should 'own' a different aspect of the test (and, if so desired, switch roles). One person may administer the task with the user, answering any questions, and generally facilitating the task. Another person should be recording notes/video/photos/etc. A third person could be absent so not to overwhelm the user, could be watching for technology problems, or could also be recording notes. As much as possible, keep things consistent between your user tests: use the same script, follow the same protocol, answer questions in the same way. Immediately after each study, record your reactions down on paper (physical or digital). There is a good chance you will forget things so write them down asap!
After your studies are complete, take some time to reflect on the whole process (writing those thoughts down) and then go through your entire experience from the top. Often when we analyze results of a user study, we forget or skew what we've seen. Look at the results as objectively as possible. You can do this by codifying actions and results and looking for trends. Sit down with your group in front of a large whiteboard and talk about overall trends. Where do breakdowns occur? Was there a general pattern to user behavior? When you identify some interesting points, talk deeply about them - ask each other questions, recreate the different user tests, analyzing user decisions, other paths they could have taken, and so on. This is your chance to explain user behavior with theories of your own. As often as possible, use quotes for study participants - quotes are a great way to ground your discussion and they add a lot of texture to your results. Improve your prototype by improving the design in light of these results. Remember that you don't need to test EVERY ASPECT of your prototype - but you want to show concrete testing and redesign on 1 or more features/aspects of your prototype.
You will use the results of this assignment to inform your final iterations and updates for your group project. For more information on the scope of the final project, see the final project page.
| Grading Dimension | Guiding questions | Bare minimum | Satisfactory effort & performance | Above & Beyond |
| Experiment Design (max 40 points) |
How well-planned was the user test? Did you justify your approach to the user study and what components you chose or chose not to include? Is the test unbiased and designed to draw out interesting information? | 0-34 points; Study plan exists but is not particularly innovative | 35-37 points; Study plan is clearly motivated and well thought-through. Study was designed to produce useful data. | 38-40 points; Study plan is very clearly motivated or innovative in a way that will ensure rich and interesting data. |
| Clarity (30 points) |
How well does your write-up describe your findings? Is the evaluation plan included? Any surveys or questionnaires (text/image)? Did you describe revisions and changes to prototype and why, based on what you learned from your study? Is there a paragraph on design reflections on this user test process, potentially in contrast with paper prototyping? Did you include photographs of your user test in action? | 0-24; Write-up is sparse, missing key information, leading to group continue developing without taking the user feedback into account. | 25-28; Complete, detailed, and useful writeup that can will be useful for the next iteration in development. | 29-30; Clear, concise, and insightful write-up presented in an organized fashion. All parts that were asked for are addressed. |
| Execution (max 30) |
How well was your study executed? Was it successful? Did you find any substantial issues with your interface design? | 0-24; User study was done "for show" and does not feel authentic. No or very little useful information found. | 25-28; User study was well executed and let to some good learnings. | 29-30; User study helped to uncover new insights not possible before. |
| Revision rationalization (max 35) |
Did you come up with multiple changes to make to your application based on user feedback? Is it clear what that user feedback is? Are there justified reasons for making or not making particular changes? | 0-28; Test was not very useful and did not lead to many revisions | 29-33; Several possible revisions presented, for different portions of the user interface. | 34-35; Revisions are creative and addresses the actual problem generating the user feedback, with the user in mind. |
| Insights (max 5) |
Did you find anything interesting while reflecting on the process? | 0-3; Purely obvious insights discussed. | 4; Non-obvious insights presented. | 5; Insights are perceptive and novel. |