Project Overview
In this course, you will complete a quarter-long research project. This project will be completed in groups of three.
At a high level, successful projects will raise an important research question, and plan and execute a methodology for answering that question. Often, this methodology will include building and evaluating a prototype system, but hacking is not strictly necessary. All projects require a study — obviously a much more thorough study will be expected of projects that do not involve system building. The goal of the project abstract draft (described below) is to help you scope your work appropriately.
To get a sense of what a good scope for a project is, here are some examples of final papers. Because several of these projects went on to become a work-in-progress publications, their page length may differ from your final deliverable:
- Visimu: a game for music color label collection, Borui Wang, Jingshu Chen
- Brainstorm, define, prototype: timing constraints to balance appropriate and novel design, Andy Elder, Elaine Zhou
- For novices playing music together, adding structural constraints leads to better music and may improve user experience, Luke Samuel Dahl, Sebastien Charles Robaszkiewicz
- You Can't Force Calm: Designing and Evaluating Respiratory Regulating Interfaces for Calming Technology, Kanit Wongsuphasawat, Alex Gamburg, Neema Moraveji
Groups who do excellent projects will be encouraged to submit their research to the UIST poster track. These submissions are due in early summer.
For information on how the project will be evaluated, see the individual rubrics under each assignment and the grading page.
Forming Groups
This project will be completed in groups of three. E-mail cs376@cs.stanford.edu if you'd like a different-sized group. You will be subject to lengthy and brutal questioning. No teams with a single member. You will form groups yourselves. Use Piazza to help find team members. When discussing a potential partnership with someone, you should discuss your background (e.g., programming proficiency or other skills you bring), availability (e.g., do you plan to primarily work evenings or mornings? weekdays or weekends?), and motivation level (ambition for a Turing award? Or to just barely graduate?). It's important to be honest with your partners up front, and follow through on commitments you make.
You will submit your group along with the Project Ideas (Round Two) assignment.
Project Ideas (Round One)
If you want to have good ideas, you must have many ideas.
Choosing an ambitious and yet feasible project is the black art of research. There are two important keystones to making this work: 1) read a lot of papers in an area of interest, and 2) generate a lot of ideas.
We also encourage you to join existing HCI research opportunities for your class project. Doing so will connect you with a mentor who can help guide the project. Here is a list of collaboration opportunities. You are welcome to use these in your list of projects for this assignment.
The purpose of the Round One assignment is to help you generate many ideas. Along the way, you will see a lot of papers. For this assignment, generate ten proposed research projects that would fit within the scope of this ten-week course. The ideas may cover any topic and any method covered (e.g., building systems, studying people): check with the staff in office hours if you are unsure of fit. For each project idea, give it a title and describe it in no more than three sentences. In addition, for each idea, cite one published paper or article that is a central piece of published, academic related work, and describe how your project can learn from it in one sentence. Use Google Scholar as well as the references section in papers listed in this course to start poking through related work. No need to rank your ideas; all are equal for now.
Your project ideas can span any area or areas of HCI research. Look ahead to topics and readings later in the class for inspiration. Use past projects listed in the overview to help ground what may be possible in one quarter.
Grading rubric
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Research contribution 3 points |
0: No idea present. | 1: The idea is present, but it does not clearly articulate why it represents significant new knowledge to HCI or why it has wide applicability. | 2: The idea demonstrates incremental new knowledge in HCI and has somewhat general applicability. | 3: The idea introduces substantial new knowledge in HCI that is generally applicable. |
| Feasibility 1 point |
0: The idea has not been scoped to be completable in ten weeks. | 1: The idea has been scoped to be completable in ten weeks. | ||
| Related work 1 point |
0: Related work is not substantially relevant to the project. | 1: Related work is relevant to the project. | ||
Project Ideas (Round Two)
For this assignment, you form teams of three, then work with your team to iterate on and brainstorm even more ideas. Generate and submit twenty five project ideas collectively as a team. As before, ideas must contribute to human-computer interaction research and be feasible to complete during the quarter. As a team, you may decide to advance ideas from your individual Round One assignments into Round Two, but at least ten ideas must be new or significantly re-envisioned as a result of your team brainstorming. Upping the ante from Round One, each project must now have two papers of published academic related work attached and their relevance described. For each idea, indicate if there are listed project mentors or other researchers who you'd like to work with on the project. No need to rank your ideas; all are equal for now.
Grading rubric
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Research contribution 3 points |
0: No idea present. | 1: The idea is present, but it is so unclear that it is difficult to determine its contribution to HCI. | 2: The idea demonstrates a small or moderate contribution to HCI. | 3: The idea represents substanial innovation in HCI, at the level of a work-in-progress or full paper. |
| Feasibility 2 points |
0: The idea is likely infeasible to complete in ten weeks, even if it is scaled back. | 1: Some version of this idea could likely be completed in ten weeks. | ||
| Related work 2 points |
0: No related work present, or related work is so far afield as to be irrelevant. | 1: Related work is relevant to the project. | ||
Project Abstract Draft
You must submit a draft of your project abstract. The abstract should aim to be around one full typewritten page (~500 words) with citations in APA format. Course staff will provide feedback on the draft to assist in the preparation of a final version.
Option 1: If you are designing or developing a new technology, your abstract should cover the following topics:
- Problem: What is the problem that you are solving, and why is it important? For strong examples, read the first couple of paragraphs in the Introduction section of systems/applications papers in this course or on the HCI Group web site.
- Insight and theoretical contribution: What is the unique perspective that you bring to the research? In other words, what is your "aha!" insight that others haven't established? How does it solve the problem? Keep in mind that even impressive technology prototypes are, at their core, answering a research question or establishing a thesis.
- System: What is the exact system that you are building to instantiate this insight? What will it do? Include a low- or medium-fidelity sketch of the envisioned system.
- Related work: Write a paragraph describing 4-5 pieces of published research, how they inform your project, and where your work trancends this knowledge.
- Evaluation plan: how will you know if your system solved the original problem? (This should include the design of your study.) Grounding this in methodologies that other researchers have used (e.g. by drawing from the class readings) is a good idea. There are three major points you should hit here.
- Study design: What are you going to do? Be detailed and precise. This can involve a large range of example outputs from your system, a field deployment, or a controlled study.
- Measures: How will you know you succeeded? What will you measure? How will you measure it?
- Ecological Validity: Why does your study answer your research question? Why does your evaluation address your hypothesis? Make sure your study, and the variables you're measuring, properly address the question you are asking.
- Recruitment Plan: most evaluations involve participants. How will you get them? For pilot studies, we suggest you recruit from within the class -- "trading" participation with other groups is a great way to learn about what others are doing. For larger studies (e.g. for those not building a system), you need a clear recruitment plan.
Option 2: If you are developing a study, a new method, or data analysis, your abstract should cover the following topics. Keep in mind that the staff's grading bar for study design and execution here is much higher than if you are building a system:
- Research Question: What are you trying to answer? What are the theoretical contributions of your work to existing research in your area?
- Hypothesis: what do you think the answer to your question is, and why? State your hypothesis in terms that you will actually be able to deliver on within the space of a quarter. For example, having a new technology increase someone's income might be your ultimate goal, but you may not be able to measure a change in income in 10 weeks. In this case, increasing income could be part of your motivation, but not your hypothesis. Your hypothesis needs a more proximal measure.
- Related Work: Write a paragraph describing 4-5 pieces of published research, how they inform your project, and where your work trancends this knowledge.
- Method: how will you explore your hypothesis, and why is that the right approach? (This should include the design of your study.) Grounding this in methodologies that other researchers have used (e.g. by drawing from the class readings) is a good idea. There are three major points you should hit here.
- Study design: What are you going to do? Be very detailed and precise.
- Evaluation: How will you know you succeeded? What will you measure? How will you measure it?
- Ecological Validity: Why does your study answer your research question? Why does your evaluation address your hypothesis? Make sure your study, and the variables you're measuring, properly address the question you are asking.
- Study Recruitment Plan: how will you get participants for your study? For pilot studies, we suggest you recruit from within the class -- "trading" participation with other groups is a great way to learn about what others are doing. For larger studies (e.g. for those not building a system), you need a clear recruitment plan.
For a guide to the APA format, go to APA Style. Note that the information on the site is possibly too detailed for the abstract. If you want a good example of the detail expected for the final paper, look at Dynamic Speedometer: Dashboard Redesign to Discourage Drivers from Speeding, Manu Kumar and Taemie Kim.
We encourage you to iterate multiple times on this abstract. While there is only one formally defined point for receiving feedback from course staff, you should seek out more informal feedback as you work on this. E-mail us at any point if you'd like us to take a look at your current submission, or come to office hours if you'd like to discuss in person. You are free to change directions after submitting your draft, but the sooner you nail down a direction, the better your project is likely to be.
Tips on running effective studies with people:
- If you have instructions, present them to participants in written form. You'll have a lot on your mind. Likely too much to remember to say everything you want. Written instructions also help insure that everything is consistent across participants.
- Remember to instrument your software so that it logs all relevant user actions. How often did people ___?
- Whatever you try first is almost guaranteed to have bugs so iteratively prototype your experiments just like you iteratively prototype interfaces. Test early and often and leave time for iteration in your schedule.
- If you're asking someone to do a task, it helps to provide a good motivating scenario. Appendix A of this paper provides a pretty good example of a scenario for a design tool.
Systems abstract grading rubric
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Problem 4 points |
1: The problem is unclear, unimportant or not well-motivated. | 2: The problem is clear but only motivated at a mediocre level. The solution feels like a hammer trying to find a nail. | 3: The problem is clearly articulated and well-motivated. | 4: The problem poses a novel perspective or a major opportunity for innovation. |
| Insight 4 points |
1: The insight is not well described or has been covered previously in prior work. | 2: The insight is clear, but makes only an incremental contribution to the field. | 3: The insight delivers a reasonably novel idea relative to the literature. | 4: The insight makes a significant contribution to the literature. |
| System 2 points |
1: The system description is unclear or incomplete. | 2: The main idea of the system is made clear through writeup and sketch. | ||
| Related work 3 points |
1: Major related work is missing. | 2: Much of the relevant related work is present, but either some is missing or its relationship to this project is weak. | 3: Related work is present and well-represented. Its relationship to the project is clear. | |
| Evaluation plan 4 points |
1: The method is unlikely to lead to an evaluation of the inight or system. | 2: The evaluation plan might work, but is vague or has methodological flaws, or there are obvious better methods. | 3: The evaluation is likely to test the proposed insight or system, but has some minor methodological flaws or is overly complicated. | 4: The evaluation is methodologically valid, not overly complicated and well thought through. |
| Feasibility 2 points |
1: Project may need to be substantially scaled back to finish in ten weeks. | 2: Project can likely be completed in ten weeks if the team works hard. | ||
Study abstract grading rubric
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Research Question 4 points |
1: The research question is absent or trivial (the answer is obvious). | 2: There is a promising question but it is not clearly stated. | 3: The question is clearly stated but has only minor impact on the field. | 4: The question is clearly stated and its answer has major impact on the field. |
| Hypothesis 4 points |
1: The hypothesis is the same as research question or trivial. | 2: There is a hypothesis, but it is unclear, too ambitious, or not clearly motivated. (What makes you hypothesize so?) | 3: The hypothesis is clearly stated, but is only vaguely motivated. (What makes you hypothesize so? Can you ground it in prior work?) | 4: The hypothesis is clearly stated and well motivated. |
| Related work 3 points |
1: Major related work is missing. | 2: Much of the relevant related work is present, but either some is missing or its relationship to this project is weak. | 3: Related work is present and well-represented. Its relationship to the project is clear. | |
| Study design 4 points |
1: The method is unlikely to lead to an evaluation of the hypothesis. | 2: The evaluation plan might work, but is vague or has methodological flaws, or there are obvious better methods. | 3: The evaluation is likely to test the proposed hypothesis, but has some minor methodological flaws or is overly complicated. | 4: The evaluation is methodologically valid, not overly complicated and well thought through. |
| Measures 2 points |
1: The measures being used are not reflective of the concepts being studied, are difficult to measure, or are likely to exhibit bias. | 2: The measures being used have ecological validity and construct validity, and can be measured. | ||
| Feasibility 2 points |
1: Project may need to be substantially scaled back to finish in ten weeks. | 2: Project can likely be completed in ten weeks if the team works hard. | ||
Project Abstract Revision
This is the final revision of your project abstract. It is weighted much more heavily than the draft.The formal requirement is that you revise your project abstract draft to address the staff's comments on your draft.
The rubric is the same as with the abstract draft.
Progress Meeting
Course staff will meet individually with each project group to provide feedback on your progress. These meetings will happen in the last 30 minutes of our scheduled class time. Please use the link to the Google Doc below to sign up for a time slot. You should be prepared to present your working system, discuss your study plan, and have pilot results. Use the online submission system to submit any materials you'd like to discuss (e.g., prototypes, data, draft writing.) This is not graded: it is your opportunity to get our focused feedback on your project. Come to the meeting prepared to show and tell with preliminary results and how you plan to course-correct based on your early experimentation. How will you revise your system/design/experiment/framing so that your project really pops? What will the title of your final paper be? In other words, how will you summarize your research contribution in just a few words? This exercise will helps you focus and sharpen your efforts on what will best address your research question. This focus will be especially important as time gets tight: some things will matter more than others.
Sign up for a slot!Project Faire (Round One)
This is your first checkpoint for the project. The goal is to focus on prototyping or piloting the core element of your research idea. Reflect on your abstract and its core insight or question. What is the pilot study or prototype that would most clearly examine it? Do not spend much time working on aspects of the project that are orthogonal to your research question. For example, if you are testing a new debugging interface, focus on implementing one or two scenarios with special cased data, not the entire underlying infrastructure or the overall IDE plug-in architecture. As another example, if you are studying whether people think that online comments that contain self disclosure (e.g., "I'm scared of Michael Bernstein's eyebrows") are more likely to occur on ephemeral or permanently archived social media, focus on gathering the status updates for the study and analyzing the results from a pilot run: don't worry about the details of your labeling interface.
If you are creating a system, the deliverable is a basic working prototype of your research idea. It does not need to be polished, but you should have the basic idea at a functional prototype level. If you are running a study, the deliverable is pilot results from at least five people who ran through an early version of your study, including a first round of data analysis.
Submit any relevant materials, including a one-page document describing your project status and goals between now and the Round Two project fair, via the class web site. We'll also be using the last 30 minutes of class to show off our projects to each other, so bring any demo or study materials you've got.
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Depth 8 points |
2: The system or study is too thin to evaluate the research question. Critical components are missing. | 4: The system or study has a basic pilot/prototype, but it is only barely at a depth to point to next steps for the research. | 6: The system or study has a pilot/prototype that exercises the main research idea, provides evidence of its final feasability, and points to next steps for the research. | 8: The system or study has struck at the core of the idea and demonstrates clear directions for the research. |
| Focus 3 points |
1: Too diffuse: attempting to prototype/pilot too much of the idea at once meant that the core question was largely left unexplored. | 2: The system or study explores the main question or insight, but also spent significant time on less critical aspects of the research. | 3: The system or study focuses directly on the most important parts of the research question. | |
| Plan 2 points |
1: Plan is underspecified, not ambitious enough, or too ambitious. | 2: Plan clearly details the goals for Round Two and a realistic timeline to achieve them. | ||
Project Faire (Round Two)
We are nearing the end of the quarter, so it's time to start showing off your best stuff. This is your final checkpoint before the final project presentations. The goal is to have the core element of your research project complete. If it's a system, that means that the core research functionality should be implemented, and (perhaps with an additional coat of paint to come soon) ready for an evaluation. If it's a study, you should have your final materials complete, the study should have at least ten participants so far, and you should have a first statistical/qualitative analysis of your data complete. If it's a study, it is OK for this version to have null results as long as you also have an idea of how to proceed.
Submit any relevant materials, including a one-page document describing your project status and goals between now and the final project presentations, via the class web site. We'll also be using the last 30 minutes of class to show off our projects to each other, so bring any demo, study materials, or data that you've got.
The rubric is the same as for Round One.
Final Presentation
At the end of the quarter, you will present your research results to the class and outside guests. We have invited a couple HCI luminaries. Feel free to invite interested friends and colleagues!
- Each group will have 5 minutes to present, 2 minutes for questions, and 1 minute for setup. This time limit will be strictly enforced – groups should set up during the question session of the group before them. To enable this, unplug the video cable from your laptop before answering questions.
- Test (and debug) your laptop video projection before presentations begin. Time spent fiddling with display settings will count against your presentation time.
- Structure your presentation like a pyramid — begin with a one-sentence statement of your research result. This will get everyone on the same page. Then, offer a short (e.g., 1 slide, 4 sentences) description of what you did and why you did it. Then, explain things in detail.
- This presentation is short enough that you can write out everything you want to say long-hand. Do this! This will allow you to convey information efficiently and effectively. Read through it enough times so that you have it basically memorized, but not so memorized that you get flustered if you skip a word or someone asks a question.
- Know your audience! You can expect that everyone in the class knows everything you learned in class. So, you don't need to re-introduce the whole field of HCI. A sentence or two to situate your work in the field is good, but spend the rest of the time telling us what you did.
- When presenting, stand near your slides. And look at the audience.
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Clarity 4 points |
1: The research question, method and results are unclear. | 2: The basic thesis of the work was communicated, but major components were buried or confusing. | 3: The main ideas were clear, but the details were confusing, incomplete, or signaled a misunderstanding. | 4: The presentation clearly sets out the project's goals, results, and implications, and all details were clear. |
| Effectivenss 4 points |
1: The presentation was unconvincing in its communication of the core problem, question, and solution. | 2: The presentation was unconvincing in its communication of at least one of the core problem, question, or solution. | 3: The presentation made a reasonable case for the importance, novelty and validity of the research. There was at least one noticeable weak point in the argument. | 4: The presentation made a strong case for all aspects of the project, and did so with forcefulness or flair. |
| Timing 2 points |
1: The presentation ran out of time or misappropriated time between components of the research. | 2: Balanced coverage of introduction, idea, system or study design, and results. | ||
Final Paper
In addition to the presentation, you will present your findings in a final paper. This paper should be written in the same style as the paper we have been reading all quarter. However, it will be shorter, so pay attention to space limits.
Page limit: Final papers should be 3-5 pages long in the SIGCHI papers template. While this may sound short, it is much harder to write an effective, complete short paper than it is to ramble. A good approach to writing a great short paper is to write a long one first, and then trim it down to the most vital parts. Appendices are acceptable and optional (they don't count towards the page limit), but won't be graded. Add one for materials you want an interested reader to see (for example, when we post your project on the website for next year), but don't need to be graded. The page limit includes references.
Much of the advice from above for preparing your presentation applies to the paper as well. Here are a few more suggestions for preparing your paper:
- Find a paper that you particularly like because of how it's written, and use it as a template. This paper needn't be on the same topic, but a close mapping in terms of type of contribution (e.g. a tool paper vs. a theory paper) will give you more guidance as to how to structure your paper.
- The title and abstract are the most important parts of a paper, and should clearly convey what you did. Motivate your specific problem (not the field as a whole), and focus on what you did. After reading the abstract, the reader should know what your contribution is – don't speak in generalities. For example, instead of saying "We analyze different methods for preparing cookies with interesting ingredients by running a user study.", say "We present three new recipes for chocolate chip cookies each employing a unique ingredient: jellybeans, tofu, and corn nibblets. Cookies were compared using a blind, within-subjects taste test with 30 individuals. The cookie with tofu was found to have superior mouth feel when compared with the other two, but subjects preferred the taste of the corn cookie by a 2:1 margin."
- Review the Project Abstract assignment. Make sure you clearly address each of the important bullets from the abstract in your final paper.
- Please use the APA heading structure to describe your study & results. Clearly tie your analysis to your hypotheses.
- Use pictures to show your interface and graphs to present your data. Graphs should generally aggregate across participants, and show standard error bars. (Only show individual data points if the reader learns something more by doing so.) Figures should be captioned with what you believe the reader should infer from the figure (e.g. Participants rated tofu cookies to have 25% better mouth feel. Differences between jellybeans and corn nibblets were not significant).
| Category | Insufficiency | Adequacy | Proficiency | Mastery |
|---|---|---|---|---|
| Research idea 4 points |
1: Unclear or indistinguishable from standard application development. | 2: Vague, only of tangential relevance to the HCI research community, or incremental in nature. | 3: A clear research question or goal, but not cleanly articulated as a novel contribution beyond prior work or is not a major contribution. | 4: A clear research question or goal that makes a significant addition to HCI research. |
| Execution 4 points |
1: The project has some complete components, but critical aspects are incomplete. | 2: The basic elements of the research are complete, but either they are flawed or important aspects are still missing. | 3: All relevant aspects of the project have been completed, but they have flaws. | 4: The research question has been explored and answered thoroughly through the project and writeup. |
| Evidence 3 points |
1: The evaluation or study is incomplete or has major analysis flaws. | 2: The evaluation or the study is complete, but has significant analysis flaws. More leniency is applied to systems projects on this rubric item. | 3: The evaluation or the study is complete and convincingly argues for a result. | |
| Related work 2 points |
1: The description of related work is incomplete or surface-level. | 2: The writeup covers major points of related work, and explains how this research extends them. | ||