Peer Review Draft Due: Wednesday, November 20 at 4:00pm. Submit on Canvas.
Optional TA Feedback Revision Due: Wednesday, November 27 at 11:59am. Submit on Canvas.
It's time to start putting everything together into the paper. This is an exciting moment — this is when all the work you've been doing starts to take shape into a paper that you can submit for publication. This won't be a complete draft, but it will put together the bones of the paper and incorporate text for a substantial proportion of the paper. You can continue to iterate on the final results. However, a common failure point for researchers is not to save enough time to write a convincing paper before the deadline, which is why we're starting now.
In section, your draft paper will be passed to other students for a mock peer review, and you'll do a mock peer review on their paper.
Following the pathway laid out in lecture, first identify a strong model paper to use as you are writing. Keep in mind that a model paper should be making the same genre of argument as your paper; it doesn't necessarily need to be tightly related to the actual topic of the research contribution. There likely exists a good model paper somewhere in your related work.
You will be using this model paper as a template as you outline your own paper, so choose carefully. We strongly recommend checking with your TA to make sure that it's a good fit.
Just as you did early in this class, you'll work backwards from the model paper to an outline. You'll be using this outline as a template in order to develop yours. Your outline should contain the following components:
An example for the Mechanical Novel paper on crowdsourcing short story writing:
- Introduction [1pg]:
- Argues for the bit flip: prior work does a one-way decomposition of work into constituent tasks, but this doesn't work for interrelated goals such as story writing, so they introduce a reflection-revision crowdsourcing technique that more closely mimics how writers actually write
- Figure 1. Provides a high-level overview of the main workflow being proposed by the paper
- Related Work [1pg]: covers other papers on crowdsourcing complex work, and on how authors write stories
- Collaborating through context-free tasks: describes prior research on splitting up crowd work into small tasks
- Crowdsourcing with global goals in mind: describes prior research on weaving together those split-up tasks with a global goal
- Mechanical Novel [1.5pg]: describes the high-level/low-level iterative loop that makes the system work
- Designing workflows based on expert practice: summarizes the expert practice that they are drawing inspiration from in creating their system
- Initialization — creating a first draft: describes the general set up of the system
- Reflect — choosing a high-level goal: describes the first major part of the system, how the contributors identify a high level change they want to make
- Figure 2. Shows a screenshot of the reflection interface
- Revise — translate goals into actionable tasks: describes the second major part of the system, how the contributors split that high level change into concrete low level contributions to execute
- Figure 3. Shows a screenshot of the decomposition interface
- Figure 4. Shows a screenshot of the voting interface
- Evaluation [4.5pg]: tests whether the approach succeeds at writing
- Figure 5. Shows an example story as it proceeds through revision cycles
- Figure 6. Visualizes the control system vs. the Mechanical Novel system
- Benchmark study
- Tested how well the process fixed stories with known problems in them. Subsubsections capture specific avenues of investigation, for example the kinds of changes that each system made.
- Table 1. Summarizes the inputs to the evaluation
- Table 2. Results: how often did the system identify the right part to fix?
- Table 3. Results: how often did the system accurately make the right fix?
- Story writing study
- Tested how well the process could write a story from scratch, including quoted examples
- Table 4. Summarizes the inputs to the second evaluation
- Table 5. Raw results of the readers' evaluation of the stories written by the crowd
- Table 6. Report of the kinds of changes that contributors made as they revised the stories
- Discussion [1pg]: articulates why the approach worked in practice, and how to expand it
- Enabling flexibility and encouraging diversity: unpacked the reasons why Mechanical Novel did better than the control — what about the idea was right, and what mattered less in practice?
- Going beyond short stories: a future work discussion of what would need to be changed about the system in order to write longer stories
- Designing collaboration around reflection and revision: what are the possible generalizations of the reflection-revision technique introduced in this paper?
- Conclusion [.25pg]: summarizes the main contributions again, as well as the results of the study. Closes with a "zoom out" to the broader vision.
As described in class, use this outline as a template and translate it to support the argument in your paper. Translate each section and sub-section heading into what the equivalent heading is for you; translate each sub-thesis into what the equivalent sub-thesis is for you; translate each figure into what the equivalent figure is for you.
You are welcome to drop some elements of the outline, add some elements that weren't present in their outline but you feel should be present in yours, or tweak lengths in the outline. The model paper is a template, not a straightjacket. However, if you're straying too far from the model paper outline (more than 20% or so), it suggests that perhaps the model paper is not a good fit for your research, and you need to find a different one.
Before writing in earnest, we strongly suggest getting feedback from your TA on this outline.
Now, it's time to flesh out your outline into a full draft paper. You are welcome to use content that you generated in previous assignments, e.g., the introduction, related work sections, and evaluation method. You may need to tweak your previous writing as your project has evolved.
At this stage, your draft paper can leave the Discussion and Conclusion sections in outline form, and Results can be incomplete or just pilot data. Most likely, you'll be focusing most of your effort on writing out the details of your approach, and reporting any results you have so far. Include a References section with citations at the bottom; references are not included in the word count.
Budget plenty of time for getting your paper to "work" in the right academic template. Here's the template for each section:
Many folks find Overleaf a useful collaboration platform for writing papers in research paper templates. You'll need to learn LaTeX if you don't know it already, but it supports simultaneous editing.
Your draft paper will be traded with another group for peer review in section after the deadline. We, the staff, will also send feedback on the draft. However, we recognize that many teams will want to add a bit more data or results into their drafts before we look, or may want to revise their paper in reaction to the peer review feedback. So, while the assignment deadline for peer review is 11/20 at 4pm, we will leave Canvas open for submission and will give feedback on whatever version is there as of 11/27 at noon. The further you are, the more helpful we can be. Revising the paper between the deadline and the 27th is optional; we are doing this because several teams wanted a bit more time before we looked.
We will be trading papers for peer review in section after your turn in your draft.
Executing this project will be your team's focus for the rest of the course. Here is a reminder of the timeline:
Submit a PDF with (1) your draft paper, (2) your model paper title and outline, and (3) your outline on Canvas by Wednesday, November 20th for peer review. You can then take peer review feedback and continue to revise and resubmit your draft until noon on November 27. TAs will grade and give feedback on whichever draft is in Canvas on that date.
This is a group assignment; create a group for your team, and one member should submit on behalf of the group.
The November 20 submission will be used in section for peer review and as the basis for your Completeness grade. You can then take peer review feedback and continue to revise this draft until noon on November 27. Your TA will give feedback and grade the remaining parts of the rubric using whichever version is uploaded to Canvas at that time.
Your draft paper will be graded on the following rubric: