Scott taught an HCI course online through Coursera, which, of course, is interesting because a lot of the types of problems that the students dealt with in the course were not ones that could be easily graded in a multiple choice or other standardized manner. Scott is going to share his learning from that.
Scott Palmer:  Thank you all for coming back after dinner. As many of you know, I have a background in design and I'm a huge fan of the teaching methods that design has in discipline and poise. I'm going to give you a really personal narrative of my experience with a number of different assessment methods in design classes. The personal favorite seems really appropriate for an after-dinner talk.
Scott:  I think one of the core methods of design teaching and learning is the use of the studio method. The studio in design teaching comes from the founding of the Ecole des Beaux‑Arts in Paris in 1819. It's endured for about 200 years. You fast forward about a century later to the Bauhaus in Germany in the early 20th century. Most classic art schools that you see in America and Western Europe follow the pattern that was developed at the Bauhaus. They had three different physical locations. This is the Berlin one a couple years ago.
I think art and design education can teach us a lot about how to structure physical space. This is a photograph I took of the product design loft at Stanford. One of the things that I like about this space is you've got Barbie dolls and upside down umbrellas and just stuff all over the place.
If you want to know what your peers are up to, if this is your studio space, two seconds and you know what's going on. When somebody changes direction or learns something new, you know what's going on. If you want to know, "Who should I ask about bending fiberglass or glue guns or anything else," you can see that.
The visibility that you get in a physical studio is incredibly powerful.
Interestingly, I think computer science used to be this way, also. I was an undergraduate at Brown. I couldn't find a picture of when I was there, but I'm in between these two eras. Everybody was in one room. If you wanted to know what are you going to learn next year...
Yeah, Zoron is in this picture on the left, actually.
[laughter]
Zoron:  Yeah, I remember that. [laughter]
Scott:  Erin might be on the right. [laughs] What's really nice is, you want to know what you're going to learn next year? It's there. You've got a stack fault in your stock, you can turn to the person to the left, and you can get some advice. There's an incredible cohort experience. A lot of my best friends are still from this time.
I think we've lost that. The worst thing that happened to computer science education is cheap laptops, because cheap laptops mean that everybody scurries into their dorms. Many students tell me they still work collaboratively, but it's cliques rather than everybody in the same space.
I started teaching in Stanford in 2005 and, in the winter, Bill Verplank and I taught a design studio class. Bjorn was actually in the class, back there.
[laughter]
Scott:  Wendy Ju was the TA. She's somewhere in there. In this class, one notable feature of the student evaluations that came back was the, "How fair is the grading," questions. The course was rated in the 13th percentile. My education came back from an art school.
My feeling was, "Wow, the amount of feedback we gave these students was so much better than I ever got at RISD. These ingrates don't know what they're talking about."
[laughter]
Scott:  Eventually, I changed my mind. I had an interesting experience where I tried several experiments in the next few years and was able to inch that number up. I do think that part of the issue is that, being in an engineering school, the expectations for what is fair are very different. That was underscored for me when I went to my reappointment meeting with our deans. We got talking about teaching and the assistant dean says, "Wait a minute. You mean that your assignments are subjective?" He was totally horrified.
Thankfully my colleague Terry Winograd was there. He said "You mean, like this meeting?"
[laughter]
Scott:  What he's trying to say is that a lot of the most important kinds of assessment that we do are subjective things where the answer isn't B, or C, or D. What you're really trying to do is figure out what matters and what's important. Still, I think there's a long way we can go, in terms of making grading fair. In the summer of 2009, I got really interested in self‑assessment. If you are, or would like to be, this is a fantastic book that I highly recommend.
Here's the physical class that I teach online. Dan Weld talked about the phylogenetic tree and AI classes. I think if you did the same thing for HCI classes, you would find that James Landay's class is probably the root of a lot of these.
This is a project‑based course where we go through the whole user‑centered design cycle. There are a bunch of things that are interesting about this class. 160 students from 22 different majors. Very cool.
I think of one of the important roles that design plays in a computer science or engineering curriculum is it's the one place that we teach problem finding as well as problem solving.
When we introduce self assessment, one of the things that we did, not at all novel‑‑most of what I'm not going to talk about is novel, even though most of it was new to me‑‑is we created grading rubrics.
These help students be able to assess their own work. The weekly ritual for my class in the physical world is, each Friday at 8:00 AM, we have an assignment due.
You go to a 12 to 15‑person studio and then, at the end of studio, you open up your laptop and you assess your own work. It's important that you do it there. Otherwise, you'll run off and party, and totally forget about it.
This is our current ritual and then, later that afternoon, the teaching staff gathers together and grades everything blind.
In addition to the physical co‑location, so I can see what each of my peers are doing in the studio, which is a great way to get a feedback of, "Am I slacking or am I working well, or how else might I have done this?"
We've worked with John Mitchell's Coursework system so that all of the assignments that student post are write‑only up until the deadline, and read‑only afterwards. In the Facebook era, students absolutely look at each other's work.
Now, I got nervous when introducing self‑assessment. My goal was to say, "If students assess their own work, the ideal outcome would be that everybody just gets their own grade."
There are two worries you could have here. One is that students will say, "I get an A+," and another one is that they will say, "I'm not that good. I get a D‑," when in reality they did pretty well. Both of those seem unfair.
The algorithm that we currently use‑‑I won't claim it's optimal, just we've been trying it for the last couple years‑‑is you get your own grade if your self‑assessed score is close to your TA‑assigned score. If they're notably different, you get the TA grade. If you think about this for a little bit...
One student came up to me after lecture one day mad as hell, and he said, "This is totally an injustice. I can't believe this is so wrong," because what you could do is, students could give themselves epsilon above what they've actually earned, and you would get an unfairly high grade.
I shared this with the class the next lecture and I said, "One student's really worried about this."
"If you have such an incredible, intuitive skill of knowing exactly what your TA is going to give you, that every week you can give yourself epsilon above what they're going to give you, and you hit that maximum every week, that skill will be so valuable in your life ahead that I'm more than happy to give you an epsilon above your grade."
[laughter]
Scott:  What happened with that? We had a 0.91 correlation between the self‑assessed scores and the staff‑assigned scores, and 69 percent of students on any individual assignment got their own grade. This goes up over the quarter. It tends to be lower at the beginning and it tends to be quite high at the end.
Man 1:  Was there a consistent over‑assessment or under‑assessment?
Scott:  I've looked at this, and I tried to grab it when putting this talk together. I couldn't find it again. High performing students, on the whole, tend to underrate themselves, and low performing students tend to overrate themselves. This is consistent with all of the literature, and I think one reason that you see for this is...
What's interesting is that while, on balance, the average TA score and the average student score are about the same, but the variance of self‑assigned scores is narrower than the variance of staff‑assigned scores.
The bozo students say, "I did pretty good." They're deceiving themselves, or maybe they just don't want to face the music.
The superstar students, they're hard on themselves. They said, "I did pretty good, but not great. I could've done better." That's one reason why I think they're doing so well.
Man 2:  By the way, is another reason, possibly, motivation, that these students that underrate themselves try harder?
Scott:  Yes. One of the reasons why I think the best performing people do so well is they hold themselves to extremely high standards. All of us have had times in our life where we held ourselves to really low standards, and we're like, "Why does everybody think I'm such a schmoe?" Other times we're like, "I don't get it. Why is everybody else not taking this so seriously?"
I think that the one incredibly valuable role that self‑assessment plays is it holds up a mirror to, what standards do you hold for yourself and what are the broader standards that are being envisioned? I think you may still decide that you're going to hold yourself to extremely high standards.
Man 2:  It's different, what am I asking. You said standards, but people think that they're performing terribly and then work extra hard. It's not the same thing as holding a real high standard. I know someone who's very, very bright, who never thought he actually got anything. He's extremely bright. I thought to myself, "Wow, there's a meta‑cognitive notion of understanding where you decide that you must have been well enough to know it, but it's not probably well enough as somebody else." People who think they just can't get anything have these brilliant understandings. I'm saying it's something deeper than just higher standards.
Scott:  Yeah. Some of it falls into what you count under the umbrella "standard," but that's probably a longer question. There's a bunch of other universities that use these materials, and it's been interesting to talk with those that are using the self‑assessment materials, some are and some aren't, about how that's gone. I had an opportunity this past year to think about putting my class online.
When the first batch of Stanford classes was going out, Andrew asked me if I wanted to put the HCI class out. I said, "I'm not quite ready yet, but I'll do it. I'll do it this coming spring."
One of the things that was interesting was thinking about the limitations of multiple choice. One thing I've learned is that multiple‑choice questions are much more powerful than I first thought they were and designing good ones is really hard.
With that said, as Luis von Ahn pointed out as after dinner last night with Rosetta Stone, multiple choice is way tacky in comparison to broader free‑response. We settled on a technique called calibrated peer assessment. I learned about this particular technique from Ed Hutchins, who uses it in his anthropology class.
In the HCI online class that we taught, as Andrew alluded to earlier today, you would grade five of your peers, and I'll tell you a little bit more about that. That meant that you got five scores from each of your peers, and the score that was assigned to you was the median of those five scores. We sort of made that up.
It was fun to design this with Daphne, because we'd have these long meetings where we would start out with something extremely simple and, two‑thirds of the way through the meeting, we would have an eight layer Bayes net that was going to do something incredibly fancy.
Then, we realized that we wouldn't be able to reason about its failure modes and we had no idea how it would perform anyhow, so we would go back to the simple thing.
We did this for about six weeks straight. I keep coming back to the simple median of five. In the coming days and weeks, I hope they do some analysis of how well this could work. "Could you go for it? Could you go through these?" Stuff like that. "Could you do something fancy?"
We launched this class online. It was really fun. We covered about half the topics of my Stanford class, and it had the same video lectures and other stuff.
I recorded pretty precisely and spoke pretty quickly, which I don't normally do, but I try to thwart the people that watch at double speed, because I think sounding like a chipmunk for an entire quarter doesn't seem too appealing.
[laughter]
Scott:  We'll see how that goes. I think the best strategies are still open here. I'd be happy to talk about more about that video recording setup. I set off on and had a couple of innovations there. People were asking about numbers, so I pulled some quickly.
31 unique videos, a little more than 400 total minutes of video, took me forever to record. As in, I was in the lab until midnight everyday for most of spring quarter. I'm not at all convinced that the quality was better for it, but it did take a lot of time.
[laughter]
Scott:  Starting out, don't believe Andrew Ng when he tells you, "You just walk in the room and hit record." As a novice, it took me a lot longer. I think part of that is Ken’s point about experts are notoriously bad at estimating how long it will take novices to do stuff.
Scott:  Yeah?
Man 3:  Why? You're shooting a 20‑minute video. Why does it take so long?
Scott:  Because if you have the opportunity to record it with no deadline, you'll say, "That wasn't the best way to explain the topic. I should learn a little bit more about that." The way it would normally work is, for a 15, 20‑minute video, I would work on it for a couple hours, then I would hit the record button. I would record for about two hours straight, one take, because that made it easier for the particular video editing software we were using‑‑ScreenFlow, for those that are curious.
Most of that video is me thinking, or looking stuff up on the Internet, or fixing slides or whatever.
[laughter]
Man 4:  It's Scott Klemmer uncut, right?
Scott:  Oh, my TAs had fun doing that. [laughter]
Man 5:  It's free data for somebody who wants to study how professors author... [laughter]
Scott:  Oh, lord, is it not that interesting. [laughter]
Man 3:  Were you faster at the end?
Scott:  Yeah, yeah. I got a lot faster. Also, it's a medium to get used to, which also I'll talk more about offline. We had quizzes and all this other stuff, and a forum and assignments, and we made the rubrics better, and we had this peer evaluation system, which...I want to get to the punch line, so I'll fast forward through this, because Andrew talked about a bunch of it. One thing that's not unique about this class, but I think is interesting, is the most common crowdsourcing model is, "Hey Turkers, can you do this stuff for me?"
What I think is interesting about the online class is that there's a group of people, each of whom is helping each other, and the community field of this model of collaboration is importantly different than the online delegation model.
In particular, you get a real benefit in terms of the empathy, and student engagement, and it automatically scales, which is really nice.
It works best if the community's really enthusiastic. Jennifer Widom told me, "Don't worry about cheating; focus on the students who want to learn."
Woman 2:  How does that scale? If there are 100,000 people, how do you [indecipherable 17:32] with anyone?
Scott:  The scaling I was talking about, in particular, was the, "I grade five people. I get scores from five people," scales automatically. The community for 100,000 worked far, far better than I would've ever thought. If you just scan the forums, you'll get a sense of a strong community there. Jennifer told me, "Don't worry about cheating at all." That was really fantastic advice.
To sum up the differences between the online and the in person one, we got about half the material in about half the time. You get a statement of accomplishment. You could get an apprentice accomplishment for watching the videos and doing the quizzes, or a studio statement of accomplishment, if you also did all the projects.
The way it worked is you have assignments due weekly, and then, at the deadline, 11:00 PM every Tuesday, my TA would log on at 11:00 and grade eight to 10 examples, which would then serve as the training examples for the students to do peer assessment.
The calibration part of this is that you grade the same ones as the TA grades without seeing their score, and then you get feedback on how well you did. The hope is that you get more calibrated over time.
Man 6:  Did they do anything special to pick those eight to ten examples?
Scott:  Not really.
Man 6:  Like choosing, for example, from those scores who, in the previous assignment...
Scott:  We didn't do anything fancy anywhere, half intentionally and half because we were overwhelmed. In eyeball terms, we tried to get a range of abilities, like if the roll of the dice gives you ten As or ten Ds, they would pick a different one, but that's really it. Yeah, Lydia?
Lydia:  Did they grade it together?
Scott:  I had TAs sequentially, because copyright accessibility and other issues and developing the peer assessment system pushed our launch date back. When Robi graduated June 17th, Chinmay took over. David?
David:  This is kind of a [indecipherable 19:51] in the past. Median of five works fine if the flaky graders are equally likely to curve upward or curve downward. Is that what you saw, or do they tend to always grade high or always grade low?
Scott:  Hold that thought.
Man 7:  I have another question.
Scott:  Yep?
Man 7:  If the students have to first grade up to five assignments that the TA graded, and then grade another five, wasn't that overly excessive?
Scott:  The grading workload is not low. Each student had to do a fair amount of grading. Many believed that was really valuable. Many also believed that it was burdensome. We had eight training examples, in case you did badly on the first couple, so we had a train‑to‑mastery strategy. Once you did well in the training phase, you got let out into the broader garden.
Man 7:  We have been hearing about the fact that 50 percent or 60 percent only submitted the first assignment correctly, and had to make it up at the end of the course. By having peer review assignment, students are relying on the peers actually being serious and putting in the effort, because otherwise some other student who is very serious hurts, and that hurts the experience.
What do you find [indecipherable 21:21] the participation of the assignment more casual and not very seriously?
Scott:  All of the numbers that you hear for these online courses are more misleading than they are useful. The big giant number that's like, "1,200,000 signed up for my class," really means, "1,200,000 typed their email address into a field," which is staggering but that's not at all the same thing as, "Took my class." What is notable is that, if you have, "How many people watched at least one video for this class?"
I just cut off the, "Type their email address in," because I think it's a bogus number, but if you say, "Watch at least one video," that's a pretty big number. 7,000 submitted at least one quiz.
2,500 completed at least one assignment, so the 27,000 people that did not, cannot peer grade. In order to be allowed to peer grade, you need to have submitted the assignment. There are multiple reasons why we did that. You can guess what they are.
We had 800 complete all five. We had weekly assignments, which was a stupid idea, in retrospect. If you have in‑person students, weekly works great. If you have people who have families, and travel and lives, weekly deadlines just don't work. When we roll out the course again in the fall, we will probably space that out more.
We had 10,000 people fill out the survey, came from 124 countries... Yeah?
Man 8:  Can you elaborate on your opinions about weekly assignments?
Scott:  There are two deadline strategies for an online class. One is a hard deadline strategy, where we say, "After Tuesday at 11:00 PM, you're out of class. There's no submitting after that." The other one is a soft deadline strategy, where we say, "You can submit after Tuesday at 11:00, but we'll dock you some points." If you have a soft deadline strategy, short, fast assignment deadlines are great, because people don't do it until the last day.
If you have a hard deadline strategy, it is totally true that on any given week, somebody's grandma's going to be sick and somebody else is traveling. For hard deadlines, rapid fire doesn't work.
Man 7:  Can I quickly re‑ask my question?
Scott:  Sure.
Man 7:  Even if people who actually did these assignments work really hard, were there cases where people give the assignments on time but did not give the grade on time, which hurt the overall productivity of the system, of the grading process?
Scott:  Great question. Assignments were due Tuesdays at 11:00 PM Pacific. The peer assessment was due Sundays at 11:00 PM Pacific, I think, a couple of days later. You did not get credit for submitting an assignment unless you submitted the peer assessment. People of all ages...given all of the questions, if you should tell me to stop talking at some point, I won't feel the least bit bad. The questions are more exciting than the content.
We do lots of different stuff. There were all these meet‑ups...
Man 9:  Was that self‑employed, was your big...
Scott:  Yeah. My hunch is that there were a lot of young stay‑at‑home parents. People transcribed at least some... Coursera contracts with Lamar to provide transcripts for the videos. Not all of them were translated into all of these languages, but at least one was translated into each of these languages and that was done by the students in a crowdsourced fashion, which is pretty cool.
Woman 3:  Going back to the occupations, you said there were lots of stay‑at‑home parents. One thing I've been wondering about all these lectures is, what were the gender demographics like? I'm wondering if you have more women in these classes than in traditional computer science classes.
Scott:  We had 10,114 people fill out the survey. Of those, 3,113 were women, so that's one‑third women, which is slightly higher than my in‑person class, and both of those numbers are significantly higher than the other computer science classes in their cohort.
Man 10:  Do you have a sense for a link between some demographics, like gender and completion?
Scott:  That would be in the data, and we have not yet looked at it. Here are study groups from all over the world. That was totally one of the most rewarding things about the class. It was really fun to watch the forums just come alive as soon as the class launched, and have people doing fellow Russian study group, and Tagalog study groups. I don't even know what the Hungarian one says, and they seem to be having a grand time.
Man 10:  Do you know if there were any physical groups?
Scott:  Yes. We had ten physical meetup locations where people were getting together on a regular basis. Andrew showed you a photograph of a San Francisco meet‑up for my class. Here are some graphs, and some things that happened. You heard from Jennifer and Andrew about Amy Cunningham. The course launched on my birthday, and I woke up that morning to my email. One of my emails was from Jane, who is the shepherd of all the online classes. She's really the person who makes all of this stuff happen. We had Amy, which was great.
Here are some assignments that people did, so this is from Russia, this is an e‑finding assignment. Here's one from Spain that was an e‑finding assignment. Then, we moved on. We used Balsamiq for rapid prototyping. Again, Russia and Spain.
People created limitation plans for the same assignments, and then they go out and do user testing. They did a really great job. The quality was extraordinary.
Man 9:  Do they test online or in‑person?
Scott:  We did sort of small scale usability testing. In my Stanford class, we actually run AV tests online and get into control experimentation and online testing a little bit more. I may do a follow‑on chunk of this class that includes more online testing
Man 11:  The lectures got the translated, but the assignments were all in English?
Scott:  Assignments were only available in English. Why we didn't translate them, I don't know. It was required that people submit in English. We didn't make that clear enough for the first assignment. We got stuff back in all languages. [laughter]
Scott:  One open question is, I really want to, as an experiment, allow Spanish as a second submission language. There are a huge number of Spanish speakers in this class. Coursera is not quite ready to do that, but I'm kind of hoping] to do Spanish.
Man 9:  Was that the second biggest language?
Scott:  Yes, you could have the Spanish students grade each other. That's exactly what I'd like to do. I think the best way to go with all of these things is baby steps, so I think introducing one more language would be the next step to do. Spanish, for my class, was by far the second largest language. I may have country data in here if we get that far. Peer itself correlate pretty well. You do see that there are a whole bunch of people who self‑grade 100 where it's not reasonable, but that's not as flagrant as you might wonder. I was really worried about doing this peer grading and scale. I took it to be just an enormous risk that was probably going to blow up extremely badly.
In fact, we held off releasing most of the assignments for the class until we saw the first one, because we were going to change everything if it went badly. I kind of thought it would, but Daphne and Andrew, in that case, gave me a lot of courage to try it, and it worked pretty well.
As you compare them, there's a pretty strong correlation, and more than half of the students, even in the online class, are receiving their own grade, which I think is super cool.
Man 7:  I have a question about...This came up in Andrew's talk also, where he showed the diagram of the previous study, where he showed the correlation. The correlation is great, but then we were talking about the global level, we were talking about the statistical level, and we said that, in general, the trend is good.
But, I think any individual student will believe that he has or she has done good work, but the grade that they received that, sometimes in fairness, is actually much lower, and maybe much higher also. Then, that leads to a bad experience.
Is there a mechanism in place to improve it? Did you see these cases? There also the question of [indecipherable 30:38] , because statistics, to me, just doesn't cut it.
Scott:  Sure. For me, the most interesting number is, for a pretty tight threshold, half the students are receiving their own grade. That says half the students and their peers are spot‑on in agreement. Now, you could flip that around and you could say, "But the other half are not." I think what I was really surprised by is, I was ready for a good fraction of the class was going to be real cranky about their grade. That did not turn out to happen.
There are that many people who are cranky about their numerical score. There are a much larger number of people who are cranky about the fact that they didn't get more qualitative feedback, which, given the language diversity issues, we limited the amount of qualitative feedback that students could give to each other.
Man 11:  What percentage of the students Lee described as behaving badly where your resources would overwhelmed? 10 percent, 20 percent badly? They grade themselves 100 and then they yell at you about it. Where does that threshold for you?
Scott:  I don't know, and I'm sort of an empiricist on this one. I think in these classes, if you had a large number or even a modest number of rows, you could beat them. My sense is fight the fires as they come up. David?
David:  A pair of shortcuts. First off, I just realized, in doing this grading [indecipherable 32:12] college credit so that their grade matters. I mean, they got their certificate but...
Scott:  People care enormously about their grade, even though you could argue it doesn't matter. I'll give you a data point for that. On one of our quizzes, for reasons that I still don't understand, Chinmay midstream changed one of the quiz questions and it switched which answer was right. For the first chunk of students, B was the right answer. For the next chunk of students, C was the right answer.
This is one of these things like, "In the fog of war, you make a dumb mistake." The amount of venom that went onto the forums about that inconsequential change in grade, well it's hundreds and hundreds of pages. If people are that bent out of shape about a single quiz question, and they're not about this...
I had the same experience teaching my HCI class at Stanford where I expect that one of the major things that people are going to weigh in on is some sort of evil around self‑evaluation. They don't even notice it. It's, "In lecture seven, there was a typo on the slide."
[laughter]
Scott:  It's pretty interesting.
David:  When you give them their grade, do they, nevertheless, see the TA, or their peer grade?
Scott:  What do they see, when they receive own grade? For both my in‑person class and online, we talked about a bunch of different options. I'm not sure what the current implementation is. That's an excellent question.
David:  Do you give a letter grade or percentile? Do you curve it? 
Scott:   At the end of the class, if they're above some threshold, they get a statement of accomplishment, which says, "You did the HCI class. You scored X out of Y."
David:  I guess this is where you asked me to hold my question, if median is the right thing. Was the median the right thing? Were the students biased enough, so that you should say you take the second highest instead of median?
Scott:  You're exactly right that this was the graph that I was holding for and it doesn't really answer your question, because it only shows the median score on the Y‑axis. It doesn't show all five. I thought you might ask that question given that we did this analysis for the CHI data several years ago. I was hoping to have that answer for this talk, and that one didn't happen yet. Andy?
Andy:  Just add some data. I do a lot of peer assessments in my classes in the same way. The problem is actually more simple, because even though you have some people [indecipherable 35:09] , really nobody can get a good score on the assignment.
Scott:  It's the same regression to the mean as when you have twelve reviewers on your paper. [laughter]
Scott:  All right, this was fun. "How much do you feel you learned by assessing others' work?" Likert scale, lots, little. People learned lots from assessing others' work. "How much did you feel like you learned from assessing your own work?" Still strongly headed this way, but less so. I take that to mean that people felt like they learned a lot in assessing other students' work, and I think that it's absolutely true.
Andy:  Do you have a sense for...they could learn a lot from looking at other peoples work, independent of whether they assess it. I think they're both interesting.
Scott:  Yes, the number of controlled experiments you could run on this domain is huge. I have a list of them, and we've done none of them.
Woman 1:  Just to interrupt for a second. In the interest of time, you have two more minutes worth of stuff, so you can decide if you want to take questions, or if you want to present more slides, it's up to you.
Scott:  We'll do questions till the end. Interestingly and perhaps unsurprisingly, a clever question that Jane put in the quiz was how much effort you put into assessing others and how much effort you thought the person on the other side of the table spent assessing you. The people taking this class are human. They believe they put lots of work into it. They thought their peers put lots of work into it too, but less than they did.
[laughter]
Scott:  A real cool one, inspired by some research that Pam Heinz has been doing on cross cultural perceptions of design, Chinmay decided to look at...We don't tell you what country the assignment you're grading is from or the name of the assignment you're grading. Even though you're blind to country of origin, you may be able to infer it in various ways. Do you grade assignments from your own country higher than others? The answer is, yes, robustly so, and even if you hold out any of the usual suspects.
You can include the US or not. You can include other large countries or not, and this trend continues. We don't know cause, but I think it's really interesting. I'll pick one last slide to show.
Woman 2:  Do you think that's specific to HCI design?
Scott:  I don't know. It's just this weird thing that happened.
Man 7:  I will say, obviously same language grading, because you will have more than one scores pushed up in some direction here if you have more than one grading policies.
Scott:  Sure. The really interesting direction for future work is how do you clump the tiers. There's lots of strategies. The last thing I'll say and then I'll take future questions after Bjorn and Rob get a chance to talk  I was amazed by how much DIY there was by students on the forum. Not only by Amy Cunningham, but by lots of students.
One example is, one student took all of the readings I recommended and put this up as a resource for everybody. For each of the lectures, you can see all of the readings that I refer to in the lecture. Yeah, the student may also get a couple of bucks from Amazon for doing that, but it's pretty cool.
Transcription by CastingWords