Designing Interactions that Combine Pen, Paper, and Computer

Ron Bing Yeh
PhD, 2007
Pen and paper are powerful tools for visualizing designs, penning music, and communicating through the written language. This coupling is mobile, flexible, graspable, and robust. It has even evolved in response to technology: we print out electronic articles to read, and scribble annotations on them before meeting with colleagues. The introduction of digital pens that capture handwriting has now made it feasible to augment forms, notepads, and maps with computation. Applications can recognize handwriting, upload notes to the Web, and detect pen gestures for initiating a search. In doing so, we combine paperís affordances with the benefits of technology, including search, redundancy, and remote collaboration. We refer to these as paper+digital interfaces. Developing paper+digital interfaces is challenging. Programmers need to abstract input into high-level events, coordinate interactions across time, and manage output on devices. This is difficult, because interface programmers are accustomed to working with graphical applications that provide real-time feedback on a single display. Additionally, debugging paper interfaces requires added effort, such as printing an interface before testing. As a result, few people currently build paper+digital interfaces. This dissertation explores how pen and paper can be used in concert with computers to make tasks more efficient, engaging, and robust.


This project is known for: fieldtools butterflynet