High Performance Pen Interfaces

   Ken Hinckley, Microsoft Research

Seminar on People, Computers, and Design
Stanford University October 22, 2004

Much of the user experience with current pen-operated mobile devices is clumsily adapted from the point-and-click interfaces of desktop graphical interfaces. I will discuss some efforts at Microsoft Research to advance the state-of-the-art in pen interfaces, including new capabilities made possible by using the pen for collaborative interactions, as well as my take on the classic problems of gesture-based interaction. The talk will focus on two projects, Stitching and Scriboli.

Stitching is a new interaction technique that allows users to forge wireless network connections between pen-operated mobile devices, by using pen gestures that span multiple displays. To stitch, a user starts moving the pen on one screen, crosses over the bezel, and finishes the stroke on the screen of a nearby device. This can be used as the basis for a variety of collaborative tasks, such as copying images from one tablet to another that is nearby, expanding an image across multiple screens, or establishing a persistent shared workspace. I will also discuss design issues that arise from proxemics, that is, the sociological implications of users collaborating in close quarters.

Scriboli is a pen interaction testbed. Our fundamental goal in Scriboli is to design a grammar for pen interfaces, including nouns (scopes), verbs (commands), and delimiters that help the system determine the structure of pen gesture input phrases. I will discuss our initial progress towards this goal and demonstrate how Scriboli can support integrated scope selection, command activation, and direct manipulation all in a single fluid pen gesture. Hence the work offers a nice extension of Stanford alumni Francois Guimbretiere's work with the FlowMenu (Francois is also a collaborator on Scriboli). The name "Scriboli" is a play on the Tivoli electronic whiteboard (Xerox PARC) that evokes the fast and informal "scribbling" nature of the interactions we seek to design.

Ken Hinckley is a research scientist at Microsoft Research. His interests span many areas of human-computer interaction including input devices, interaction techniques, sensors and sensing systems, mobile devices, and ubiquitous computing. For recent papers, see his publications.

The projects I will discuss are collaborative efforts. Collaborators on Stitching are Gonzalo Ramos, Francois Guimbretiere, Patrick Baudisch, and Marc Smith; for Scriboli, they are Patrick Baudisch, Gonzalo Ramos, and Francois Guimbretiere.

Yang Li (first author), Zhiwei Guan, and James Landay of the University of Washington conducted an ink/gesture mode switching study on which I was a collaborator, and will discuss in the context of Scriboli.


View this talk on line at CS547 on Stanford OnLine

Titles and abstracts for previous years are available by year and by speaker.