CS377A Project Proposal
Toolkit for orientation/gesture based interaction on mobile devices

Project Team

Group meetings are: Tues 11am-1pm near Wallenberg, sometimes Fri 2-3pm Gates (B01)

Abstract

We will study the feasibility of orientation- and gesture-based interaction for handheld devices.

Several people have already developed applications that use sophisticated (and expensive) position/orientation tracking techniques which are not very mobile. This project will instead find out what can we accomplish with cheap and very mobile tracking which is less accurate.

Our first step will be to build a hardware/software toolkit that allows us to write pocket PC applications which are aware of device tilt and a crude magnetic compass. Unlike some previous approaches, these methods of orientation are nearly ubiquitous and cheap to implement.

Sample applications could include some type of mapping, friend finding, virtual bulletin board, controller, and gesture interaction, human studies. We will decide specifics after getting some familiarity with the technical limitations of the orientation data we have as well the display limitations of the devices.

Prior Art

The concept of virtual reality has been around for decades, and several pioneers have developed wearable computers (see this account by Steve Mann of MIT). To most people, however, wearable computing is neither affordable or socially acceptable. It would be nice if we could somehow make use of the ideas of these pioneers in a socially acceptable format.

Buxton and Fitzmaurice provide a good comparison of different types of virtual reality including Chameleon, which uses a position tracking system and a handheld display to create a window into a virtual reality. The Chameleon system seems like a compelling compromise between immersion and usability. Because it was developed in the 1990's, these systems were not portable - they require a graphics workstation and position tracking technology using a mechanical boom or short range magnetic sensors. It's not clear if the 3D approach would work with crude position sensors.

Ping Yee's Peephole used an Ascension Tech "Bird" for position and orientation tracking. The device can be made mobile relative to a fixed base (tabletop) or the user (backpack), but doesn't provide an absolute orientation. Although Peephole used 3D tracking, the applications tested were not really applications in the sense of Chameleon, but 2D applications that made use of 3D position as an input to pan or zoom.

Placelab is a position tracking system that uses 802.11 and other radio beacons for passive position tracking. In one demo they provide accuracy to about 20m. We assume that this technology will improve over time, so we might make use of it, but are going to emphasize the orientation aspect.

The Studierstube project has a very exciting effort to develop handheld augmented reality. These devices seem to principally use computer vision for position/orientation tracking, but show that the 3D graphics problems are solvable on handhelds.

UCSD Active Campus project has some interesting applications using PDAs. These appear to be 2D map-based, so maybe we can contrast with their approach.

Cricket is a system at MIT which uses ultrasonic beacons to establish position and orientation indoors.

Many papers warn that magnetic disturbance makes compasses limited indoors or in vehicles. [1] [2] [3] Limited testing with a compass indoors shows that magnetic disturbances are profound near common items (speakers, large appliances, automobiles). It is possible to detect when a magnetic disturbance is occuring. Perhaps the demo will only work in limited circumstances or will require manual correction factors.

Driving questions

Application Ideas

After the toolkit is done we will pick two or three of these to test with.

Note: The team hasn't formed a consensus yet about what applications are the best.

Quarter Deliverables

References

Preliminary Implementation Description

Data acquisition hardware

  • Tilt sensing will be done with two Analog Devices ADXL202E.
  • 3D Magnetic field strength will be measured by HMC1002
  • We'll bring the data to a an Atmel AVR ucontroller
  • The ucontroller's serial port will connect to a bluetooth transmitter, probably this one from Lemosint. (We might decide that the bluetooth link isn't worth the effort and instead just use a serial cable/PC link for the demo.)
  • Bill of materials is about $100, it would be less than $50 if we integrated the sensors into the PDA and had volume purchasing.

    Software infrastructure

  • Target platform is C#.net with pocket PC
  • Need a modules that receives the sensor data
  • Need a module that conditions the tilt sensor data - calibration/static offset removal/noise filtering
  • Need a module that conditions the compass data
  • Consider integrating with Placelab positioning?
  • Need to write/locate routines for sparse 3D graphics

    Straw Man Schedule

    Note by joel: not ratified by team