CS547 Human-Computer Interaction Seminar   (Seminar on People, Computers, and Design)

Fridays 11:30am-12:30pm PT · Gates B3 · Open to the public
Previous | Next
Henry Lieberman
MIT Media Lab
Integrating User Interface Agents with Conventional Applications
January 9, 1998

In most experiments with user interface agents to date, it
has been necessary either to implement both the agent and the
application from scratch, or to modify the code of an existing
application to enable the necessary communication. Instead, we
would like to be able to "attach" an agent to an existing
application, while requiring only a minimum of advance planning
on the part of the application developer. Commercial applications
are increasingly supporting the use of APIs and scripting languages
as mean of achieving external control of applications. Are these
mechanisms sufficient for software agents to achieve communication
with applications?
I report some preliminary experiments in developing agent
software that works with existing, unmodified commercial applications
and agents that work across multiple applications. I describe
a programming by example agent, ScriptAgent, that uses a scripting
language, Applescript, to record example procedures that are
generalized by the agent.
Another approach is examinability, where the application grants
to the agent the right to examine internal data structures. We
present another kind of learning agent, Tatlin, that compares
successive application states to infer interface operations.
Finally, we discuss broader systems issues such as parallelism,
interface sharing between agent and application, and access to
objects.


Henry Lieberman has
been a Research Scientist at the MIT
Media Laboratory's Software Agents group since 1987. His
interests are in the intersection of artificial intelligence
and the human interface. His current projects involve intelligent
agents for the Web that learn by "watching what you do". He
has also built an interactive graphic editor that learns from
examples, and from annotation on images and video. He worked
with graphic designer Muriel Cooper in developing systems that
supported intelligent visual design. Other projects involve
reversible debugging and visualization for programing environments,
and new graphic metaphors for information visualization and navigation.
From 1972-87, he was a researcher at the MIT Artificial Intelligence
Laboratory. He started with Seymour Papert in the group that
originally developed the educational language Logo, and wrote
the first bitmap and color graphics systems for Logo. He also
worked with Carl Hewitt on actors, an early object-oriented,
parallel language, and introduced the notion of prototype object
systems and the first real-time garbage collection algorithm. He
holds a doctoral-equivalent degree from the University of Paris
VI and was a Visiting Professor there in 1989-90.