Integrating User Interface Agents with Conventional Applications

Henry Lieberman, Media Laboratory, Massachusetts Institute of Technology
lieber@media.mit.edu

Seminar on People, Computers, and Design
Stanford University January 9, 1998

 

In most experiments with user interface agents to date, it has been necessary either to implement both the agent and the application from scratch, or to modify the code of an existing application to enable the necessary communication. Instead, we would like to be able to "attach" an agent to an existing application, while requiring only a minimum of advance planning on the part of the application developer. Commercial applications are increasingly supporting the use of APIs and scripting languages as mean of achieving external control of applications. Are these mechanisms sufficient for software agents to achieve communication with applications? 

I report some preliminary experiments in developing agent software that works with existing, unmodified commercial applications and agents that work across multiple applications. I describe a programming by example agent, ScriptAgent, that uses a scripting language, Applescript, to record example procedures that are generalized by the agent.

Another approach is examinability, where the application grants to the agent the right to examine internal data structures. We present another kind of learning agent, Tatlin, that compares successive application states to infer interface operations. Finally, we discuss broader systems issues such as parallelism, interface sharing between agent and application, and access to objects.

 

Henry Lieberman has been a Research Scientist at the MIT Media Laboratory's Software Agents group since 1987. His interests are in the intersection of artificial intelligence and the human interface. His current projects involve intelligent agents for the Web that learn by "watching what you do". He has also built an interactive graphic editor that learns from examples, and from annotation on images and video. He worked with graphic designer Muriel Cooper in developing systems that supported intelligent visual design. Other projects involve reversible debugging and visualization for programing environments, and new graphic metaphors for information visualization and navigation. 

From 1972-87, he was a researcher at the MIT Artificial Intelligence Laboratory. He started with Seymour Papert in the group that originally developed the educational language Logo, and wrote the first bitmap and color graphics systems for Logo. He also worked with Carl Hewitt on actors, an early object-oriented, parallel language, and introduced the notion of prototype object systems and the first real-time garbage collection algorithm. He holds a doctoral-equivalent degree from the University of Paris VI and was a Visiting Professor there in 1989-90.

 

Titles and abstracts for all years are available by year and by speaker.

For more information about HCI at Stanford see