Ideally, man-machine interfaces should respond to people's intuitive modes of communication. Recent advances in real-time computer vision offer the promise of endowing man-machine interfaces with direct perception of human users. In this talk I'll first discuss visual control and compositing in virtual environments, including direct navigation and manipulation. I'll then cover recognition of hand and face gestures: offering iconic, diectic, and emotional signals to computer systems. Finally I'll discuss Avatars and agency, and in particular the issue of literal control vs. autonomy in virtual user representations. I'll show examples of several systems developed which use visual perception-- including real-time hand, face, and body tracking-- to create interactive virtual environments. Each of these systems utilizes real-time vision techniques with no gloves, markers, or other intrusive artifacts.