Beyond Being Real: A Sensorimotor Control Perspective on Interactions in Virtual Reality

Parastoo Abtahi, Sidney Q. Hough, James A. Landay, Sean Follmer
The ACM Conference on Human Factors in Computing Systems, 2022
We can create Virtual Reality (VR) interactions that have no equivalent in the real world by remapping spacetime or altering usersí body representation, such as stretching the userís virtual arm for manipulation of distant objects or scaling up the userís avatar to enable rapid locomotion. Prior research has leveraged such approaches, what we call beyond-real techniques, to make interactions in VR more practical, efficient, ergonomic, and accessible. We present a survey categorizing prior movement-based VR interaction literature as reality-based, illusory, or beyond-real interactions. We survey relevant conferences (CHI, IEEE VR, VRST, UIST, and DIS) while focusing on selection, manipulation, locomotion, and navigation in VR. For beyond-real interactions, we describe the transformations that have been used by prior works to create novel remappings. We discuss open research questions through the lens of the human sensorimotor control system and highlight challenges that need to be addressed for effective utilization of beyond-real interactions in future VR applications, including plausibility, control, long-term adaptation, and individual differences.