Externalizing Mental Images by Harnessing Size-Describing Gestures: Design Implications for a Visualization System

Brown, S. A., Chu, S. L., & Rani, N. (2020, September)

AVI ’20: Proceedings of the 2020 International Conference on Advanced Visual Interfaces

https://doi.org/10.1145/3399715.3399920

People use a significant amount of gestures when engaging in creative brainstorming. This is especially typical for creative workers who frequently convey ideas, designs, and stories to team members. These gestures produced during natural conversation contain information that is not necessarily conveyed through speech. This paper investigates the design of a system that uses people’s gestures in natural communication contexts to produce external visualizations of their mental imagery, focusing on gestures that describe dimension-related information. While much psycholinguistics research address how gestures relate to the representations of concepts, little HCI work has explored the possibilities of harnessing gestures to support thinking.

We conducted a study to explore how people gesture using a basic gesture-based visualization system in simulated creative gift design scenarios, towards the goal of deriving design implications. Both quantitative and qualitative data were collected from the study, allowing us to ascertain what features (e.g., users’ spatial frames of reference and listener types) of a gesture-based visualization system need to be accounted for in design. Results showed that our system managed to visualize users’ envisioned gift dimensions, but that visualized object area significantly affected users’ perceived accuracy of the system. We extract themes as to what dimensions are important in the design of a gesture-based visualization system, and the possible uses of such a system from the participants’ perspectives. We discuss implications for the design of gesture-based visualization systems to support creative work and possibilities for future directions of research.