Highly Interactive Visualizations in Augmented Reality

Leonel Merino

Visualizations enable us to interact with the visual representation of data and to identify complex patterns. Now imagine that you don‘t need to sit in front of a computer display and use the keyboard and mouse to interact with a visualization, but that you can walk through human-sized visualizations while selecting elements with hand gestures. These are Situated Visualizations. They allow a new form of interactive analytical thinking. However, existing toolkits for creating such Situated Visualizations are not yet flexible or agile enough to produce usable visualizations. As a first step to achieve Agile Situated Visualizations, we have developed AVAR. AVAR is a toolkit that generates particularly lively and expressive visualizations.

Visualizations In-Situ

Through Situated Visualizations, complex data sets are made visually perceptible in a new way.
Augmented reality is used to embed these visualizations directly in our physical environment.
They aim is to close the gap between people and their data by supporting data understanding and decision making in context. Moreover, Situated Visualizations stimulate cognitive aspects such as engagement, embodiment, and recall.
Existing toolkits are not yet flexible or agile enough, which makes the resulting Situated Visualization less applicable. There is a lack of toolkits that guide the authoring of Situated Visualizations and, at the same time, offer ready-to-use building blocks to speed up development.

Fig. 2: Visualization embedded in the physical environment. Example scenario for interacting with an Agile Situated Visualization using AVAR.

Our Approach

With our toolkit AVAR, one can quickly create Agile Situated Visualizations that combine real objects with visual representations of data. We hope to promote logical thinking in this way.
The advantage of Agile Situated Visualization is, that one can easily explore multiple facets of a data set without having to leave the immersive environment in augmented reality for changing the parameters.
Users can add previously filtered data and see directly the view changing. AVAR has to overcome three main challenges: short feedback loops when modifying visualizations, integral support for defining and interacting with visualizations, and be expressive enough to define multiple visualization designs, but simple enough to be used in immersive augmented reality.

AVAR: An Agile Situated Visualization Toolkit

AVAR is a prototypical implemention for Agile Situated Visualization. In it, users can interact with the visualization by simply walking through it while employing a Microsoft HoloLens® and a Bluetooth keyboard as input/output devices.

Fig. 2: Examples for Interacting with Agile Situated Visualizations using AVAR.

AVAR integrates a fully operational programming language, Pharo Smalltalk, into an immersive environment. Why Pharo? Because it is interpreted and highly reflective. This means no waiting for a script to compile, but working in a live environment, and an eased integration to external environments. In addition it has powerful 2D/3D visualization engines (Roassal2 and Woden), which implement multiple visualization techniques such as parallel coordinates, treemaps, node-link diagrams, and space-time cube matrices.
AVAR also integrates a Unity 3D application that communicates with Pharo as a backend, handles user interaction, and renders a graphical user interface in augmented reality.
We examined the usage of the AVAR toolkit for authoring Situated Visualizations in a study with seven expert Pharo programmers.
We could confirm two things by working with Agile Situated Visualizations. First, that users need to address visualization processes in a continuous loop and not sequentially. And second, that users depend on having multiple features available to express various visualization designs in an iterative fashion.


Situated Visualizations in immersive augmented reality offer the possibilty to explore visualizations of real world data embedded in their physical context. With Agile Situated Visualizations one can build multiple Situated Visualizations in an agile fashion. This is a new way to examine various facets of a data set.
Our toolkit AVAR is the first step towards applicable Agile Situated Visualizations. With AVAR one can deal with a visualization as a whole without leaving the immersive environment. Our design choices impacted the users‘ behavior, namely spending much more time with the visualization. Especially the live feedback boosted the highly interactively engagement. The employment of multiple available features when programming a Situated Visualization is a real advantage. In the future, we plan to improve our design, investigate other means for interaction, and conduct further evaluations.

About the author

Leonel Merino is the Manager of the Graduate School. He conducts research on the intersection of data visualization, human-computer interaction, software engineering, and virtual and augmented reality.


This issue is a contribution to the DFG2020 campaign Because Research Matters.

@dfg_public #fortheknowledge



Print Version INSIGHT No. 1

INSIGHT No. 1 – Highly Interactive Visualizations in Augmented Reality

Responsible for the public relations activities of the SFB-TRR 161. He cares about the communication between science and public with the aim to provide an insight into the world of research.

Leave a Reply

Your email address will not be published.