Intelligent Verification/Validation for XR Based Systems
1. The future of XR: Current ecosystem and upcoming opportunities
Intelligent Verification/Validation for XR
Based Systems
Rui Prada
Instituto Superior Técnico, Universidade de Lisboa
INESC-ID
The future of XR: Current ecosystem and upcoming opportunities - May 27, 2021
https://iv4xr-project.eu
Project Ref. EU H2020-ICT-2018-3 - 856716
3. Testing XR Systems
Highly interactive
Rich and complex environments
Diverse multi-modal input and output
Testing demands high human labour (user
testing)
5. Team
University and Research
- INESC-ID, Portugal (Coordinator)
- Utrecht University, The Netherlands
- Fondazione Bruno Kessler, Italy
- Universitat Politecnica Valencia, Spain
- Umeå University, Sweden
Industry
- Gameware Europe, UK
- Good AI, Czech Republic
- Thales (SIX and AVS), France
https://iv4xr-project.eu
6. Autonomous Testing Agents
Actively pursue testing goals
Intelligent coverage of the interaction
space
Identify potential interaction paths
Represent users with different profiles
7. Testing goals
Testing Functional Properties
The behaviour of the system
Testing User Experience (UX)
The impact on the user
10. https://github.com/iv4xr-project
aplib + iv4XR EDSL
aplib:
basic agents and runtime system
action and tactic
goal and goal-combinators
iv4XR:
test agents and testing related basic support
extensions: pathplanning, world object model, emotion,
learning, ...
iv4XR
framework,
from a different
perspective
https://github.com/iv4xr-project
13. Two types of FTA
Two types of FTA’s are being
developed:
The first type of agent makes
deliberations to choose the
appropriate strategies that will
allow him to do goal-solving.
The second type of agent is
intended to test the
functionality of the XR system
using exploration.
15. Fully Automated Exploring/Testing
Space Engineers
TESTAR as iv4XR agent
While exploring:
We test for robustness
To infer a model that can be used
for Model-Based Testing (MBT)
Find the best representation of
the World Object Model (WOM)
19. Automated assessment of UX
Agent
Cognitive
model
Emotional
model
Difficulty
Interaction
Policies
Personas
20. Emotional Prediction Through Machine
Learning
Predictive Model
Feed
Predicts
Expected
Emotional
Changes
Emotional
Changes
(Continuous Self
Reporting)
Training
SUT
(Observations)
21. Emotional Prediction Through Machine
Learning
The Case Study:
A simple top-down 2D
game
Continuous self-
reporting after the fact
of the 3 dimensions of
the PAD emotional
model (Pleasure,
Arousal and Dominance)
22. Emotional Prediction Through Machine
Learning
The emotional
reporting was cut into
slices, which were then
categorized as
decreasing, steady or
increasing.
A machine learning model
based on Random Forests
was then trained to
predict these 3 classes.
24. Automated Assessment of Cognitive
Emotions (OCC Model)
Intelligent agents are deployed to generate tests based on UX test
specifications. This is achieved by deploying a Computational model of
emotion designed to provide affective processing in our intelligent agents
26. Three Pilot Studies
We are making use of three pilots to test the framework:
Space Engineers - a 3D game
Players make use of several tools and fabrications (blocks) to obtain
resources and explore a solar system with player constructed vehicles and
machines.
Maev - A simulation environment
A defend/attack scenario in a nuclear powerplant. Defenders test security
measures to stop external intrusions.
LiveSite - A monitoring system for infrastructure and construction sites
Test the reading of sensors.
Sensors at these sites can produce up to thousands of readings per
second. There are many virtual sensors.
27. How are we using the testing agents?
Space Engineers requires ~20000 tests for each major
release.
One arduous test that we are working on automating with
iv4XR is checking textures of blocks depending on their build
state.
The Maev powerplant scenario currently requires two teams
of humans:
Iv4XR will take control of the incursion forces, automatically
testing the efficacy of the proposed security measures.
LiveSite buildings and critical infrastructures have a large
number of sensors that produce many readings per second.
Iv4XR constructs tests based on their predefined relationships
and thresholds, but also investigates found errors to
determine if the readings are correct, or if there is some
anomaly (faulty/miscalibrated sensors).
28. Intelligent Verification/Validation for XR
Based Systems
https://iv4xr-project.eu
@iv4xr
https://github.com/iv4xr-project
rui.prada@gaips.inesc-id.pt
Project Ref. EU H2020-ICT-2018-3 - 856716