Work torwards the creation of a tool to test narrative games during the writing of the story, without the need of users. The tool presents a set of metrics that can be used to assess the experience that the story may promote, in terms of its interaction flow. Presentation at Conference on Games (Cog) 2021.

1. Validating the plot of Interactive
Narrative games
https://iv4xr-project.eu/
Carolina Veloso and Rui Prada
INESC-ID and Instituto Superior Técnico, Universidade de Lisboa
Portugal

2. Problem
- Authoring interactive narrative is hard
- Need to balance authorial control and players’
agency
- Most solutions include runtime adaptation (e.g.
drama managers)
- There is a lack of tools to facilitate testing at
authoring time (before putting players in the loop)
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

3. E.g. Space Modules Inc.*
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716
*Samuel Mascarenhas, Manuel Guimarães, Rui Prada, João Dias, Pedro A. Santos, Kam Star, Ben Hirsh,
Ellis Spice, and Rob Kommeren “A Virtual Agent Toolkit for Serious Games Developers” in proceedings
of 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-7. IEEE, August 2018.

4. Common problems
-To maintain authorial control
-Adding features to provide agency
- Narrative graph exponential growth
- Keeping track of gameplay variables that affect the
story
-Measuring the impact on players’ experience
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

5. Goals
-Maintain the control of the authors to express
their intentions
-Provide a testing tool for authors, while writing
the story
-Present a set of metrics that assess the design
goals of the narrative
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

6. Context (iv4XR project)
-Provide a toolset for automation of playtesting
-Reduce the costs of production for faster
iteration in the development cycles
-Test operational properties (e.g. navigation bugs)
-Predict User eXperience (e.g. different user
profiles, emotional engagement)
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

7. A Narrative Model (following Twine)
-Passages
-[Guarded] Options -> lead to a passage
-Narrative graph
- Nodes
- Conditional edges
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716
A storyboard in Twine 2.0
https://twinery.org

8. A model for Testing Narratives
- Possible paths (number of paths)
- Analysis of visited passages (frequency and distribution)
- Endings hit percentage
- Stroke points (visited by all) and passages most/less visited
- Lost plot (never visited) and dead-ends (not a planned ending)
- Analysis of variables
- Testing conditions (highlight occurrences)
- Timeline (inspect the variables’ changes over time in a path)
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

9. Story Validator*
- Exhaustive Depth-First
Search
- Visual tool
- Annotated full graph
- Individual paths
inspection
- Text scripts and charts
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716
*https://github.com/iv4xr-project/in-story-validator

10. Testing procedure
-Create a story in Twine
-Load the story in the Story Validator tool
-Check metrics and properties
-Test hypothesis
-Change the story in Twine and reload
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

11. Emulating personas
- A persona is defined by a set of variables that it tries
to maximize
- E.g. the happiness of the character
- Greedy search
- Local choice of the option with highest value at each step
of the story
- A story path is generated for the persona
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

12. 2 User studies
-Usability (5 participants)
- 83.5 / 100 (SD = 9.45) - System Usability Scale (SUS)
-Efficacy (20 participants)
- Story with 5 problems
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

13. Efficacy Study (20 participants)
- 5 participants with experience in Twine
- Most participants identified the problems (taking 52.3 s ; SD
= 22.66)
- And could propose a solution (taking 146.2 s ; SD = 31.84)
- Still some problems finding unvisited passages and dead-
ends in specific paths
- The final version had improved usability (SUS 92.4 ; SD =
4.76)
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

14. Conclusions
- There is a need for testing tools for interactive narrative
games
- Promising field for Game AI
- We propose some first steps (i.e. visual tool and metrics)
- Future work
- Dealing with non-deterministic effects of the choices (e.g.
the variables are computed by a model - AI character)
- Integrate the validator in authoring tools
https://iv4xr-project.eu/
Project Ref. EU H2020-ICT-2018-3 - 856716

15. Validating the plot of Interactive
Narrative games
https://iv4xr-project.eu/
@iv4xr
Rui Prada (INESC-ID): rui.prada@gaips.inesc-id.pt
Story Validator: https://github.com/iv4xr-project/in-story-validator
Project Ref. EU H2020-ICT-2018-3 - 856716