Games are crafted to provide unique experiences, but players don't always behave as you would expect. In this presentation, Alexandra Turcan and Ruan Pearce-Authers from Dambuster Studios will explain how they combine UX methods with telemetry and biometrics to quantify player in-game behaviour.

1. The Science of Fun
Data-driven Game Development
Dr. Alexandra Țurcan, Deep Silver GamesLab Manager
Ruan Pearce-Authers, Senior Online Services Engineer

2. About us

3. Psychologist
Researcher
Scientist
Data
scientist in
the games
industry
UX
Researcher
in Deep
Silver
GamesLab
Alex

4. Ruan
Community
management
(Crytek)
Server
programmer
(Warface)
DevOps
(Homefront:
The Revolution)
Senior Online
Services
Engineer
(Deep Silver)

5. Dambuster Studios
We make video games.
We are one of Deep
Silver’s development
studios, along with
Volition (USA) and
Fishlabs (Germany).
You can find us on Canal
Street, near
Broadmarsh.

6. Homefront: The Revolution
- is an open-world first person shooter where you must lead the Resistance movement in guerrilla
warfare against a superior military force.
Homefront "Freedom fighters" trailer

7. Data-driven
game
development

8. What is data-driven game development?
• Including data in the game development process
• This data can be generated from playtests with both internal and
external players
• It can be used to make more informed decisions during game
development
• At Dambuster, we collect data in our research lab, Deep Silver
GamesLab

9. GamesLab: research and playtesting facility

10. • Fun is the ultimate goal of the game
development process
• “Fun” is a subjective experience, and for
each player it could come from
something else:
• easy game,
• hard game,
• collaborative,
• individual,
• etc.
What is “fun”?

11. • Observing player behaviour
• Asking for player opinion and feedback
UX methods
• Used to objectively quantify player in-game behaviour
• Take the load off observational methods
Telemetry
• Used to quantify unconscious player behaviour
• Eye tracking, skin conductance
Biometrics
Million dollar question: How do you measure fun?

12. UX

13. UX Research
• Traditionally, it’s the process of understanding the impact of
design (e.g. websites) on an audience
• Example methods:
• Interviews
• Surveys
• Observations
• Focus groups
• Heuristic evaluation / expert review
• Adapted to video games, but it doesn’t scale well and it’s not
easy to automate
• Pro: can answer the “why” questions

14. Telemetry
pipeline

15. What is telemetry?
• Data collected from running games
• Used for both development and retail builds
• Gives us quantitative behavioural data
• Example telemetry:
• Discrete events – player started a level, killed an enemy, died, etc
• Continuous data streams – player position, health, etc

16. Sending telemetry
• Game code is instrumented to send events appropriately
• As events are emitted, we send them to a remote server
• Events are sent in the exact order they’re emitted
• This is very useful later on!
• Think of this as a stream of data
• The first event in a telemetry stream is purely metadata
• Platform, user ID, etc

17. Telemetry event examples
{
"type": "telem_start",
"t": 1484132366,
"v": 1,
"data": {
"Account": {
"User": "UKPC4020",
"Platform": "editor"
},
"Build": {
"Game": "OnlineSample",
"Type": "dev"
},
"Hardware": {
"OS": "windows"
}
}
}

18. Telemetry event examples
{
"type": "player_spawn",
"t": 1484132366,
"v": 1,
"data": {
"Name": "PlayerBlueprint_C_0",
"Pos": {
"X": 326.24,
"Y": 3.55,
"Z": 268.38
}
}
}

19. Telemetry event bus
• A lightweight HTTPS/WebSocket server receives all these events
• They’re forwarded into Kafka, a log database
• Supports many consumers, with non-destructive reads
• We retain data based on storage capacity and lifetimes
• Can replay events if we find processing bugs, or need to extract
new data

20. Telemetry stream processing
• We’re now all set to start making use of the data!
• Processors listen for one or more specific event types
• They have local memory to combine events
• Can use RAM directly, or a Redis instance
• Output summarised/combined events to:
• Separate queues for further processing
• Permanent data storage

21. Example telemetry stream

22. Example telemetry processor in JavaScript

23. Permanent telemetry storage
• We use MongoDB as a default
• Rich aggregation framework for queries
• Scales well from an ops perspective
• But there’s no one-size-fits-all solution to data storage
• Special cases include: time-series data
• These databases are then used by analysis scripts and
dashboards

24. R is a functional language and
environment for statistical
computing and graphics.
https://www.r-project.org/
Telemetry analysis

25. Telemetry architecture recap

26. Telemetry
use cases

27. Telemetry for designers
- Where do players die? -

28. Telemetry for designers
- Can the player see the enemy who’s shooting them? -
• Player can’t see the enemy:
• not_in_FoV
• rendered / not_rendered
• Enemy is occluded:
• in_FoV
• not_rendered
• Player could be seeing the
enemy:
• in_FoV
• rendered

29. Telemetry for designers
- Do players become more accurate shooters over time? -
lm(accuracy ~ time) => Model explains 0.004% of variance.
Time does not predict accuracy on its own. Other predictors?

30. • Player accuracy doesn’t improve over time.
• Reasons why that might be:
• Not enough practice?
• Too long delay between play sessions?
• Other variables might be affecting accuracy (e.g. ammo count:
when players have loads of ammo they can afford to not be as
accurate as when they are running out of ammo)?

31. Telemetry for detecting subtle issues

32. Telemetry for business intelligence
- Timeline of mission popularity -

33. Telemetry for online services
- What is the relationship between ping and distance? -
• Distance is only weakly
correlated with session
ping (correlation
coefficient = 0.28).
• With every 1 km increase
in distance, the average
ping only increases with
0.01 ms.
• The value of the ping
does not change with
distance.
• So, a good matchmaking
algorithm should not
depend solely on
geographical distance.

34. Biometrics
introduction

35. Why use biometrics at all?
• It nicely complements the quantitative behavioural data provided
by telemetry and the qualitative player opinions provided by UX
methods
• It’s the only tool we have that measures unconscious player
behaviour
• Can’t be faked – player or researcher subjectivity does not affect
it

36. Eye-tracking
• Traditionally used in academic research to
study human visual attention
• Adopted more and more in the games
industry because it can answer questions
about player behaviour that cannot be
answered through any other means
• It is non-invasive – eye-tracking hardware
snaps onto the PC monitor and tracks eye
movements from a distance - the player
will not be inconvenienced in any way

37. Skin conductance
Galvanic skin response (GSR)
aka
Skin conductance (SC)
aka
Electrodermal activity (EDA)
• Measures changes in skin electrical
conductivity
• Reliable indicator of emotional state
• Not under conscious control

38. Stimulus
Activate body’s
autonomic
nervous system
Sweat
Skin moisture
content
increases
Skin’s electrical
conductivity increases
• Even if a person is not visibly sweating, sweat is still building up
underneath the skin, which means that GSR can still be measured.
How GSR works

39. Valence
Arousal
Can be measured with facial
recognition.
Can be measured with skin
conductance.
Emotions

40. Biometrics
implementation

41. Eye-tracking technical specs
• Works by reflecting near-infrared light off the cornea
• The hardware gives us (x,y) screen coordinates
• This raw data is entirely useless without game world context
• Can’t be included in telemetry directly
• Let’s see what can actually be done with this!

42. Eye-tracking telemetry (3D)
• This tells us which objects in the world are being looked at
• Deproject those (x,y) coordinates into a world position and
direction
• Cast a sphere through the world, see what we hit
• Customisable radius for accuracy
• When the hit object changes, log a “started_looking” event to
telemetry with object details

43. Eye-tracking debug visualisation

44. Eye-tracking telemetry (2D)
• This tells us how the player uses the UI/HUD
• The game needs to provide a 2D map of all interesting elements
• e.g. Circle for a minimap, small rectangle for a health bar
• Perform intersection testing on these elements, log events when
this changes

45. GSR technical specs
• GSR data is recorded in microSiemens
• Our sensors operate at 10hz (we get one reading every 100ms)
• Roughly speaking: sweat means more current goes through
• The data is just a single floating-point value
• Suitable for direct inclusion in the telemetry stream
• Means we can check response to any in-game events

46. Pacing volumes
• GSR data can’t be used in absolute terms
• Designers add markup objects to the world, called pacing
volumes
• These define expectations for emotional intensity at given areas
in the game
• e.g. “calm”, “moderate”, “intense”
• We can then check how well our expectations match up with
reality

47. GSR analysis algorithm
• Raw GSR data needs to be analysed before it can be interpreted
• An algorithm in R outputs a series of significant peaks coupled
with the events that caused them
Ledalab algorithm in Matlab (http://www.ledalab.de/) by
Benedek & Kaernbach (2010). We translated it to R.

48. Biometrics
use cases

49. Example eye-tracking use case

50. Example GSR use case

51. Other GSR use cases
• Can be used to answer questions like:
• Are various areas of the game more or less engaging for the player?
• Does dying elicit a similar emotional response as wining the mission?
• Do players panic when they run out of ammo?
• Does removing certain game elements make the game less
engaging/more boring?

52. • Dark Souls 3
• Homefront: The Revolution
GSR overlaid on game-play:
GSR tracking Dark Souls 3

53. Tools recap

54. Get involved!
• Sign up for playtests
• https://gameslab.dsdambuster.com/
• Contact us @AleTurcan and @returnString