Vanessa Echeverria, Roberto Martinez-Maldonado, and Simon Buck- ingham Shum.. 2019. Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. In Proceedings of ACM CHI conference (CHI’19). ACM, New York, NY, USA, Paper 39, 16 pages. https://doi.org/10.1145/3290605.3300269 Collocated, face-to-face teamwork remains a pervasive mode of working, which is hard to replicate online. Team members’ embodied, multimodal interaction with each other and artefacts has been studied by researchers, but due to its complexity, has remained opaque to automated analysis. However, the ready availability of sensors makes it increasingly affordable to instrument work spaces to study teamwork and groupwork. The possibility of visualising key aspects of a collaboration has huge potential for both academic and professional learning, but a frontline challenge is the enrichment of quantitative data streams with the qualitative insights needed to make sense of them. In response, we introduce the concept of collaboration translucence, an approach to make visible selected features of group activity. This is grounded both theoretically (in the physical, epistemic, social and affective dimensions of group activity), and contextually (using domain-specific concepts). We illustrate the approach from the automated analysis of healthcare simulations to train nurses, generating four visual proxies that fuse multimodal data into higher order patterns.

1. Towards Collaboration Translucence:
Giving Meaning to Multimodal Group Data
Vanessa Echeverria, Roberto Martinez-Maldonado & Simon Buckingham Shum
Connected Intelligence Centre, University of Technology Sydney, AUS
@vanechev @RobertoResearch @sbuckshum
ACM CHI 2019, Glasgow, Scotland

2. OUR CONTEXT: TRAINING NURSES IN HIGH
PERFORMANCE TEAMWORK
Multiple student simulation teams
in action at once, with 1 instructor
Simulations can be cognitively,
socially and emotionally intense
(that’s the point)
There’s too much going on to see
it all, or remember it all
Scope for computational support to
augment debriefings?

3. STAKEHOLDER ENGAGEMENT
Co-design methods were used
to gain insights from both
students and educators about
their experiences running, and
performing, simulation
exercises
What were educators’ fantasy
“superpowers”?
(e.g. omnisience)
At what points would students
value better feedback?
Prieto-Alvarez, C. et al. (2018). Co-designing learning analytics tools with learners. In: Learning Analytics
in the Classroom: Translating Learning Analytics Research for Teachers. Routledge

4. INSPIRATION: “SOCIAL TRANSLUCENCE”
Erickson et al. (CHI’99) tackled the challenge of providing missing
social cues for online discourse platforms
Erickson, T., Smith, D. N., Kellogg, W. A., Laff, M., Richards, J. T., & Bradner, E. (1999). Socially translucent systems: social proxies, persistent
conversation, and the design of “babble”. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 72-79). ACM.
Translucence ≠ Transparency
In f-f social spaces/places, we use
translucence to disclose specific
information at an appropriate fidelity,
e.g. frosted glass doors / walls
H. Schnädelbach and D. Kirk, People, Personal Data and the Built Environment
(Springer Series in Adaptive Environments). Springer International, 2018.

5. GROUP CHAT PROXY
A visualization indicating at a glance an important aspect of the history,
or current state, of an online social space
Erickson, Thomas. (2009). 'Social' systems: Designing digital systems that support social intelligence. AI & Society, 23. 147-166. 10.1007/s00146-007-0140-3.
Figure 2. A social proxy for a group chat in the Babble system:
(a) an active chat (b) after chat has ceased.

6. ONLINE LECTURE PROXY
A visualization indicating at a glance an important aspect of the history,
or current state, of an online social space
Erickson, Thomas. (2009). 'Social' systems: Designing digital systems that support social intelligence. AI & Society, 23. 147-166. 10.1007/s00146-007-0140-3.
Figure 4. Three instances of the lecture proxy
(a) the norm (b) an audience member
interrupting
(c) many audience
members speaking,
which violates the norm

7. COULD VISUAL PROXIES FOR COLLOCATED
INTERACTION ASSIST WITH DEBRIEFINGS?
This creates
Analytics Challenges
• Data Capture
• Data Fusion
• Activity Semantics
HCI Challenges
• Visualisation
• Interactivity

8. THE ANALYTICS CHALLENGE:
SENSE AND CAPTURE ACTIVITY

9. THE ANALYTICS CHALLENGE:
FROM DATA FUSION TO MEANINGFUL ACTIVITY
Once activity data sources have been gathered, and integrated
(data fusion), the primary challenge is to assign meaning
What counts as a significant event in human activity? This requires
contextual, qualitative insight (cf. Quantitative Ethnography: Shaffer 2017)
We need a systematic way to model the structure and combination
of data streams based on qualitative insights from educators, and
the research literature. 2 examples…
David Williamson Shaffer (2017). Quantitative Ethnography. Cathcart Press

10. WHAT MAKES A NURSE’S POSITION SIGNIFICANT?
Clinical expertise informed the modelling of 5 meaningful zones for positional data
i) the patient for cases where
nurses are located on top of or
very close to the patient
ii) next to patient for cases where
nurses are either side of bed
iii) around the patient for cases
where nurses are 1.5 to 3
metres away
iv) bed head where nurses
commonly stand to clear the
airway during CPR
v) trolley area where nurses
access medication or equipment

11. ADDING A RESEARCH-BASED MODEL OF COLLABORATION
ACAD: Activity-Centred Analysis & Design framework
PHYSICAL SET — physical and digital space and objects; input devices,
screens, software, material tools, furniture
EPISTEMIC TASKS — implicit and explicit knowledge oriented elements that
shape the participants’ tasks and working methods
SOCIAL SITUATION — the variety of ways in which people might be grouped
together (e.g. dyads, trios); scripted or emerging roles; and divisions of labour
AFFECTIVE RESPONSES — an extension to ACAD, building on evidence
from healthcare simulation research
R. Martinez-Maldonado, P. Goodyear, J. Kay, K. Thompson and L. Carvalho. 2016. An Actionable Approach to Understand Group Experience in Complex, Multi-surface
Spaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI'16, 2062-2074. DOI: https://doi.org/10.1145/2858036.2858213

12. THE ANALYTICS CHALLENGE:
Making multimodal streams meaningful
From multimodal logs to higher-order constructs:
Embodied strategies
Actions and procedures
Communication with patient
Changes in emotional arousal
# and length of utterances by the patient
# and length of utterances by nurses
Presence in meaningful zones
Wrist acceleration intensity
Actions registered by the manikin
Electrodermal activity peaks
Critical procedures
Distance to the patient and the trolley
Interactions with objects
Teamwork communication
Proximity to patient/objects
Intensity of physical activity
Physical
Social
Epistemic
Affective
Dimensions of
collaboration
Multimodal
observations
Higher-order
constructs
Patient-centred
care
&
Teamwork
1
2
Constructs for collaborative activity
(from ACAD Framework)
Curriculum
outcomes
Multimodal data sources

13. THE ANALYTICS CHALLENGE:
Making multimodal streams meaningful
From multimodal logs to higher-order constructs:
Embodied strategies
Actions and procedures
Communication with patient
Changes in emotional arousal
# and length of utterances by the patient
# and length of utterances by nurses
Presence in meaningful zones
Wrist acceleration intensity
Actions registered by the manikin
Electrodermal activity peaks
Critical procedures
Distance to the patient and the trolley
Interactions with objects
Teamwork communication
Proximity to patient/objects
Intensity of physical activity
Physical
Social
Epistemic
Affective
Dimensions of
collaboration
Multimodal
observations
Higher-order
constructs
Patient-centred
care
&
Teamwork
1
2
Constructs for collaborative activity
(from ACAD Framework)
Curriculum
outcomes
Multimodal data sources

14. THE ANALYTICS CHALLENGE:
Making multimodal streams meaningful
From multimodal logs to higher-order constructs:
Embodied strategies
Actions and procedures
Communication with patient
Changes in emotional arousal
# and length of utterances by the patient
# and length of utterances by nurses
Presence in meaningful zones
Wrist acceleration intensity
Actions registered by the manikin
Electrodermal activity peaks
Critical procedures
Distance to the patient and the trolley
Interactions with objects
Teamwork communication
Proximity to patient/objects
Intensity of physical activity
Physical
Social
Epistemic
Affective
Dimensions of
collaboration
Multimodal
observations
Higher-order
constructs
Patient-centred
care
&
Teamwork
1
2
Constructs for collaborative activity
(from ACAD Framework)
Curriculum
outcomes
Multimodal data sources

15. THE ANALYTICS CHALLENGE:
Making multimodal streams meaningful
From multimodal logs to higher-order constructs:
Embodied strategies
Actions and procedures
Communication with patient
Changes in emotional arousal
# and length of utterances by the patient
# and length of utterances by nurses
Presence in meaningful zones
Wrist acceleration intensity
Actions registered by the manikin
Electrodermal activity peaks
Critical procedures
Distance to the patient and the trolley
Interactions with objects
Teamwork communication
Proximity to patient/objects
Intensity of physical activity
Physical
Social
Epistemic
Affective
Dimensions of
collaboration
Multimodal
observations
Higher-order
constructs
Patient-centred
care
&
Teamwork
1
2
Constructs for collaborative activity
(from ACAD Framework)
Curriculum
outcomes Multimodal data sources

16. THE MULTIMODAL MATRIX:
Combining data sources to operationalise constructs

17. THE MULTIMODAL MATRIX:
Combining data sources to operationalise constructs

18. THE MULTIMODAL MATRIX:
Combining data sources to operationalise constructs

19. THE MULTIMODAL MATRIX:
Combining data sources to operationalise constructs

20. THE MULTIMODAL MATRIX:
Combining data sources to operationalise constructs
Modelling decisions: how to map data type(s) to constructs Segments can be added by machines or humans

21. THE HCI/FEEDBACK CHALLENGE:
Making activity visible through proxies
Critical actions
performed by
nurses
Affective/cognitive
arousal via EDA
peaks
Patient-centred verbal
communication, and
within nursing team
Patient-centred
movement around
the simulation zones

22. COLLABORATION TRANSLUCENCE:
Proxy for Patient-Centred Verbal Communication
Patient loses consciousness
Patient asks for help
Team A
Leader
RN 4
RN 2
RN 3
Leader
Phase1
Patient
RN 3
Edge thickness =
frequency of
interaction
RN =
Registered Nurse
Node size =
frequency of
speaking

23. COLLABORATION TRANSLUCENCE:
Proxy for Patient-Centred Verbal Communication
Patient loses consciousness
Patient asks for help
Team A
Leader
Patient recovers
RN 4
RN 2
RN 3
Leader
RN 4
RN 2
Team C
Leader
RN 4
RN 3
Leader
RN 4
RN 3
Patient
Patient
RN 3
Phase1Phase2
Patient
Team B
Leader
RN 4
RN 2
Leader
RN 4
RN 2
Patient
Patient
RN 3
RN 3
Patient

24. COLLABORATION TRANSLUCENCE:
Proxy for Patient-Centred Verbal Communication
Patient loses consciousness
Patient asks for help
Team A
Leader
Patient recovers
RN 4
RN 2
RN 3
Leader
RN 4
RN 2
Team C
Leader
RN 4
RN 3
Leader
RN 4
RN 3
Patient
Patient
RN 3
Phase1Phase2
Patient
Team B
Leader
RN 4
RN 2
Leader
RN 4
RN 2
Patient
Patient
RN 3
RN 3
Patient

25. COLLABORATION TRANSLUCENCE:
Patient-centred movement
Three different positions from which nurses performed chest compressions:
By the bed (A) Over the patient (B) Over the bed (C)

26. COLLABORATION TRANSLUCENCE:
Proxy for Patient-Centred Movement
Each circle represents one zone of interest around the patient’s bed,
size reflecting relative occupation time, links showing the transitions
trolleyolley
Team C
trolley trolley

27. COLLABORATION
TRANSLUCENCE:
Position and Movement
To assist reflection, the
proxies can be compared
and contrasted between
phases, or within phases
across teams, optionally
coupled with video

28. COLLABORATION TRANSLUCENCE:
Patient-centred team coordination of tasks

29. COLLABORATION TRANSLUCENCE:
Student responses were very positive
(detailed evaluation being written up)
“…while RN4 and RN2 were doing
the fluids I was staying with the
patient. It is good to step back and
look at what each person was
doing, one thing at the same time, I
think it shows you how you worked
as a team”
“it seems like a lot was done in
clumps, you [RN3] were talking to
the patient, looking for
information while others were
doing the observations, that
seems practical to me”

30. COLLABORATION TRANSLUCENCE:
Patient-centred team coordination
enhanced version using “data storytelling” principles (Echeverria et al. 2018)
Echeverria, V., R. Martinez-Maldonado, R. Granda, K. Chiluiza, C. Conati and S. Buckingham Shum (2018). Driving Data Storytelling from Learning Design. In
Proceedings of the International Conference on Learning Analytics & Knowledge, LAK'18, 131-140. ACM. https://doi.org/10.1145/3170358.3170380

31. COLLABORATION TRANSLUCENCE:
Patient-centred team coordination (enhanced version)

32. SUMMARY: A METHODOLOGY TO GENERATE
COLLOCATED COLLABORATION ANALYTICS
Method to inform the modelling of
quantitative activity data with qualitative
insights into what makes it meaningful
From teamwork as…
Ephemeral activity
(no evidence to inform
debriefs)
Opaque to computational
analysis
Sensors generate persistent
traces
Semi-automated data fusion and
analytics
“Collaboration Translucence”
via visual proxies
Positive feedback from students
and instructors

33. POTENTIAL FUTURE TRAJECTORIES…
Towards fully
automated
feedback
Future learning
spaces will be
configured
TECHNICAL
INFRASTRUCTURE
Further testing
with Health
Expansion to
other disciplines
Privacy/ethics
EMPIRICAL
STUDIES
The use of personal
replays to review
and reflect
Fictional
dashboards for
teaching
AFFORDANCES
FOR LEARNING