My talk at #frAIday talks at Umea University - June 4, 2021. I talked about some principles for building social interactive agents and presented some examples of such agents.

1. Building competent social
interactive AI
Rui Prada
Instituto Superior Técnico, Universidade de Lisboa
INESC-ID
#frAIday talks at Umea University - June 4, 2021

2. Social Competent AI Agents
• Autonomous agents that act according to social context
• Are able to take social context into account in
• Decision making
• Interpretation of the world
• Socially aware agents
• Accommodate their needs to social context
• E.g. People eat and sleep at appropriate times and places

3. Interactive AI agents
• Have the users’ actions/input into account
• Respond appropriately to the user
• One of the agents in the social context is the user

4. Social Context
• Who: roles and relations
• What: activity and tasks
• Where: places and locations
• When: time and events
Combined with
• Volition: goals, needs

5. Groups and Culture
• Interpretation and adaptation to social context differs across
social groups
• What is relevant in the context and what is acceptable behaviour
• Agents belong to social groups
• Social groups entail culture

6. Social Importance Dynamics
• Social Importance (SI) as a key concept in social interactions
• Precondition for acceptable behaviour
• Claim and confer dynamics
(Kemper Status-power Interaction Dynamics)
• Difference groups/cultures give social importance to
different things
(Hofstede Model of Culture)
Samuel Mascarenhas, João Dias, Rui Prada, Ana Paiva “A Dimensional Model for Cultural Behaviour in Virtual Agents” in Applied Artificial Intelligence, vol. 24 (6), pp. 552-574,
July 2010. Taylor & Francis.
Samuel Mascarenhas, Nick Degens, Ana Paiva, Rui Prada, Gert Jan Hofstede, Adrie Beulens, Ruth Aylett: “Modeling culture in intelligent virtual agents: From theory to
implementation” in Autonomous Agents and Multi-Agent Systems. pp. 1-32, 2015. Springer.

7. Social Importance Dynamics
• Actions invoke SI claims
• Agents attribute SI to others
• An action performed by agent a
is accepted by agent b if b
confers enough SI to a to
support the action claim
• The rules for SI claim and
conferral differ across groups
• E.g.: the SI given to strangers is
different across cultures

8. Social Power
• Social power mediates social interactions
• Influence towards change
• Influence is a function of the power exerted and the
resistance to change
• Social power as dynamics of social importance
• Power = SI conferred to the actor
• Resistance = SI claim of the change
Gonçalo Pereira, Rui Prada, Pedro A. Santos: “Integrating social power into the decision-making of cognitive agents” in Artificial Intelligence. vol. 241, pp. 1-44, December 2016.
Elsevier.

9. Social Power
• Social power has different sources
(French and Raven)
• Reward
• Coercive
• Legitimate
• Expert
• Referent
• Power strategies highlight the
sources in the context

10. Places as Social Context
Diogo Rato, Marta Couto, Samuel Mascarenhas, Rui Prada “Can an agent be social when alone? An experimental study on adaptive behavior” submitted to IVA 2021

11. Places as Social Context
• 136 participants from Mechanical
Turk
• Two agent policies for change
• Random
• Context-based
• Places with or without explicit
marks
• Social Motivation of Intelligent
Agents Scale

12. Socially Situated Cognition
• Social meaning of objects
• E.g. An apple can be food, a gift, a toy, a weapon, …
• Social categorization and social identity
• The agents and their social groups in a given context
• Social affordances
• What you can do with the agents and objects in the context
• Socially affordable
• What is acceptable
Diogo Rato, Samuel Mascarenhas, Rui Prada “Towards Social Identity in Socio-Cognitive Agents” arXiv preprint arXiv:2001.07142 (2020).

13. Cognitive Social Frames

14. Cognitive Social Frames

15. Cognitive Social Frames

16. Cognitive Social Frames

17. Cognitive Social Frames
Resources generate the agent behaviours

18. Social AI agents in Minecraft
• Context
• Time, location, agents
• Social practices
• Activated by context
• Social roles
• Locations have social properties
• Expected activity
• Ownership
• Agents have categories/identities
• Define relevant social practices
• E.g. Lumberjack

19. Socially Aware Conversational Agents
• Conversation as social knowledge
• Simple conversation beats as
social practices
• Context filters acceptable
practices
• Who (social roles, relations)
• State of conversation
• Goals
• Practices define sentences
available to use

20. Socially Aware Conversational Agents
• Doctor appointment
• Interview
• Goal to gather information
• No big dialogue trees that restrict
the conversations
• The user can be any actor
• Easy to author simple (reusable)
practices

21. HRI Group-based Emotions
Displaying emotions as individual or group
Determine the cognitive unit for the emotional appraisal
Positive effects for group identification, trust and likability
Filipa Correia, Samuel Mascarenhas, Rui Prada, Francisco S. Melo, Ana Paiva: “Group-based emotions in teams of humans and robots” in proceedings of HRI'18 - International
Conference on Human-Robot Interaction, pp. 261-269, Chicago, IL, USA, March 2018. ACM/IEEE.

22. Social Power for Persuasive Robots
Mojgan Hashemian; Ana Paiva; Samuel Mascarenhas; Pedro A. Santos; Rui Prada: “The power to persuade: a study of social power in human-robot interaction” in proceedings of RO-
MAN’19 - the 28th IEEE International Conference on Robot and Human Interactive Communication, pp. 1-8, New Delhi, India, October 2019. IEEE.

23. Social Robots as Team Leaders
• Leadership types
• Transactional (TA): focus on task
• Transformational (TF): focus on people
• 108 people (Portuguese companies) 36
teams of 3
• Productivity: higher for TA
• Engagement: higher for TF
• Role Ambiguity: no sig. difference
• Trust: no sig. difference
Sara L. Lopes, José Bernardo Rocha, Aristides I. Ferreira, Rui Prada “Social robots as leaders: leadership styles in human-robot teams” submitted to RO-MAN 2021

24. Agents to Test UX
• Software testing with autonomous (testing) agents
• Automate User eXperience testing
• Social and Emotional agents
• Predict UX by running the agents in the environment
• Predict emotion from interaction traces
Pedro M. Fernandes, Manuel Lopes Rui Prada “Agents for Automated User Experience Testing” in proceedings of the AIST’2021 – the 1st International Workshop on Artificial
Intelligence in Software Testing, International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp.. 247-253, IEEE. 2021
Rui Prada, ISWB Prasetya, Fitsum Kifetew, Frank Dignum, Tanja EJ Vos, Jason Lander, Jean-yves Donnart, Alexandre Kazmierowski, Joseph Davidson, Pedro M Fernandes “Agent-
based Testing of Extended Reality Systems” in proceedings of ICST’- 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 414-417, IEEE. 2020.
https://iv4xr-project.eu

25. Predict Emotion from Interaction Traces
• Machine Learning approach
• Users play a game
• Self-report annotation: PAD model
• 88 participants, 3 maps
• 264 traces
• Predict 3 classes: increase, decrease, stable
https://iv4xr-project.eu

26. Predict Emotion from Interaction Traces
Arousal
Accuracy: 86.69 %
Average Confusion Matrix:
Pleasure
Accuracy: 90.95 %
Average Confusion Matrix:
Dominance
Accuracy: 82.07 %
Average Confusion Matrix:
48.39 0.46 0.10
8.32 119.33 10.71
5.81 9.21 110.67
70.84 0.59 0.45
7.35 147.54 8.58
2.56 4.04 102.65
47.49 1. 0.56
12.71 153.24 13.69
6.2 5.42 89.96
https://iv4xr-project.eu

27. Building competent social interactive AI
Rui Prada
rui.prada@tecnico.ulisboa.pt
#frAIday talks at Umea University - June 4, 2021
https://iv4xr-project.eu
https://www.humane-ai-net.eu