Presentation at the SSA 2022: The 46th Annual Conference of the Semiotic Society of America. Abstract: The detrimental effects caused by uncontrolled technology usage and screen time have motivated designers in academia and industry to explore solutions that promote digital well-being. This paper draws on the social semiotic approach to multimodality to examine the semiotic resources applied in designing and presenting one case study concerning such solutions—Little Signals, six artifacts commissioned by Google. An analysis was performed on the project’s website’s content, paying careful attention to an introductory video and artifact gallery. Proximity, distance, focus, and analogy appear as distinctive video storytelling choices. These convey unobtrusiveness, invisibility, ephemerality, intimacy, control, and familiarity. The resources of size, shape, material, color, and motion applied to define the artifacts’ appearance, behavior, and data presentation also help reinforce it. Besides examining the relationship between these meaning potentials, resources, and digital well-being artifacts, this paper also discusses the apparent attempt to give smart-home devices a benign character.

1. Omar Sosa-Tzec
San Francisco State University,School of Design
DigitalWellbeingTechnologythrougha
SocialSemioticMultimodalLens
A Case Study
The 46th Annual Conference of the Semiotic Society of America
October 15,2022–Virtual

2. Photo Credit: Nick Beardslee
PersonalIntroduction
• Designer
• Background in Human-Computer
Interaction Design,Information Design,
and Computer Science
• Assistant Professor of Visual
Communication Design

3. Background

4. Photo Credit: Keira Burton
SmartphoneasaCentralArtifactinModernSociety
• Global communication and connectivity
• Flexible work practices
• Mobile entertainment
• Ubiquitous access to services and products
Cecchinato et al.(2019)
Vanden Abeele and Nguyen (2022)
Büchi (2021)

5. Photo Credit: CottonBro
SmartphoneOveruseandWellbeing
• Loneliness
• Anxiety
• Depression
• Concentration issues
• Sleep deprivation
• Physical pain
• Lack of boundaries between personal and
professional life
Bernroider, Krumay,and Marigold (2014)

6. Photo Credit: Monstera
DigitalWellbeing
• Emerging concept in humanities and social sciences that has now become
a research domain addressed by scholars of multiple disciplines
• Happiness (hedonia and eudaimonia) in social environments where
digital media is omnipresent
• A balanced relationship with technology:
• Controlled pleasure
• Maximizing the support to achieve goals
• Minimizing loss of control and functional impairment
Vanden Abeele (2021)
Vanden Abeele and Nguyen (2022)
Büchi (2021)

7. Photo Credit: Monstera
DesignforDigitalWellbeing
• “Interest in understanding how to support more meaningful interactions
with technology that allow users to be in control and better self-manage
their use to limit any potential negative e
f
fects and achieve their goals”
• Designers in academia and industry create and evaluate digital tools or
products that promote digital wellbeing
• Design challenges introduced by the variety of occupations,personal
preferences,and social contexts of use
Cecchinato et al.(2019)

8. Howtodesigne
f
fectiveanddelightful
digitalwellbeingtoolsandproducts?
central question for digital wellbeing design practice

9. WhytoExploretheApplicationof
SocialSemioticsandMultimodality?

10. Photo Credit: Anete Lusina
Design,Semiotics,Rhetoric,andMultimodality
• Design as a practice focused on communication connects directly
with semiotics
• Designers expect that signs will a
f
fect behavior,belief,or attitude in
the users (viewers)
• Design products are rei
fi
ed arguments by which designers attempt
to prescribe how users should live the everyday life
• Argument by experience: multimodal argumentation
• A bridge between design and social semiotics and multimodality
appears (for designers) from identifying design products in context
as a source of meaning and experience
Ehses (2008)
Buchanan (2001)
Groarke (2015)

11. Photo Credit: Anete Lusina
Design,Semiotics,Rhetoric,andMultimodality
•Design as a practice focused on communication connects
directly with semiotics
•Designers expect that signs will a
f
fect behavior,belief,or
attitude in the users (viewers)
•Design products are rei
fi
ed arguments by which designers
attempt to prescribe how users should live the everyday life
•Argument by experience: multimodal argumentation
•A bridge between design and social semiotics and
multimodality appears (for designers) from identifying design
products in context as a source of design and experience
Ehses (2008)
Buchanan (2001)
Groarke (2015)
Nelson & Stolterman (2012)
Kress (2010)
Norman (2013)
De Souza (2005)
Fogg (2003)
central question in design practice
Whatarethedecisionsthatwillproducean
e
f
fectiveanddelightfuldesign?
Choice
Resource
Semiosis
A
f
fect
Persuasion
Intent
A
f
fordance
Interfacesignembodiment
Multimodality
Interactionand UX

12. Photo Credit: Anete Lusina
Design,Semiotics,Rhetoric,andMultimodality
•Design as a practice focused on communication connects
directly with semiotics
•Designers expect that signs will a
f
fect behavior,belief,or
attitude in the users (viewers)
•Design products are rei
fi
ed arguments by which designers
attempt to prescribe how users should live the everyday life
•Argument by experience: multimodal argumentation
•A bridge between design and social semiotics and
multimodality appears (for designers) from identifying design
products in context as a source of design and experience
Ehses (2008)
Buchanan (2001)
Groarke (2015)
embodied/embedded knowledge that
can help promote best practices
central question in design practice
Whatarethedecisionsthatwillproducean
e
f
fectiveanddelightfuldesign?
Choice
Resource
Semiosis
A
f
fect
Persuasion
Intent
A
f
fordance
Interfacesignembodiment
Multimodality
Interactionand UX
Nelson & Stolterman (2012)
Kress (2010)
Norman (2013)
De Souza (2005)
Fogg (2003)

13. Doesasocialsemioticmultimodalapproachhelpcaptureand
articulatedesignknowledgetopromotebestpracticesfor
digitalwellbeingdesign?
methodological question

14. CaseStudy

15. Photo Credit: little signals.withgoogle.com
GoogleLittleSignals
• Part of the Digital Wellbeing Experiments showcased on Experiments
with Google website
• Little Signals is a 6-object collection that explores“how to stay up-to-date
with digital information while maintaining moments of calm”
• They represent an application of the Calm Technology framework to
digital wellbeing design
Little Signals (2022)
Weiser and Brown (1997)

16. Approach
social semiotics multimodality
•Inspection of semiotic resources:
a
f
fordances and meaning potentials
•Analysis of design: organization and
dominance of semiotic resources
•Analysis of intent: designer
’
s choice
and context of use
Semiotic Resources: material resources and
immaterial conceptual resources
Mode: socially organized set of semiotic resources for
making meaning
Jewitt,Bezemer,and O
’
Halloran (2016)
Kress (2010)
Kress and Van Leeuwen (2020)
data
•Separate Website: emphasis on
introductory video and object gallery
•Webpage on Digital Wellbeing
Experiments (Experiments with Google)
The e
f
fective communication of this project is possible
due to all the content and its integration—text,image,
sound,and interaction—hence it became necessary to
consider all the content for analysis not just the
artifacts—especially,given the lack of
fi
rst-hand user
experience with the artifacts
observations

17. Observations

18. StatedResourcesAppliedtoLittleSignals’Design
Photo Credit: little signals.withgoogle.com
Google Little Signals
Blow
Wiggle
Spin
Grow
Chime
Bounce
Push
Twist
Move
Graph
Bounce
Flip
Breath
Shake
Stretch
Rotate
Knock
Point
Wave

19. IntroductoryVideo
’
sChoicesandMeaningPotentials
storytelling
choices
•Depth
•Proximity
•Focus
•Analogy
meaning
potentials
•Unobtrusiveness
•Invisibility
•Ephemerality
•Intimacy
•Control
•Familiarity
•Signi
fi
cance
Source:
little
signals.withgoogle.com

20. Identi
fi
edResourcesforDigitalWellbeingDesign
Detachable
Indivisible
Modular
Unibody
Exact Prism
Approximate Prism
Geometric
Organic
Solid
Hollow
Uniform
Long
Compact
Slim
Form
Body
Shape
Proportion
Color
Hue
Saturation
Value
Material
Tactility
Opaqueness
Rough
Smooth
Opaque
Translucent
Occurrence
Direction
Duration
Speed
Motion
Periodic
Episodic Unidirectional
Multidirectional
Linear
Rotatory
Brief
Long
Quick
Slow
Calm Technology Object for
Digital Wellbeing
Interactivity
Action
Engagement
Sound
Gentle
Absent
Tap
Push
Flip
Observe
Attend
Ignore

21. Discussion

22. Photo Credit: little signals.withgoogle.com
OnSocialSemioticsandMultimodalityasaCriticalFrameworkforDesigners
• Social semiotics and multimodality provide (interface/experience/HCI)
designers with a framework to think of digital wellbeing technology as
multimodal artifact
• The focus becomes the chosen resources,their integration,the
meaning potentials that they could unfold during the user
experience,and the signi
fi
cance of such meanings
• This framework highlights the relation between creation,
contextualized meaning,user experience,and agency from both
designer and user
• Designers would
fi
nd transductionas a key concept to understand
the evolution and interrelation between design of di
f
ferent kinds

23. Photo Credit: little signals.withgoogle.com
DrawbacksConcerningtheApplicationofSSM
• A major drawback in the application of this framework is the circularity
or unclear boundaries concerning the notions of resource and modes
• It seems seems necessary to set the delimit of the notions and
theory beforehand
• Experiential knowledge derived from using the design being inspected
becomes a crucial factor to respect the contextual dimension of
meaning-making—in this case,the meaning produced during a real
user experience

24. Photo Credit: little signals.withgoogle.com
OnTechnologytoFixTechnologyandAttainingDigitalWellbeing
• The project
’
s information online—especially the introductory video—
works as a multimodal argument that presents a vision where (more)
technology is the solution for the uncontrolled use of technology
• Technology that still seeks to delight—just like any other kind of
digital design—in order to work as a product
• Technology that portrays technologizing home as a benign and
desirable enterprise
• Technology that suggests some technological,economic,and
social requirements to live a happy life

25. Photo Credit: little signals.withgoogle.com
NotableAspectsofGoogleLittleSignals
• The project explores changes from the center and periphery as one
way to engage with digital services and products o
f
f the screen
• It is an e
f
fective demonstration of how the physicalization of data and
material explorations can be applied in design for the everyday life
• It promotes the idea of home as the space to nurture oneself and
disconnect,
• It also emphasizes being o
f
f the screen as a fundamental factor

26. Thankyou!
tzec.com
https://design.sfsu.edu/