2. Outline
1. Multimodal theory
2. Types and features of virtual worlds
3. Communication possibilities in virtual worlds
4. Data collection and tools
5. Student and teacher strategies in a formal
context in Second Life
6. Communication features in a formal context in
Second Life
7. Concluding remarks
8. References
3.
4. Multimodal theory
• Communication occurs through different
synchronous modes (Kress & Van Leewen, 2001)
▫ Verbal (oral / text)
▫ Images
▫ Sound
▫ Movement
▫ …
• Modal density (Norris, 2004)
Multimodal theory
5.
6. Text and graphic based VW
Type of VW Affordances for language learning
MOOs - socialization and empowerment of learners
-persistent environment
-anonymity
- anxiety eased in SS with a reluctance to speak
(Warschauer, Turbee & Roberts, 1996)
Graphic based
virtual worlds
- greater sense of presence
-co-construction of knowledge through an interactive
virtual environment
(Campbell, 2003)
Graphic based
virtual worlds with
voice
-strengthens the feeling of digital representation of self
(embodiment)
-voice: key element for language learning
(Cassel et al., 2001; Gee, 2001; Sweeney et al., 2010)
Types and features of virtual worlds
7. Features of 3D virtual worlds
• Real-time online virtual space
• Sound is distance-attenuated
• Persistent environment
• Avatar
• Teleport
• Communication tools
Types and features of virtual worlds
12. Multimodal communication
Face-to-face communication MUVE communication
• Multi-channel
• Interaction/interactivity
with…
▫ Avatars
▫ Environment
▫ Objects
▫ Interface
• Verbal language
• Non-verbal elements:
gestures, proxemics, haptics
• Environment
Communication possibilities in virtual worlds
13. Modalities in Second Life
Audible mode Textual mode Visual mode Spatial mode
-Language,
prosody, pause
-Background
sound
- Local chat
- Private chat
- Notecards
-Appearance
-Gestures
- Layout of space
- Proxemics,
movements
Adapted from Örnberg, T. (2005)
Communication possibilities in virtual worlds
14.
15. Course description
• 2 groups of first-year undergraduate students of
Tourism
• 30 students (15/class)
• Course: English for tourism purposes (level 1)
• No previous experience using Second Life
• 3 modules (3 sessions each)
▫ Hotels
▫ Museums
▫ Virtual Tourism
Data collection
16. Data
• Screen recordings of the sessions (teacher’s
perspective)
• After each session:
▫ Student questionnaires
▫ Teacher/researcher journal
• End: Semi-structured personal interviews
Data collection
22. Examples: Use of chat (students)
SS T S - S
• (0:52:19) – Cco: cristina, I
don’t liset anithing (technical
problems)
• (1:02:03) – SM: cristina I
would like to do it the second
(about the task)
• (1:02:37) BR: GM, quina es
aquesta nota que mas
preguntat? Que tha donat la
cristina?
• (1:02:45) GM: eske a mi ma
enviat una ke em posa kuatre
cosas tipus… ki soc, labitacio k
vull i axo
(asking for
help/clarification)
Strategies in a formal learning context in VWs
23. Examples: Use of chat (teacher)
CHAT TRANSCRIPT FUNCTION
T: who’s vacuuming?
T: turn off your mics!
Classroom management, sorting out noise
disruptions (technical)
T: Jest, Maska, Carme, Esther,
Andrea, sofas
Keywords in instructions, classroom
management (forming groups)
IM to Ander: do you have any
special requests?
Private help during tasks
Strategies in a formal learning context in VWs
24. Example: Channel checks
Teacher prompted Student prompted
• T: OK, so now I'm going to
pair you up. For example, Jest
Dash... yes, Jest? Can you hear
me?
• T: Can you talk, E? or no? No.
And mmm andrea, yeah?
Andrea, can you talk?
• T: mhm, Daniela, did you
hear?
• GM: esta parlant algu?
• DM: hello?
• JT: you listen me?
Strategies in a formal learning context in VWs
25.
26. 1. Multiple communication threads occurring at
the same time
2. Text and audio channels often operate
separately
3. Sometimes text and audio channel are
interdependent
4. Meaning is often dependant on the visual and
spatial modes in the MUVE
5. Meaning is built through different modes
Communication features in a formal context in a VW
27. 1. Multiple communication threads
TIME TEXT CHANNEL AUDIO CHANNEL
0:53:48 -
0:54:02
CCo: HOLA!! T: a:nd who else do we have here? EC,
yeah? Who doesn't have a role yet?
0:54:02 –
0:54:31
AGe: kien va conmigo T: ah Maska, OK. So guest 5 is Maska and
the receptionist, JR? Can you hear me? JR?
Hello? Ah OK, you don't have voice either.
OK, we'll do it by chat
MJ: carlaaaaaa!
SM: mariaaaa
Dash: yoo andeer
JR: im heree
CCo: no escolto a
ninguuuuu
0:54:31
-0:54:38
AGe: ok kien eres dash T: we're going to go to [the front desk]
BR: [I'm here: yeaahhh already]
Communication features in a formal context in a VW
28. 2. Text and audio channels often operate
separately
TIME AUDIO CHANNEL TEXT CHANNEL
0:52:07 T: a:nd the receptionist for GM is going to
be… JG, are you ready?
0:52:15 JG: well… yes CCa: un moment q no se
pq no sem sent ara
0:52:16 T: OK I am going to give you, this,
notecard. OK, guest… number 4 i:s JT…
wait…
CCo: cristina, I don’t liset
anithing
0:52:24 SM: when do the class
finish?
0:52:30 GM: at 7
0:52:40 T:… and the receptionist for guest number
4 i:s…CCo where are you? Are you a
receptionist?
Communication features in a formal context in a VW
29. 3. Sometimes text and audio channel are
interdependent
TIME AUDIO CHANNEL TEXT CHANNEL
1:03:08 Dash: and I need that you make
a wake up call for me at 7am. If
it’s possible
TAnder (IM): do you have
any special requests?
1:03:18 AGe: 7am?
1:03:20 Dash: yeah. A wake up call
1:03:25 AGe: OK. Do you have any
special requests? Apart from
this?
Communication features in a formal context in a VW
30. 4. Meaning is often dependant on the visual and
spatial modes
TIME AUDIO VISUAL DESCR
0:44:53 T: Some people are going to be here,
next to the sofas, and the other
people are going to be next to the
table
T is standing between
the wall and the sofa
0:46:03 T: Can you sit on the sofa: Jordi,
Carme…
0:47:22 Please, don’t move so much! When
I tell you to stay somewhere, stay
there
SS are moving around
and are not staying with
their groups
0:48:52 DM: which table?
0:48:53 T: here T walks and stands
beside the table in the
middle
0:48:55 DM: ahhh OK
Communication features in a formal context in a VW
31. 5. Meaning is built through different modes
EXAMPLE 1
1. Audio 1:07:10 T : Norah? SM? Can you stand next to
JT?
2. Text chat 1:07:18 T: SM, stand next to JT
3. Spatial 1:07:21 SM runs to JT and pushes him back
Communication features in a formal context in a VW
32. EXAMPLE 2
1. Audio 1:13:10 T: Now, DM is having problems, I think
2. Chat 1:13:07 DM: I’m talking alone
1:13:09 JT: I don’t listen nothing
3. Interface 1:12:19 T opens her active speakers window to
adjust DM’s volume
Communication features in a formal context in a VW
5. Meaning is built through different modes
36. Concluding remarks
• There is a steep learning curve so a lot of class time is
spent on technical issues
• There are some strategies which are especially relevant in
a virtual world
• Communication occurs through multiple channels which
are sometimes interdependent and together they create
meaning
• Multiplicity of communication threads occurring at the
same time
• The different modes operate in a virtual world (and how
they are related) should be taken into account when
transcribing or analyzing interaction
39. • Campbell, A. (2003). Foreign language exchange in a virtual world: An intercultural Task-based Learning
event. Unpublished paper written in partial fulfillment of an MEd in e-Learning at the University of Sheffield, U.K.
Retrieved from:
http://e-poche.net/files/flevw.html
• Cassel et al. (2001). More than Just a Pretty Face: Conversational Protocols and the Affordances of Embodiment.
Knowledge-Based Systems, 14, 55–64.
• Gee, J. P. (2002). A sociocultural perspective on early literacy development. In S. B. Neuman & D. K. Dickinson
(Eds.), Handbook of early literacy research, 30–42. New York: Guilford Press. ecutive Summary (1999).
American Council on the Teaching of Foreign Languages ACTFL. Retrieved from
http://www.actfl.org/files/public/
• Kress, G. & Van Leewen T. (2001). Multimodal discourse: The modes and media of contemporary
communication. London: Arnold
• Norris, S (2004). Analyzing Multimodal Interaction. A methodological framework. New York & London:
Routledge.
• Örnberg, T (2005). Örnberg Berglund, T. 2005. ”Multimodality in a three-dimensional voice chat”. In: Allwood et
al. (Eds.): Proceedings from the Second Nordic Conference on Multimodal Communication
• Sweeney, P., Palomeque, C., et al. (2010). Task Design for Language Learning in an Embodied Environment. In
Vincenti, G. & Braman, J. (eds.). Teaching through Multi-User Virtual Environments: Applying Dynamic
Elements to the Modern Classroom. Hershey,PA: IGI Global.
• Warburton, 2009. Second Life in Higher Education: Assessing the potential for and the barriers to deploying
virtual worlds in learning an teaching. British Journal of Educational Technology, 40(3), 414-426
• Warschauer, M., Turbee, L., & Roberts, B. (1996). Computer learning networks and student empowerment.
System, 14(1), 1-14.
References
Notes de l'éditeur
This study is based on the notion that meaning is not created only through verbal language but that meaning is created through the interaction of different modes such as verbal, images (still and moving), sound, music, etc.
Thus, when analyzing interaction in a VW these different modes have to be taken into account.
Sigrid Norris: it is never possible to count the modes available in a communicative situation, as they are merely heuristic units of analysis. She advocates an approach to multmodality where focus is on creation of modal density. There is no inherent hierarchy among modes. Modal density can be achieved either intensity (on mode is best suited to deliver a sms under present circumstances) and complexity (when several different modes are used simultaneously to deliver the same message and none of the channels is given higher prominence than others)
Dado, pues, que los mundos virtuales son espacios de comunicación multimodal, nos aproximamos a ellos desde la perspectiva de la Teoría Multimodal
En su desarrollo en la Teoría Multimodal, que focaliza su objeto de estudio en delimitar los modos con que se desarrolla la comunicación (Kress, 2010: 78), bajo el concepto de multimodalidad se sostiene que un conjunto de modos semióticos está siempre involucrado en toda producción y lectura de textos, de manera que es preciso comprender la manera de leer estos textos como textos coherentes en sí mismos por la interrelación de modos. Así, el significado de todo acto comunicativo solo se puede captar considerando todos los modos o vías de significación que se emplean en el acto en cuestión.
Se entiende por modo cualquier recurso, vía, medio o material disponible en el contexto comunicativo, que se emplee con el propósito de crear significado(s) (Kress, Leite-García y van Leeuwen, 2000: 377-383).
In this table we can see different types of virtual worlds and their affordances for language learning. There are many studies on the successful use of this type of technology for lang learning from mid 90’s onwards.
MOOs – are text-based virtual worlds (they are solely text based) where users interact in real time using only text chat. The advantage over other text-based synchronous chat programs is that they allowed participants to describe themselves. The world is created entirely by its users (through descriptions). Interaction could take place within spaces created by users. Thus learners are empowered as the world is created by the students. These type of worlds are good to foster creativity in students. It is also a persistent environment because it doesn’t disappear when the student isn’t online, it continuous to evolve. So the user has a reasonable chance of happening on other users to interact with depending on the popularity of the world. It also allows for a certain degree of anonymity as the student can choose their nickname to enter the world and they only use text chat. Because of this relative anonymity and because of the written channel, anxiety is eased in students with a reluctance to speak.
Graphic-based virtual worlds: here users are ‘present’ represented by an avatar. Communication is via various forms of synchronous text-based chat and some visual gestures. Users can walk run, fly and teleport and are able to create content. Campbell (2003) described a course for Japanese learners of English using Active Worlds.
Co-construction of knowledge: 3D structures and landscapes acting as clear evidence of knowledge constructed collaboratively, of social interaction and communication becomes necessary amongst the learners if they are to complete the given task.
Graphic-based virtual worlds with voice: improvement: a more sophisticated and flexible design capability and synchronous voice. Great sense of presence, embodiement. Yee (2007) conducted a study and SL that confirmed that social norms of gender, interpersonal distance and eye-gaze all transfer into VW. Cassel et al (2001) describe the concept of embodiement and the avatars as embodied agents. The addition of voice brings two important benefits: it strengthens the digital reprsentation and adds a key element missing from traditional e-learning environments for language learning. In contrast to traditional e-learning interactions, where the suer is generally focused on documents or text or is limited to audio, MUVE users see their avatar talking within a particular setting. The input received by users is connected to their own projected identity, the interpretation of others’ identities, their paralanguage: body movements, gestures, chronemics (use of time in on-verbal commmunication and paraverbal features: stress, intonation and purposeful silence)
Evolution which broadens the possiblities for language learning but also makes the communication processes richer and more complex
Virtual worlds in learning are becoming increasingly popular and there is an extended belief that VW can be used as VLEs basically for 4 reasons. VW allow for multimodal communication: voice chat, text chat, private and public. There is a strong social dimension as users are represented by avatars. The residents of VW are usually quite friendly and willing to help newcomers. Apart from interacting with other avatars, you can interact with objects (e.g. when you open a door) or with the environment itself (you can contribute to create the environment). It is an immersive experience: the environment really makes you feel like if you really were in that place.
Interaction occurs between people, or in a MUVE context, between avatars.
interactivity is the interaction that occurs between the learner and the computer
Non-verbal vs paralanguage?
Ornberg conducted a study using the VW Traveler and mentions 3 modes: audible, visual and spatial.
In her study she states that the audible mode usually has the highest modal density and the spatial mode also has a great importance in a MUVE context: how the environment is set for the task, forming circles for discussions, facing another avatar when talking to him/her.
I have added the textual mode as in SL users often use the text chat together with the voice chat, for backchannelling purposes, for interruptions or to start another tread.
interfacce
Refer back to’ Modalities in S’L slide. A transcript has been created for the modes defined before: text, audio, visuals/space, interface. You can synchronize the different transcripts and the video so when selecting a part of a transcript, that part of the video automatically plays and you can see the part of the other transcripts that correspond to that moment.
Channel checks were very present during the sessions (especially the first ones) due to technical problems that some students were having, this was also due to the lack of facial expressions and other non-verbal elements such as nodding to express understanding. Although the program has the gesture incorporated, it has to be activated by opening the avatar’s gestures folder and then selecting the ‘nodding gesture’ (takes time) so nobody used it. Thus, some parts in f2f communication which are taken for granted or are more fluent, have to be made explicit through verbal communication (oral or written).
When students had technical problems they either asked for help from the teacher or asked their classmates for help
The mode which was most prominent was the oral mode. The ‘default’ class mode was the oral. However, the students often used the text chat. The use of the chat had different functions: sometimes it was to compensate for problems with the oral channel, to interrupt, to open a different thread, to complain about technical problems or just for fun. Sometimes the text chat was the mode they used to mark a difference from the ‘formal class mode’ (audio).
OTHER
Communication
Expressing non-und, asking for rep – students were not afraid of expressing non-understanding or asking for repetition, they also tried to help each other when they could
Code-switching – especially when addressing each other through the chat
Affective
- The chat was often use to joke, to talk about out-of-world topics and to socialize, they also used emoticons sometimes to clarify that they were joking or to express frustration when they were having technical problems
Channel/comp checks: most prominent strategy to ensure that the channel was open
back channeling markers such as aha, mhm so that the student knew that he was being heard and to encourage the student to carry on
- identity check: although the person who is speaking has green waves above their head, with big groups, sometimes it’s difficult to see who is speaking
- technical helper
Clasroom management
A lot of activities had to be adapted or simplified due to technical problems
- the teacher inserted a lot of spatial references to help students understand the instructions
- the use of the chat by the teacher was different, it was used to repeat key words, concepts or instructions or to compensate forproblems with the audio channel
Use of their L1 when addressing each other
In this slide we can observe 5 features which have been salient in the from the data that I gathered.
As there are different channels of communication, there are often multiple communication threads at the same time. Sometimes within the same channel.
Text and audio channels often operate separately as sometimes the students used as as their own communication channel, apart from the formal class channel which was the audio channel
Other times, both channels were used in combination
Because a virtual world is a multiomdal medium, meaning proved to be heavily dependant on the visual and spatial mode
Meaning is built through several modes, not only text and audio but also visual/spatial
T is trying to pair up students (form groups), in the chat we can see two different threads, a group of students calling each other and a student who is asking who his partner is (on chat tread is related to the class and the other isn’t)
The classroom discourse is full of references to objects in the MUVE so a visual description is necessary when transcribing
In this case, apart from the audio and the text chat, we need a visual description to complete the meaning of what is happening here
We need a description of the interface of the T’s computer to know what the T is doing to try to solve the technical problem that the student is having