Review of research metholdogies, hints and tips for PhD seminar at Aarhus University, Denmark. January 2009

1. Overview of Research
Methodologies for Social
Software Research
Aarhus University, Denmark January 21, 2009
Terry Anderson, Ph.D.
Canada Research Chair in Distance EducaAon

2. The context of Social SoFware
ImplementaAon
• DisrupAve (Christensen, 2008) simpler, not
wanted by main stream customers
• Rapid gains in funcAonality
• Cheaper
• AdapAve

3. Studying Technological Based Models
of Change
• Technological determinism (Marx, Zuboff,
Kurzweil)
• Affordances and Social ConstrucAvist (Bjiker,
Stewart and Williams, Dalsgaard
• DomesAficaAon/AppropriaAon –(Silverston,
Ling)

5. Research Paradigms

6. Paradigm 1
QuanAtaAve Research

7. • “those who are seeking the strict way of truth
should not trouble themselves about any
object concerning which they cannot have a
certainty equal to arithmeAc or geometrical
demonstraAon”
– (Rene Descartes)
• Inordinate support and faith in randomized
controlled studies

8. QuanAtaAve 1 –
CMC Content Analysis
• Anderson, Garrison, Rourke 1997‐2003
– hcp://communiAesofinquiry.com ‐ 9 papers reviewing results
focusing on reliable , quanAtaAve analysis
– IdenAfied ways to measure teaching, social and cogniAve
‘presence’
– Most reliable methods are beyond current Ame constraints of
busy teachers
– QuesAons of validity
– Serves as basic research as grounding for AI methods of the
future
– Serves as qualitaAve heurisAc for teachers and course designers

9. QuanAtaAve – Meta‐Analysis
• Ungerleider and Burns (2003)
• SystemaAc review of effecAveness and efficiency of
ICT
• The type of interventions studied were
extraordinary diverse –only criteria was a
comparison group
• “Only 10 of the 25 studies included in the in-
depth review were not seriously flawed, a
sobering statistic given the constraints that went
into selecting them for the review.”

11. Is DE Becer than Classroom InstrucAon?
Project 1: 2000 – 2004
• QuesAon: How does distance educaAon compare
to classroom instrucAon? (inclusive dates
1985‐2002)
• Total number of effect sizes: k = 232
• Measures: Achievement, Amtudes and RetenAon
(opposite of drop‐out)
• Divided into Asynchronous and Synchronous DE
11

12. Primary findings
• DE and CI are essenAally equal (g+ ≈ 0.0 to low
average effect) on all measures
• Effect size distribuAons are heterogeneous; some
DE >> CI, some DE << CI
• Generally poor methodological quality
• Pedagogical study features account for more
variaAon than media study features (Clark, 1994)
• InteracAve DE an important variable*
*Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate distance
education: A theory-based meta-analysis of empirical literature. Educational Technology
Research & Development, 54(2), 141-176.
12

13. Summary of results: Achievement
Achievement Outcomes
*Significantly heterogeneous average effect
13

14. Summary of results: Amtudes
Attitude Outcomes
*Significantly heterogeneous average effect
14

15. Summary of results: RetenAon
Retention Outcomes
*Significantly heterogeneous effect sizes
15

16. Equivalency: Are all types of
InteracAon necessary?
Anderson,
2003
IRRODL

17. Anderson’s Equivalency Theorem
(2003)
Moore (1989) disAncAons are:
 Three types of interacAon
o student‐student interacAon
o student‐teacher interacAon
o Student‐content interacAon
Anderson (2003) hypotheses state:
 High levels of one out of 3 interacAons will produce
saAsfying educaAonal experience
 Increasing saAsfacAon through teacher and learner
interacAon interacAon may not be as Ame or cost‐effecAve
as student‐content interacAve learning sequences
17

18. Do the three types of interacAon
differ? Moore’s disAncAons
Achievement and Attitude Outcomes
Achievement Attitudes
Interaction
Categories k g+adj. k g+adj.
10 0.342 6 0.358
Student-Student
44 0.254 30 0.052
Student-Teacher
20 0.339 8 0.136
Student-Content
44
74 0.291 0.090
Total
2.437 6.892*
Between-class
Moore’s distinctions seem to apply for achievement (equal importance), but not for
attitudes (however, samples are low for SS and SC)
18

19. Does strengthening interacAon improve achievement
and amtudes? Anderson’s hypotheses
Achievement and Attitude Outcomes
Achievement Attitudes
Interaction
Strength k g+adj. SE k g+adj. SE
Low Strength 30 0.163 0.043 21 0.071 0.042
Med Strength 29 0.418 0.044 18 0.170 0.043
High Strength 15 0.305 0.062 5 -0.173 0.091
Total 74 0.291 0.027 44 0.090 0.029
(Q) Between-class 17.582* 12.060*
Anderson’s first hypothesis about achievement appears to be supported
Anderson’s second hypothesis about satisfaction (attitude) appears to be
supported, but only to an extent (i.e., only 5 studies in High Category)
19

20.  Bernard, Abrami, Borokhovski, Wade, Tamin,
& Surkes, (in press). Examining Three Forms of
InteracAon in Distance EducaAon: A Meta‐
Analysis of Between‐DE Studies. Review of
Research in Educa2on

21. Because social soFware use is so
emergent need for
• Demographic studies
– Who is using what for doing what?
– How are they using mulA‐use tools?
VisualizaAon of acAvity and relaAonships
•
Network analysis
•
ConnecAon between context and type of use
•
Data mining
•

22. QuanAtaAve Summary
• Can be useful especially when fine tuning well
established pracAce
• Provides incremental gains in knowledge, not
revoluAonary ones
• The need to “control” context oFen makes results of
licle value to pracAcing professionals
• In Ames of rapid change too early quanAtaAve
tesAng may mask beneficial posiAve capacity
• Will we ever be able to afford blind reviewed,
random assignment studies?

23. Paradigm 2
QualitaAve Paradigm
• Many different varieAes
• Generally answer the quesAon ‘why’ rather
then ‘what’ or ‘how much’?
• Presents special challenges in distributed
contexts due to distance between parAcipants
and researchers
• Currently most common type of DE research
(Rourke and Szabo, 2002)

24. QualitaAve study of Social SoFware
• CriAcally important:
– In early stages of adopAon
– To track effects of user competence and efficacy
– As contexts are personalized
– as tools are appropriated by users for enArely
different tasks than those intended by developers

25. QualitaAve Example
– Dearnley (2003) Student support in
open learning: Sustaining the Process
– PracAcing Nurses, weekly F2F tutorial
sessions
– Phenomenological study using
grounded theory discourse

26. Core category to emerge was “Finding the
professional voice”

27. QualitaAve example 2
• Mann, S. (2003) A personal inquiry into an experience of
adult learning on‐line. Instruc8onal Science 31
• Conclusions:
– The need to facilitate the presentaAon of learner and teacher
idenAAes in such a way that takes account of the loss of the normal
channel
– The need to make explicit the development of operaAng norms and
convenAons
– reduced communicaAve media there is the potenAal for greater
misunderstanding
– The need to consider ways in which the developing learning
community can be open to the other of uncertainty, ambiguity and
difference

28. 3rd Paradigm
CriAcal Research
• Asks who gains in power?
• David Noble’s criAque of ‘digital diploma Mills’
most prominent Canadian example
• Are profits generated from user generated
content exploitaAve?
• ConfronAng the “net changes everything”
mantra of many social soFware proponents.
• Who is being excluded from social soFware

29. Is the extracAon of informaAon from the masses
exploitaAve or empowering?

30. • Why does Facebook own all the content that
we supply?
• Does the power of the net further marginalize
the non connected?
• Who benefits from voluntary disclosure?
• Why did the One Laptop Per Child fail?

31. QuanAtaAve vs. QualitaAve
Paradigm Wars Rekindled
• Current research “more resembles the pendulum
swings characterisAc of art or fashion, rather than
the progressive improvements characterisAc of
science and technology” (p. 16).
• Slavin (2002) in EducaAonal Researcher
• SoluAon to embrace “evidence based learning”
• Projected to increase from 5% to 75% of US Gov.
funding by 2007 for “research that addresses causal
quesAons and uses random assignments ….” Slavin, 2002
p. 15

32. Do Either QualitaAve or QuanAtaAve
Methods Meet Real Needs of PracAcing
Distance Educators?

33. But what type of research has most
effect on pracAce?
– Kennedy (1999) - teachers rate relevance and
value of results from each of major
paradigms.
– No consistent results – teachers are not a
homogeneous group of consumers but they
do find research of value
– “The studies that teachers found to be most
persuasive, most relevant, and most
influential to their thinking were all studies
that addressed the relationship between
teaching and learning.”

34. But what type of research has most
effect on PracAce?
– “The findings from this study cast doubt on
virtually every argument for the superiority of
any particular research genre, whether the
criterion for superiority is persuasiveness,
relevance, or ability to influence practitioners’
thinking.” Kennedy, (1999)

35. 4th Paradigm
Design‐Based Research
• Related to engineering and architectural
research
• Focuses on the design, construcAon,
implementaAon and adopAon of a learning
iniAaAve in an authenAc context
• Related to ‘Development Research’
• Closest educators have to a “home grown”
research methodology

36. Design‐Based Research Studies
– iterative,
– process focused,
– interventionist,
– collaborative,
– multileveled,
– utility oriented,
– theory driven and generative
• (Shavelson et al, 2003)

37. Design‐based research
• Methodology developed by educators for
educators
• Developed from American pragmaAsm – Dewey
(Anderson, 2005)
• Recent Theme Issues:
The Journal of the Instruc1onal Sciences, (13, 1, 2004),
–
Educa1onal Researcher (32, 1, 2003) and
–
Educa8onal Psychologist (39, 4, 2004)
–
See bibliography at
–
hKp://cider.athabascau.ca/CIDERSIGs/
DesignBasedSIG/
• My ar8cle at www.cjlt.ca/abstracts.html

38. IntegraAve Learning Design
(Bannan‐Ritland, 2003)

39. • “design‐based research enables the creaAon
and study of learning condiAons that are
presumed producAve but are not well
understood in pracAce, and the generaAon of
findings oFen overlooked or obscured when
focusing exclusively on the summaAve effects
of an intervenAon” Wang & Hannafin, 2003

40. Wang and Hannafin, 2003

41. • IteraAve because
• ‘InnovaAon is not restricted to the prior design
of an artefact, but conAnues as artefacts are
implemented and used”
• ImplementaAons are “ineviAtably
unfinished” (Stewart and Williams (2005”

42. Design Based research and the Science of
Complexity
• Complexity theory studies the emergence of
order in mulAfaceted, changing and previously
unordered contexts
• This emerging order becomes the focus of
iterate intervenAons and evaluaAons
• Order emerges at the “edge of chaos” in
response to rapid change, and failure of
previous organizaAon models

43. D‐B Research examples
Design-Based Research Strategies for Studying Situated Learning
in a Multi-user Virtual Environment
Chris Dede, 2004

44. • Need to study usability, scalability and
innovaAon adopAon within bureaucraAc
systems
• Allow knowledge tools to evolve in natural
context through supporAve nourishment of
staff

45. Grounded theory
• The need for theory and model building
• “There is nothing so pracAcal as a good
theory”
• We need models and theories now to guide
the rapid deployment that is taking place
• Be brave and expansive in your conclusions

46. Reasons for CriAcal Mass in Networks
of PracAce
Landquvist &Teigland, 2007

47. Personal Academic use of social
soFware
• Do create a blog to document your research
journey
• Follow blogs of those you admire
• Self archive and only give your copyright away
under unusual circumstances
• Support open access journals ie
www.irrodl.org

48. Final Hints
• Subscribe to IRRODL www.irrodl.org
• Customize Google Scholar to scan your library
connecAons www.googlescholar.com
• Subscribe to my blog the Virtual Canuck at
terrya.edublogs.org

49. Conclusion
• EducaAon research is grossly under‐resourced to
meet the magnitude of opportunity and demand
• Paradigm wars are unproducAve
• Design‐based research offers a promising new
research design model
• The semanAc web offers promise of greatly
enhanced collaboraAon and research
disseminaAon