This document discusses the potential for learning analytics to provide insights into student learning and outcomes from educational technology usage data. It provides examples from two studies conducted at a university. The first study found that LMS access data predicted student grades better than demographic variables and identified an "over-working gap" for lower-income students. The second study tested learning analytics triggers and interventions but found no significant impact on grades. The document argues for expanding learning analytics efforts, addressing challenges around data quality and governance, and integrating analytics into core applications.

1. The Virtuous Loop of Learning
Analytics & Academic Technology
Innovation
John Whitmer, Ed.D.
Director for Teaching & Learning Analytics and Research,
Blackboard
Adjunct Faculty Fellow, San Diego State University
john.whitmer@blackboard.com | @johncwhitmer
www.johnwhitmer.info
Online Learning
Consortium Collaborate
November 19, 2015

2. Quick bio
15 years managing academic technology
at public higher ed institutions
(R1, 4-year, CC’s)
• Always multi-campus projects, innovative uses
of academic technologies
• Driving interest: what’s the impact of these projects?
Most recently: California State University,
Chancellor’s Office, Academic Technology Services
Doctorate in Education from UC Davis (2013)
with Learning Analytics study on Hybrid,
Large Enrollment course
Active academic research practice
(San Diego State Learning Analytics, MOOC
Research Initiative, Udacity SJSU Study…)

3. Quick poll
A Unfamiliar; Never heard of it
Somewhat familiar; I’ve seen a reference or two
Very familiar; I follow the literature and/or use it in my practice
Expert; I’m very knowledgeable and actively contributing to the field
How familiar are you with learning analytics?
B
C
D

4. Do you ever wonder (or perhaps worry) …
How much the programs you invest
your time, energy and resources
into improve student outcomes?
And in what ways? (post-hoc) If your
programs are
helping the
right students?
(e.g. those
who need it)
If you could understand how
students interact with technology
experiences to a) create optimal
experiences for students or b)
identify students who are
struggling? (design research)

5. 1. What’s Learning Analytics
2 .What we’re learning from research
3. Examples of Learning Analytics (time permitting)
Outline

6. 1. Defining
Learning
Analytics

7. 200MBof data emissions annually
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.

8. Logged into course
within 24 hours
Interacts frequently
in discussion boards
Failed first exam
Hasn’t taken
college-level math
No declared major

9. What is learning analytics?
Learning and Knowledge
Analytics Conference, 2011
“ ...measurement, collection,
analysis and reporting of data about
learners and their contexts,
for purposes of understanding
and optimizing learning
and the environments
in which it occurs.”

10. Strong interest by faculty & students
From Eden Dahlstrom, D. Christopher Brooks, and Jacqueline Bichsel. The Current Ecosystem of Learning Management Systems in Higher Education: Student, Faculty,
and IT Perspectives. Research report. Louisville, CO: ECAR, September 2014. Available from http://www.educause.edu/ecar.

11. 2. What we’re
learning
from research

12. Study #1: Learning analytics pilot study
for Introduction to Religious Studies
Redesigned to hybrid delivery
through Academy eLearning
Enrollment: 373 students
(54% increase on largest section)
Highest LMS (Vista) usage entire
campus Fall 2010 (>250k hits)
Bimodal outcomes:
• 10% increased SLO mastery
• 7% & 11% increase in DWF
Why? Can’t tell with aggregated
reporting data
54 F’s

13. Grades Significantly Related to Access
Course: “Introduction to Religious Studies”
CSU Chico, Fall 2013 (n=373)
Variable % Variance
Total Hits 23%
Assessment activity hits 22%
Content activity hits 17%
Engagement activity hits 16%
Administrative activity hits 12%
Mean value all significant
variables 18%

14. LMS Use better Predictor than Demographic Variables
Variable % Variance
HS GPA 9%
URM and Pell-Eligibility
Interaction 7%
Under-Represented Minority 4%
Enrollment Status 3%
URM and Gender Interaction 2%
Pell Eligible 2%
First in Family to Attend College 1%
Mean value all significant
variables 4%
Not Statistically Significant
Gender
Major-College
Variable % Variance
Total Hits 23%
Assessment activity hits 22%
Content activity hits 17%
Engagement activity hits 16%
Administrative activity hits 12%
Mean value all significant
variables 18%

15. At-risk students: “Over-working gap”

16. Activities by Pell and grade
Grade / Pell-Eligible
A B+ C C-
0K
5K
10K
15K
20K
25K
30K
35K
Measure Names
Admin
Assess
Engage
Content
Not Pell-
Eligible
Pell-
Eligible
Not Pell-
Eligible
Pell-
Eligible
Not Pell-
Eligible
Pell-
Eligible
Not Pell-
Eligible
Pell-
Eligible
Extra effort
In content-related
activities

17. Study #2: Learning analytics triggers &
interventions
President-level initiative
Goals: (1) find accurate learning analytics
triggers; (2) create effective interventions
Multiple “triggers” (e.g. LMS access,
Clicker use, Grade) to identify at-risk
students, sent “awareness” messages
Conducted 2 Academic Years
(Spring 2014 – present)

18. Study Design
Select Courses
•High integration
academic
technologies
•High repeatable
grade rates
Identify
meaningful
triggers for course
•Consult with faculty
•Consider timing for
impact
Recruit Students
•Assign to
experimental/control
group
Run weekly
triggers
•Identify students at-
risk (or deserving
praise)
Send
“intervention”
message
•To students in
experimental group
only

19. Study Participation by Course
(Spring 2015)
Course Professor Format Enrolled Participating
Percent
Participation
ANTH 101 - 01 S. Kobari Online 454 126 28%
ANTH 101 - 03 S. Kobari F2F 175 67 38%
COMPE 271 - 01 Y. Ozturk Hybrid 96 63 66%
ECON 102 - 04
C. Amuedo-
Dorantes Hybrid 139 50 36%
PSY 101 - 01 M. Laumakis Hybrid 137 81 59%
PSY 101 - 02 M. Laumakis Hybrid 482 257 53%
STAT 119 - 03 H. Noble Hybrid 496 305 61%
STAT 119 - 04 H. Noble Hybrid 372 190 51%
TOTAL 2,351 1,139 61%

20. 9
10
13
10
8 8
13
11
5
3
5 5
5
6
2
2
2 2
3 3
2
2
1
2
1
2
2 2
2
2
0
5
10
15
20
25
Learning Analytics Trigger Events by Type
(Spring 2015)
High Grade Triggers
Low Grade Triggers
No Clicker
Low/No Blackboard Use

21. Correlation Individual Variables
w/Outcomes (all courses)
Significant Demographic/Educational
Prep Variables
Numeric
Grade
Repeatable
Grade
GPA @ Census 0.2989* 0.2508*
Units that Term 0.0879* 0.0643*
SAT (Comp) 0.0686* 0.0817*
EOP Status -0.0783* -0.0477
Age -0.0769* -0.0719*
Pell Eligibility -0.0843* -0.0806*
LA Interventions -0.4261* -0.2576*
LA Interventions +
Grade -0.5979* -0.4305*
Not Significant**
Student Level
Sex
Ethnicity
Enrollment Status
MajorCollege
Honors
Disabled
EOP
Dorm Resident
Low Income EFC
First Gen Some College
Learning Community
** Variables highly correlated w/other predictors were excluded in favor of variable closest to
current experience, e.g. GPA @ Census (not HS GPA), SAT Comp (not SAT Math), etc.

22. 0.74
0.25
0.1
0.38
0.64
0.5
0.33
0.46
0.68
0.13
0
0.21
0.44
0.27
0.17
0.28
0 0.2 0.4 0.6 0.8
Anth 101 (Online)
Anth 101
(In Person)
Comp Eng 271
Econ 102
Psych 101-01
Psych 101-02
Stat 119-03
Stat 119-04
R2 Value
Relationship between Learning Analytics Triggers Activated and
Final Grade (Spring 2015)
Behavioral Triggers
Behavioral + Grade
Triggers

23. 0%
20%
40%
60%
80%
100%
120%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
RepeatableGradeRateAverage
Triggers Activated
Triggers Activated vs. Repeatable Grade

24. Click to edit Master title style
What does this mean?
What students DO is more important than who
they are
LMS use is a proxy for student effort
Can we get more sophisticated? Yes.

25. Click to edit Master title style
A Typical Intervention
30
… data that I've gathered over the years via clickers indicates
that students who attend every face-to-face class meeting reduce
their chances of getting a D or an F in the class from
almost 30% down to approximately 8%.
So, please take my friendly advice and attend class and
participate in our classroom activities via your clicker. You'll be
happy you did!
Let me know if you have any questions.
Good luck,
Dr. Laumakis

26. Did the interventions make a
difference? Nope.80
86
69
80
74
79
77
73
81
87
65
82
70
79
80
75
0
10
20
30
40
50
60
70
80
90
100
Anth 101
(Online)
Anth 101
(In Person)
Comp Eng
271
Econ 102 Psych 101-
01
Psych 101-
02
Stat 119-
03
Stat 119-
04
Comparison of Course Grade Average between Experimental and
Control Groups (by Course)
Control
Experimental

27. Click to edit Master title style
Next Steps
1. Currently testing “Supplemental Instruction” near-peer
tutoring approach developed by UMKC
– Initial results look very promising/promising: >10+ increase in grade
of students who attend Supplemental Instruction vs. those who
don’t
2. Combine with predictive model analysis with Learning
Analytics (“Doing the Right Thing” score) + Demographic
variables
3. Expand to additional courses; evaluate other intervention
approaches

28. Summary findings previous LMS analytics
studies
Institution-Wide Analysis
with Only LMS Data
Course-Specific
with Only LMS Data
Course-Specific
with LMS Data & Other Sources
%GradeExplained#
60%
50%
40%
30%
20%
10%
0%
25%
4%
51%
0%
33% 31%
57%
35%
(Whitmer,
2013a)
(Campbell
2007a)
(Campbell
2007b)
(Jayaprakash,
Lauria 2014)
(Macfadyen
and Dawson
2010)
(Morris,
Finnegan et al.
2005)
Whitmer &
Dodge (2015)
Whitmer
(2013b)
Hybrid
Course
Format:
Hybrid,
online
Online

29. Factors affecting growth of learning analytics
Enabler
Constraint
WidespreadRare
New education
models
Sufficient
Resources
($$$, talent)
Clear data governance
(privacy, security,
ownership)
Clear goals
and linked
actions
Data valued in academic
decisions
Tools/systems
for data
co-mingling
and analysis
Academic
technology adoption
Low data quality (fidelity
with meaningful learning)
Difficulty of data
preparation
Not invented here
syndrome

30. Call to action
(from a May 2012 Keynote Presentation @ San Diego State U)
You’re not behind the curve, this is a rapidly emerging area
that we can (should) lead...
Metrics reporting is the foundation for analytics
Start with what you have! Don’t wait for student characteristics and
detailed database information; LMS data can provide significant insights
If there’s any ed tech software folks in the audience,
please help us with better reporting!

31. 3. Examples of
Learning
Analytics

32. Forward looking statements
Statements regarding our product development initiatives,
including new products and future product upgrades, updates
or enhancements represent our current intentions, but may be
modified, delayed or abandoned without prior notice and there
is no assurance that such offering, upgrades, updates or
functionality will become available unless and until they have
been made generally available to our customers.

33. Purpose-Built Learning Data Collection (in Bb Learn)
Learn Activity in Q4 Release
Grade History Events
Content Management Events
Test Access Events
External Launch Events
Student Contribution Events
User
Sessions
Log Events
Tracking Event Events (Activity Accumulator)
Institution
Data Store
Event
Listener
Clickstream
Listener
Event
Listener
Client
Listener
Log
Listener
Grade
Point-in-Time
Content
Point-in-Time

34. Bb Data Privacy/Confidentiality
Blackboard will Blackboard will not
• Respect regional data privacy/
confidentiality laws, starting
with keeping detailed customer data
within their region (e.g. Australia,
Germany, North America)
• Analyze only anonymized data
for cross-institutional insights
• Provide raw data to campuses
at no additional charge as part
of “core” platform services
• Sell raw or individually-identifiable
customer data

35. Platform Analytics Initiative
Improved, purpose-built data sources
• Initially about academic technology interactions
• Extending to other aspects of student experience
Validated data elements and models
• Based in large-scale analysis, using inferential statistics
and data mining on anonymized data
Integrated interventions and actions
within core application workflows
• Providing actionable insights where action can be taken immediately

36. Embedded Learning Analytics in
Learn SaaS “Ultra”

37. Learner Experience

43. Teacher Experience

49. X-Ray Learning Analytics for Moodlerooms

50. Activity over Time

51. Activity by Week Relative to
Other Students

52. Discussion Forum Wordcloud

53. Social Network Analysis of Discussion Forum w/Critical
Thinking Coefficient

54. Risk Status

55. Discussion Forum Grading Suggestions

56. Thank you!
John Whitmer, Ed.D.
john.whitmer@blackboard.com
@johncwhitmer
www.johnwhitmer.info/research