This presentation to the MoodleMoot UK/I 2017 provides an overview of Learning Analytics for VLE/LMS data and lessons learned in practice from using this data to model student risk and other characteristics. The findings come from fundamental research and application of Blackboard's X-Ray Learning Analytics application.

1. Lessons from Moodle
Data
“Dr. John” Whitmer
Director, Analytics and Research
MoodleMoot UK/I | 12-April 2017

2. 1. Learning Analytics Overview
& Bb Data Science
2. Research & Modeling Findings
1. Virtual South Carolina Moodle Predictive Model
2. Differences in Student Achievement by Tool Use
3. Discussion

3. Learning Analytics Overview

4. Educational Technology Assessment Hierarchy
Does it impact
student learning?
(Learning Analytics)
How many people use it?
(Adoption)
Does it work? (SLAs)

5. What is Learning Analytics?
Learning and Knowledge
Analytics Conference, 2011
“ ...measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding
and optimizing learning
and the environments
in which it occurs.”

6. Techniques
• Simulation if X, what Y? (“With this Ultra
Learning Analytics trigger rule, how many
students would trip notified?”)
• Hypothesis testing: investigate if a specific
relationship is true (“What’s the relationship
between time spent in a course and student
grade”?)
• Data mining: analyze underlying latent patterns
in data (“What typical patterns in tool use
characterizes BB Learn courses?”)
Key Data Sources
• Moodlerooms & X-Ray
• Learn Managed Hosting & SaaS
• Collaborate Ultra
Main Big Data Sources & Techniques

7. Commitment to Privacy & Openness
• Analyze data records
that are not only
removed of PII, but
de-personalized
(individual &
institutional levels)
• Share results and open
discussion procedures
for analysis to inform
broader educational
community
• Respect territorial
jurisdictions and safe
harbor provisions

8. Virtual South Carolina Online

9. Feature Importance in Predictive Model

10. Predictive Accuracy for Risk Categories

11. Prediction vs. Final Grade

12. Performance of Predictions

13. Models Change by Week & by Course Type

14. So what?
• Rolling out risk model & X-Ray broadly for
teachers
• Providing as useful indicator to augment
their decisions
(not source of absolute truth)
• Remaining challenge: help teachers
interpret probabilistic results

15. Large Scale Research: Student LMS Use vs.
Grade

16. Findings: Relationship LMS Time & Grade
• Question: what is the
relationship between student
time in LMS and their course
grade?
• Investigate at student-course
level (one student, one course)
• 1.2M students, 34,519 courses,
788 institutions
• Significant, but effect size < 1%

17. But strong effect in some courses
(n=7,648, 22%)

18. What makes
some for a
stronger or
weaker
relationship?
Tools used?
Course design?
Quality of
activity/effort?

19. Finding: Access to Grades
At every level, probability of higher grade increases with increased use.
Causal? Probably not. Good indicator? Absolutely.

20. Finding: Course Contents
More is not always better. Large jump none to some; then no relationship

21. Finding: Assessments/Assignments
Students above mean have lower likelihood of achieving a high grade than students below the mean

22. Finding: Discussion Forums with low/high avg use
Compare courses with low forum use to courses with forum use >1 hour / student average

23. Implications
• Move beyond LMS use as proxy for effort (where more is always better), and get at finer-grained
learning behaviors that are more useful (e.g. students who are struggling to understand material,
students who are not prepared).
• Next Steps
– fine-grained understanding of activity over time (e.g. cramming vs. consistent hard working)
– quality of course materials and course design

24. Discussion & Contact Information
John Whitmer (john.whitmer@blackboard.com)
johncwhitmer