Presentation by John Whitmer, Michael Haskell (Cal Poly SLO), and Hillary Kaplowitz (CSU Northtridge) at US West Coast Moodle Moot 2012. “Learner Analytics” has captured the attention of the media and is the topic of much debate in professional and academic circles. What lies behind the hype? In this presentation, we will discuss the state and limits to current in research in LMS Learner Analytics. We will then look at examples of Learner Analytics in Moodle, including tools for faculty and reports for reporting across the entire instance.

1. Learner Analytics: Hype, Research
and Practice in Moodle
US West Coast Moodle Moot 2012
John Whitmer, CSU Chico (& Office of the Chancellor)
Michael Haskell, Cal Poly San Luis Obispo
Hillary Kaplowitz, CSU Northridge Download slides at:
http://bit.ly/QttGnd

2. “But everything we know about cognition
suggests that a small group of people, no
matter how intellingent, simply will not be
smarter than the larger group. ...
Centralization is not the answer. But
aggregation is.”
- J. Surowiecki, The Wisdom of Crowds, 2004
2

3. Outline
1. Hype & Promise of Learner Analytics
2. Campus Case Studies
– Getting Started w/Institutional Reporting (Mike)
– Analytics at work in the classroom (Hillary)
– Evaluating course redesign (John)
3. Q & A

4. 1. HYPE & PROMISE OF LEARNER
ANALYTICS

5. John Goodlad’s Place-Based Research
 Classroom-based
research: “What is
schooling?”
 1,000 classrooms,
27,000 individuals
 14 foundations needed to
support
 Fundamental changes to
understanding of
educational practice

6. Steve Lohr, NY Times, August 5, 2009

7. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire
economy. The Economist.

8. Source: jisc_infonet @ Flickr.com
8
Source: jisc_infonet @ Flickr.com

9. Learner Analytics
“ ... measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs.” (Siemens, 2011)

10. Fundamental Questions behind
Learner Analytics
1. How are students using technology?
2. Does it matter (re: achievement,
engagement, learning)?
3. How does this relationship vary
(by student, by course, by goal)?
4. What should we do?
– Changes in student behavior?
– Changes in faculty/program?

11. L.A. Empirical Research Findings

12. 2. CAMPUS CASE STUDIES
Download slides at:
http://bit.ly/QttGnd

13. Getting Started with
Institutional Reporting
Michael Haskell
Cal Poly, San Luis Obispo

14. We perform Analytics, but are we doing Learning Analytics?
?

15. Can we wait
that long?
How far away is
Learning Analytics?
Sounds like it’s
about 2-3 Years
out…
What can we do in the meantime…

16. Institutional Reporting
What information is available?
Where is it?
How can we use it?

17. Types of Information
Content Individual Behavior Population Behavior

18. Location of Information
Web Server Logs Population
Moodle Database Structure Content Individual
Content Individual Population
Moodle Log Table (mdl_log)
Google Analytics Population

19. Moodle Database Structure
http://docs.moodle.org/dev/Database_schema_introduction
Modules by Course SQL: https://gist.github.com/3203120

20. How do we utilize this information?
Foster collaboration between Faculty
“ Top 10 Instructors Tab
In this section, the data was further categorized to find the top 10
instructors in each college who “used the most modules” and “created
the most of each module”. The first two graphs show the top 10
instructors from all the colleges.
In the first graph, the instructors who used the most modules (8
modules) were X and Y, who are from the College of Engineering and
College of Ag, Food and Env respectively. In that same section, Z
”
from the College of Science and Math is listed down three times for
classes in the top 10.
- Student Researcher

21. How do we utilize this information?
To keep a pulse on adoption

22. How do we utilize this information?
To keep a pulse on adoption
Percentage of Activated Courses by College (Spring 2012)
College of Agriculture, Food College of Architecture &
and Environmental Science Environmental Design College of Engineering
16% 21%
29%
71% 79%
84%
College of Science &
College of Liberal Arts
Orfalea College of Mathematics
Business
26%
37%
47%
53% 63%
74%

23. How do we utilize this information?
To learn how instructors leverage Moodle.
Determine where developer time is best spent.

24. How do we utilize this information?
Informed Communication
Moodle Admin: There’s a problem with Module X.
Instructional Designer: The problem will be fixed soon, but in the meantime
I have a workaround I’d like to communicate to instructors. Hmm… I don’t
want to reach out to every instructor. Can you provide a list of all the
instructors who use Module X?
Moodle Admin
No Problem.

25. Conclusions
• Current
• Manual Exploration
• A lot of Small Wins
• Future
• Automate reporting of top tens
• Open up the data to a wider audience
• Take action on data we have
• Keep an eye on LA Tools for faculty and
students

26. How can data help teachers
and students work better
together?
Hillary Kaplowitz
Instructional Designer, Faculty Technology Center
Part-Time Faculty, Cinema and Television Arts
Department
California State University, Northridge

27. Case #1
“I'm not upset that you lied to
me, I'm upset that from now on
I can't believe you.”
Friedrich Nietzsche

30. “Hey Professor,
I just looked at my assignments and
realized that my Chapter 11 summary
did not get submitted, which I'm having
trouble believing that I didn't submit it...
especially because I see that I did it,
and I always submit my assignments
as soon as I finish them.”

31. Now the hard part….
Do I believe him?
If I only I could check…

34. And it was all his idea…
The student suggested that I check Moodle and if
that didn’t work told me how to check the Revision
History in GoogleDocs with step-by-step
directions!

36. Case #2
“Life isn't fair. It's just fairer
than death, that's all.”
William Golding

37. “The quiz is unfair”

38. Hybrid Course Weekly Structure
4. Post
3. Online questions
1. Watch 2. Read 5. Class 6. Aplia
chat and and take
lectures textbook meets quiz
tutoring practice
quiz

39. But the story was not that simple…
• Reports on Moodle painted a different picture
• Student was watching the lectures at 10:00 p.m.
• Then immediately taking quiz

40. Enabled constructive feedback…
 Advised the student how the structure of the
course was designed to enhance learning
 Student revised their study habits
 Improved grades and thanked the instructor!

41. What we can do with data now
 Use Reports in Moodle to verify student claims
 Review participant list to see last access time
 Empower students to review their own reports
 Analyze usage and advise students how to study better
 Review quiz results to find common misconceptions

42. Could we help improve student learning
outcomes if we knew the effect of…
Coffee
Facebook Sequencing
Attendance Amount
Mobile Textbook
LMS LMS
Activities Access

43. EVALUATING COURSE REDESIGN:
INTRO TO RELIGIOUS STUDIES 180
43

44. Front-end: What? Why?
Evaluation for Program Assessment
• Year-long faculty course redesign program
• Case: Intro to Religious Studies: increased enrollment from
80 to 373 students first semester: 250,000 course website hits
• Outcome: increased mastery course concepts AND increased
number D/W/F students
• Why? (and for whom?)
• What is the relationship between LMS actions, student
background characteristics and student academic achievement? (6
million dollar question)
44

45. Back-end: How?
• Integrated data from LMS log files, student
enrollment records, and course grade
• LMS logfiles are “data exhaust” for server
analysis
• Filtering and cleaning reduced 250K records to
71k
• Analysis tools: Excel, Tableau
(visualization), Stata (statistical analysis)
45

46. 46

47. Grades by Hits & Dwell Time
47

48. Pell v. Non-Pell: Grades by Hit/Dwell

49. Content: the Time Differential
49

50. Call to Action
1. You’re *not* behind the curve, this is a rapidly
emerging area that we can (should) lead ...
2. Metrics reporting is the foundation for Analytics
3. Don’t need to wait for student characteristics
and detailed database information; LMS data
can provide significant insights
4. If there’s any ed tech software folks in the
audience, please help us with better reporting!

51. Draft DOE Report
released April 12
http://1.usa.gov/GDFpnI

52. Q&A and Contact Info
Download slides at: http://bit.ly/QttGnd
Resources Googledoc: http://bit.ly/HrG6Dm
Contact Info:
• John Whitmer (jwhitmer@csuchico.edu)
• Michael Haskell (mhaskell@calpoly.edu)
• Hillary C Kaplowitz (hillary.kaplowitz@csun.edu)
52

53. Works Cited
Adams, B., Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching Learning through
Educational Data Mining and Learning Analytics: An Issue Brief. Washington, D.C.: U.S. Department of
Education, Office of Educational Technology.
Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1).
Bousquet, M. (2012). Robots Are Grading your Papers. Retrieved from
http://chronicle.com/blogs/brainstorm/robots-are-grading-your-papers/45833
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era.
EDUCAUSE Review, 42(4), 17.
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and
perhaps the entire economy. The Economist.
LaValle, S., Hopkins, M., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The new path to
value. Findings from the 2010 New Intelligent Enterprise Global Executive Study and Research Project:
IBM Institute for Business Value and MIT Sloan Management Review.
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data:
The next frontier for innovation, competition, and productivity.
Parry, M. (Producer). (2012, 5/14/2012). Me.edu: Debating the Coming Personalization of Higher Ed.
Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/me-edu-
debating-the-coming-personalization-of-higher-ed/36057
Siemens, G. (2011, 8/5). Learning and Academic Analytics. Retrieved from
http://www.learninganalytics.net/
53