This document discusses system-wide learner analytics projects at Chico State University and across the California State University system. [1] It describes a research study at Chico State that analyzed the relationship between student usage of the learning management system and academic outcomes. [2] It then provides an overview of current learner analytics projects involving the Moodle and Blackboard learning management systems, including establishing common metrics and queries across campuses.

1. System-wide LMS
Learner Analytics Projects
Presenters: Kathy Fernandes and John Whitmer
ATSC Virtual Meeting
December 13, 2012
Slides @ http://goo.gl/DYqJU

2. Agenda
1. Chico State Learner Analytics Research Study
• EDUCAUSE Article (http://goo.gl/tESoi)
2. Current Projects
• Moodle
• Blackboard
2

3. 1. CHICO STATE LEARNER ANALYTICS
RESEARCH STUDY
“Logging on to Improve Achievement” by John Whitmer
EdD. Dissertation (UC Davis & Sonoma State)
3

4. Case Study: Intro to Religious Studies
• Redesigned to hybrid delivery through
Academy eLearning
54 F’s
• Enrollment: 373 students
(54% increase on largest section)
.8
• Highest LMS (Vista) usage
.6
entire campus Fall 2010
(>250k hits)
Density
.4
• Bimodal outcomes:
.2
• 10% increased SLO mastery
• 7% & 11% increase in DWF
0
0 1 2 3 4
Course Grade
• Why? Can’t tell with aggregated
reporting data
4

5. Driving Conceptual Questions
1. How is student LMS use related to academic
achievement in a single course section?
2. How does that finding compare to the relationship
of achievement with
traditional student characteristic variables?
3. How are these relationships different for
“at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are
most useful to answer these questions?
5

6. Variables
Student'Characteristic'Independent'Variables'
Gender
Under Represented Minority (URM)
Pell-Eligible
High School GPA
First in Family to Attend College
Student Major (Discipline)
Enrollment Status
Interaction URM & Gender
Interaction URM & Pell-Eligibility
Learning Management System Usage Variables
Total LMS course website hits
Total LMS course dwell time
Administrative tool website hits
Assessment tool website hits
Content tool website hits
Engagement tool website hits
Dependent'Variable:'Final'Course'Grade''' 6
!

7. Clear Trend: Grade w/Mean LMS Hits
7

8. Separate Variables: Correlation LMS Use &
Student Characteristic with Final Grade
LMS Student
Use Characteristic
>
Variables Variables
18% Average 4% Average
(r = 0.35–0.48) (r = -0.11–0.31)
Explanation of change Explanation of change
in final grade in final grade
8

9. Combined Variables: Regression Final Grade by
LMS Use & Student Characteristic Variables
LMS Student
Use Characteristic
>
Variables Variables
25% +10%
(r2=0.25) (r2=0.35)
Explanation of change Explanation of change
in final grade in final grade
9

10. At-Risk Students: “Over-Working Gap”
10

11. Filtering Data – Lots of “Noise”; Low “Signal”
500
450
400
350
300
382 250
200
150 Raw Average Hits/
151 Student
100
58 49 Filtered Average
50
26
Hits/Student
54 51
23 36
16
0
Discussion Activity Content Activity Hits Assessment Activity Mail Activity Hits Administrative
Hits Hits Activity Hits
Final data set: 72,000 records (-73%)
11

12. 2. CURRENT PROJECTS
12

13. Moodle and Bb Learner Analytics
What do these have in common?
• Multi-campus CSU groups discussing
common analytics questions & query
definitions
13

14. Moodle vs. Bb Learner Analytics
Moodle CIG (18 months old) Blackboard Learn Group
Chair: Andrew Roderick, SFSU (just starting)
CIG Chair: Terry Smith, CSUEB
§ DIY, adopt and evaluate § Bb Learn Analytics product
available “off the shelf”;
solutions from other defined and integrated with
Moodlers Peoplesoft
§ Starting with technical § Pre-built Reports and
reporting to build accurate Dashboards to ANYONE on
indicators of use campus (admin. or faculty if
§ 2 rounds of data collection authenticated)
already completed and § Charts available inside LMS
discussed for Faculty and Student
Views 14

15. Moodle Reporting & Analytics, Round 1
§ Prioritized Moodle Queries from S&PG
governance group
§ Focused on measures of adoption
(% faculty, % students, % course sections)
§ For expediency, campuses reported using
current queries used for reporting
15

16. “How many sections are using the LMS
(out of all sections offered that term)?”
CSU_09 671 2,191
CSU_08 1,098 1,162
CSU_06 2,997 7,064
Active Sections
Inactive Sections
CSU_05 2,492 3,687
CSU_04 553 614
CSU_02 2,270 3,911
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
16

17. “How many sections are using the LMS
(out of all sections offered that term)?”
CSU_09 671 2,191
CSU_08 1,098 1,162
CSU_06 2,997 Use = “visible”+”student activity”
7,064
Active Sections
Inactive Sections
CSU_05 2,492 3,687
CSU_04 553
Use = “visible”
614
CSU_02 2,270 3,911
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
17

18. Round 2: mCURL
(Moodle Common Usage and Learning Analytics)
§ 8 active CSU & 2 UC campuses
– Co-chaired: John Whitmer, CO ATS and
Mike Haskell, Cal Poly SLO
§ Starting with same measures of adoption,
prioritizing “wish list” of more advanced analytics
§ Local database conventions and campus
practices make accurate comps. challenging
18

19. Faculty LMS Adoption
How many faculty are using the LMS in one or
more course sections?
19

20. mCURL Next Steps
§ Refine queries for accurate comparative course
and student adoption measures
§ Select additional queries: depth and breadth of
use
– # tools used
– # students in each section
– frequency of use
§ Create repositories for campuses to share
unique local queries 20

21. Blackboard Analytics for Learn (A4L)
§ CSU ATS Co-Lab Agreement – working together
– Functionality: from early alerts/course reporting
to institutional-level analytics
– Up to 4 campuses participating (3 confirmed)
– Period: December 2012-December 2013
– Individual campus Scope of Work for setup of
infrastructure and services
§ Kick-off meeting next week
21

22. Co-Lab Goals
1. Develop methodologies and processes to identify, aggregate,
and transform LMS usage data into information for analytics.
2. Improve campus usage of learning analytics for decision-making
for student success, curriculum improvement, and technical
services.
3. Create shared measures, database reports, and algorithms,
drawing on campus best practices and research innovations.
4. Increase campus awareness of applications and technical tools.
5. Document campus efforts and disseminate to other campuses.
6. Provide professional development in learning analytics.
22

23. Student at a Glance
23

24. Instructor at a Glance
24

25. Dean Dashboard
25

26. Feedback? Questions?
Kathy Fernandes
(kfernandes@csuchico.edu)
John Whitmer
(jwhitmer@calstate.edu)
26