This study evaluated the relationship between student use of a Learning Management System (LMS), student characteristics, and academic achievement in a large undergraduate hybrid course. The results showed: 1. LMS use, particularly assessment activities, had a stronger correlation with course grades than student characteristics like high school GPA. 2. A regression model combining LMS usage data and student characteristics moderately predicted student success, suggesting refinement could strengthen results. 3. LMS data required extensive filtering to be useful for analysis, and student variables needed screening for missing data. The conclusions were that LMS behavioral data may better predict outcomes than demographics alone, and combining LMS and student data in a complete model relates to academic achievement

1. Logging On To Improve Achievement
Evaluating the relationship between use of the Learning
Management System, student characteristics, and academic
achievement in a hybrid large enrollment undergraduate course
Research Highlights: Presentation to SoLAR Storm
November 15, 2012
John C Whitmer (jwhitmer@calstate.edu)
Committee Chair: Dr. Paul Porter, Sonoma State University
Slides:
http://slidesha.re/sFKjcm

2. Introduction
• Educational Doctorate
Degree (EdD) candidate
(University of California
Davis & Sonoma State
University)
• Advanced to candidacy,
defending ~ January 14
• Associate Director,
California State University
LMSS Project,
Chancellor’s Office

3. Presentation Outline
1. Study Case & Context
2. Results for Instructional Practices
3. Results for LMS Data Analysis
4. Conclusions

4. STUDY CASE &
CONTEXT

5. Problem: Student Graduation
• Less than 50% of college/university students graduate
within 6 years
• California State University: 52.4%
(first-time freshman, 2000 cohort)
(CSU Analytic Studies, 2011)
• Students from under-represented minority racial/ethnic
groups graduate at much lower rates
• California State University: 38.3%
(African American students, first-time freshman, 2000 cohort)
(CSU Analytic Studies, 2011)
• Contributing factor: mega-enrollment intro courses
• Infrequent interaction, prevent faculty/student relationships

6. Case: Introduction to Religious Studies
• Redesigned to hybrid 54 F’s
delivery through Academy
eLearning
• Highest LMS usage entire
campus Fall 2010
(>250k hits)
• 373 students (54% increase)
• Bimodal results
• 10% increased SLO mastery
• 7-11% increase in DWF

7. Research Questions
1) Is there a relationship between student LMS usage and academic
performance? Does this relationship vary by the pedagogical
purpose underlying LMS usage? (correlation)
2) Is there a relationship between student background characteristics
or current enrollment information and academic performance?
(correlation)
3) Does analyzing combined student characteristics and current
enrollment information increase the predictive relationship between
combined LMS usage data and student success?
(multivariate regression)
4) Does a student’s economic status and student of color status
vary the predictive relationship between combined LMS usage,
combined background characteristics and current enrollment
information? (multivariate regression, restricted model)

8. Independent Variables: Student Characteristics
1. Gender
2. Under-Represented Minority
3. Pell-Eligible
4. High School GPA
5. First in Family to Attend College
6. Major-College
7. Enrollment Status
Under-Represented Minority and Pell-Eligible
8. (interaction)
Under-Represented Minority and Gender
9. (interaction)

9. Independent Variables: LMS Usage
# LMS Usage Category LMS Tools within
Category
1. Administration Activity Hits Announcement
Calendar
2. Assessment Activity Hits Assessment
Assignments
My-grades
3. Content Activity Hits Content-page
Web-links
4. Engagement Activity Hits Discussion
Mail

10. Research Methods (Cliff’s notes version)
1. Extract data, validate with appropriate “owner”
2. Transform variables
• measures of interest (e.g. “URM”, not race/ethnicity)
• analysis methods (categorical into numeric)
3. Examine data for
• outliers, missing data, data distributions, etc.
• colinearity between variables (e.g. independence)
4. Join data into single data file, collapse to one record/student
5. Run analysis

11. Results for Instructional
Practices

12. Correlation: LMS Usage w/Final Grade
Scatterplot of
Assessment Activity Hits v.
Course Grade

13. Correlation: Student Char. w/Final Grade

14. Most interesting finding (so far):
Smallest Largest
>
LMS Use Variable Student Characteristic
(Administrative Activities) (HS GPA)
r=0.3459 r=0.3055

15. Regression R2 Results Comparison

16. RESULTS FOR LMS
DATA ANALYSIS

17. Lms Logfiles: “Data Exhaust”
1. Logfile tracks server actions
(not educationally relevant activity)
2. Duplicate logfile hits for single student action
3. To remedy, filtered logfiles by:
• Time (> 5sec, <3600 sec)
• Actions (no “index views”, more)

18. Logfile Data Filtering Results
450
382 400
350
300
250
200
151
150
58 100
54 51 49 36
23 26 16 50
0
Final data set: 72,000 records
(from 250K+)
Filtered
Measure Raw Avg Avg Reduction
Discussion Activity Hits 382 54 706%
Content Activity Hits 151 51 296%
Assessment Activity Hits 58 23 249%
Mail Activity Hits 49 36 136%
Administrative Activity Hits 26 16 159%

19. LMS Use Consistent across Categories
Factor Analysis of LMS Use Categories

20. Missing Data On Critical Indicators

21. Conclusions
1. At the course level, LMS use better predictor of
academic achievement than any student characteristic
variable. Behavioral data appears to supercede
demographic information (what do, not who are).
2. Moderate strength magnitude of complete model
demonstrates relevance of data, but suggests that
refinement of methods could produce stronger results.
3. LMS data requires extensive filtering to be useful;
student variables need pre-screening for missing data.

22. Ideas & Feedback
Potential for improved LMS analysis methods:
• social learning
• activity patterns
• discourse content analysis
• time series analysis
Group students by broader identity, with unique variables:
• Continuing student (Current college GPA, URM, etc.
• First-time freshman (HS GPA, SAT/Act, etc)

23. Contact Info
John Whitmer
jwhitmer@calstate.edu
Skype: john.whitmer
USA Phone: 530.554.1528
By WingedWolf
Damián Navas