Presentation delivered at the San Diego State University "One Day in May" conference on May 22, 201 by John Whitmer, Hillary Kaplowitz, and Thomas J. Norman Universities archive massive amounts of data about students and their activities. Students also generate significant amounts of “digital exhaust” as they use academic technologies. How can faculty and administrators use automated analysis of this data to save time and conduct targeted interventions to improve student learning? The emerging discipline of Learner Analytics conducts analysis of this data to learn about student behaviors, predict students at-risk of failure, and identify potential interventions to help those students. In this presentation, we will discuss the contours of this discipline and review the state of research conducted to date. We will then look at several examples of Learner Analytics services and hear from California State University educators who are using these tools to help their students. Finally, we will suggest some immediate ways that Analytics can be conducted at San Diego State. Presenters: John Whitmer, California State University, Chico Hillary Kaplowitz, California State University, Northridge Thomas J. Norman, CSU Dominguez Hills

1. Learner Analytics and the “Big Data”
Promise for Course & Program
Assessment
San Diego State University “Day in May”
22 May 2012
John Whitmer, CSU Chico (& Office of the Chancellor)
Hillary Kaplowitz, CSU Northridge Download slides at:
Thomas Norman, CSU Dominguez Hills http://bit.ly/Kb6gsV

2. Outline
1. Promise of Learner Analytics
2. Case Studies
a) Analytics at work in the classroom (Hillary)
b) Improving classroom discussion and mastery of
program level outcomes (Thomas)
c) Evaluating course redesign (John)
3. Analytics Tools @ SDSU
4. Q & A

3. 1. PROMISE OF LEARNER ANALYTICS

4. John Goodlad’s Place-Based Research
 Classroom-based
research: “What is
schooling?”
 1,000 classrooms,
27,000 individuals
 14 foundations needed to
support
 Fundamental changes to
understanding of
educational practice

5. Steve Lohr, NY Times, August 5, 2009

6. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire
economy. The Economist.

7. Current GPA: 3.3
First in family to
attend college
SAT Score: 877
Hasn’t taken college-
level math
No declared major
Source: jisc_infonet @ Flickr.com
7
http://slidesha.re/IgKSTX
Source: jisc_infonet @ Flickr.com

8. Academic Analytics
“Academic Analytics marries large data sets with
statistical techniques and predictive
modeling to improve decision making”
(Campbell and Oblinger 2007, p. 3)
8

9. DD Screenshot
9

10. Learner Analytics
“ ... measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs.” (Siemens, 2011)

11. Fundamental Questions behind
Learner Analytics
1. What are students doing (or not doing)?
Which students are we talking about?
2. Does it matter (re: achievement,
engagement, learning)?
3. What should we do?
– Changes in student behavior?
– Changes in faculty/program?

12. SIGNALS
Purdue Signals Project http://www.itap.purdue.edu/studio/signals/

13. Wordcloud of student evaluations of Course Signals (Arnold, 2010)

14. Signals Course Outcomes
 Fall 2009
 Compare same course (w/Signals v. w/o Signals)
-6.41% DWF
+10.97% A/B
(Arnold, 2010)

15. KHAN
Khan Academy http://www.khanacademy.org/

17. Source: wpclipart.com

18. Will Learner
Analytics
replace
educators with
computer
By Peter Nowak, Macleans.ca
algorithms?

19. “Robots are
grading your
papers!”
-Marc Bousquet,
4/18/2012 Blog post,
Chronicle of Higher
Education

20. Contrasting State-of-the-Art Automated
Scoring of Essays: Analysis
 Mark D. Shermis, University of Akron (funded by
Hewlett Foundation)
 Compared 8 robo-graders to human grading on
standardized essay questions from 6 states
 Outcome: very small difference in results
(.03 - .12 score difference)
 Conclusion: are scoring algorithms sophisticated ….
or are standardized essays simplistic … or do we
need to stop the dichotomy of computers and
people?

21. -- Tom Vander Ark, Gettingsmart.com

22. Or analytics can support faculty by …
1. Providing behavioral data to investigate student
performance
2. Informing faculty about students succeeding or
at risk of failing a course
3. Warning students that they are likely to fail a
course – before it’s too late
4. Helping faculty evaluate the effectiveness of
practices and course designs
5. Customizing content and learning activities

23. 3. CSU CASE STUDIES

24. How can data help teachers
and students work better
together?
Hillary Kaplowitz
Instructional Designer, Faculty Technology Center
Part-Time Faculty, Cinema and Television Arts
Department
California State University, Northridge

25. Case #1
“I'm not upset that you lied to
me, I'm upset that from now on
I can't believe you.”
Friedrich Nietzsche

28. “Hey Professor,
I just looked at my assignments and
realized that my Chapter 11 summary
did not get submitted, which I'm having
trouble believing that I didn't submit it...
especially because I see that I did it,
and I always submit my assignments
as soon as I finish them.”

29. Now the hard part….
Do I believe him?
If I only I could check…

32. And it was all his idea…
The student suggested that I check Moodle and if
that didn’t work told me how to check the Revision
History in GoogleDocs with step-by-step
directions!

34. Case #2
“Life isn't fair. It's just fairer
than death, that's all.”
William Golding

35. “The quiz is unfair”

36. Hybrid Course Weekly Structure
4. Post
3. Online questions
1. Watch 2. Read 5. Class 6. Aplia
chat and and take
lectures textbook meets quiz
tutoring practice
quiz

37. But the story was not that simple…
• Reports on Moodle painted a different picture
• Student was watching the lectures at 10:00 p.m.
• Then immediately taking quiz

38. Enabled constructive feedback…
 Advised the student how the structure of the
course was designed to enhance learning
 Student revised their study habits
 Improved grades and thanked the instructor!

39. What we can do with data now
 Use Reports in Moodle to verify student claims
 Review participant list to see last access time
 Empower students to review their own reports
 Analyze usage and advise students how to study better
 Review quiz results to find common misconceptions

40. Could we help improve student learning
outcomes if we knew the effect of…
Coffee
Facebook Sequencing
Attendance Amount
Mobile Textbook
LMS LMS
Activities Access

41. Using Learner Analytics to Improve
Classroom Discussion and Mastery
of Program Level Outcomes
Thomas J. Norman, Ph. D.
California State University,
Dominguez Hills
tnorman@csudh.edu

42. Solving the Student Effort Challenge
• Prior surveys revealed that a majority of
Management students were reading 5 chapters
or less of the assigned 15 chapters
• The course average on the
cumulative final was around 70%
• Using online assessments has boosted these
scores 7-8 percentage points!
• These are tools made available by McGraw Hill
and Aplia that you can use too:
– McGraw Hill $39- $99 with eBook
– Cengage Aplia $99 with eBook

43. Benefits of Online Assignments
• Assignment are due Sunday
at 11:45 or 11:59 p.m.
• They ensure students have
read and begun working
with the concepts BEFORE
classroom discussion and
activities
• Provides immediate
feedback
• Automatically graded

44. Aplia Real Time Metrics
Progress and Mastery
At risk

45. LearnSmart Tale of 3 Students
1
2
3
Student 1 warned to keep up, ignored warning and failed course
Student 2 knew material, completed homework in 6 hours A student
Student 3 struggled early, but caught up and did well A- student

46. Analysis by AACSB Categories

47. Performance by Learning
Objective/Difficulty
Why do my students do better at
medium difficulty questions?
Not working!
This is working!

48. EVALUATING COURSE REDESIGN:
INTRO TO RELIGIOUS STUDIES 180
48

49. LMS Learner Analytics @ Chico State
Evaluation for Program Assessment
– Academy e-Learning course redesign
– Intro to Religious Studies: increased enrollment from
80 to 327 students first semester
– Outcome: increased mastery course concepts AND
increased number D/W/F students
– Why? (and for whom? And what did they do?)
– What is the relationship between LMS actions, student
background characteristics and student academic
achievement? (6 million dollar question)
49

50. Grade Distribution
50

51. Grades by LMS “Dwell Time”
51

52. Grades by Dwell & Tool

54. 3. ANALYTICS TOOLS @ SDSU

55. Bboard Learn Tools

56. BLACKBOARD LEARN: Activity Stats
Reports

57. Evaluation: Course Reports

58. Performance Dashboard

59. Evaluation: Early Warning System

60. Grade Center: “Smart Views”

61. Student View: “Report Card”

62. Call to Action
1. You’re *not* behind the curve, this is a rapidly
emerging area that we can (should) lead ...
2. Metrics reporting is the foundation for Analytics
3. Don’t need to wait for student characteristics
and detailed database information; LMS data
can provide significant insights
4. If there’s any ed tech software folks in the
audience, please help us with better reporting!

63. Draft DOE Report
released April 12
http://1.usa.gov/GDFpnI

64. Q&A and Contact Info
Resources Googledoc: http://bit.ly/HrG6Dm
Contact Info:
• John Whitmer (jwhitmer@csuchico.edu)
• Hillary C Kaplowitz (hillary.kaplowitz@csun.edu)
• Thomas Norman (tnorman@csudh.edu)
Download presentation at:
http://bit.ly/Kb6gsV
64

65. Works Cited
Adams, B., Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching Learning through
Educational Data Mining and Learning Analytics: An Issue Brief. Washington, D.C.: U.S. Department of
Education, Office of Educational Technology.
Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1).
Bousquet, M. (2012). Robots Are Grading your Papers. Retrieved from
http://chronicle.com/blogs/brainstorm/robots-are-grading-your-papers/45833
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era.
EDUCAUSE Review, 42(4), 17.
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and
perhaps the entire economy. The Economist.
LaValle, S., Hopkins, M., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The new path to
value. Findings from the 2010 New Intelligent Enterprise Global Executive Study and Research Project:
IBM Institute for Business Value and MIT Sloan Management Review.
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data:
The next frontier for innovation, competition, and productivity.
Parry, M. (Producer). (2012, 5/14/2012). Me.edu: Debating the Coming Personalization of Higher Ed.
Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/me-edu-
debating-the-coming-personalization-of-higher-ed/36057
Siemens, G. (2011, 8/5). Learning and Academic Analytics. Retrieved from
http://www.learninganalytics.net/
65