«Small Interventions to Support Student Self-regulation Online ». René Kizilcec

1. Small Interventions to Support
Student Self-regulation Online
René Kizilcec
Cornell University

2. Demand for online learning
is increasing globally
Supply is also increasing,
providing more access to
non-traditional populations
Goodman et al. 2018
Lower costs per student
Deming et al., 2015; Chirikov et al., 2020
Harder for learners to stay on
track and achieve their goal

3. Dynarski, 2018

4. Bettinger & Loeb, 2017
DeVry University (large for-profit; 100k+ students)

5. “While all types of students in the study suffered decrements
in performance in online courses, those with the strongest
declines were males, younger students, Black students, and
students with lower grade point averages.”
Xu & Jaggars, 2017
AtWashington Community andTechnicalColleges:
Challenges and Inequities in Online Education

6. Participation and Achievement Gaps in the US and Globally

7. 25 million MOOC enrollments
39% from non-OECD members
Visualization by Max Roser. Data source: Robert Barro & Jong-Wha Lee (www.barrolee.com).
Years of
Education:
ACCESS STUDENT SUCCESS
Interventions to Provide
Scalable Support,
Inclusive Environment

8. Behavioral Science Interventions
• Intervene on mechanisms that contribute to social problems
Thaler & Sunstein, 2008;Walton &Wilson, 2018
• Targeting pro-social behaviors: educational achievement, healthy eating, physical
activity, getting medical check-ups, voting, etc.
• Nudges: changes in choice architecture
• Self-regulation interventions: e.g. plan making, goal setting, monitoring
• “Wise” interventions: changes in interpretation of self/environment
• E.g. Mindset, Grit, Social Belonging, Utility-value,Values Affirmation, Purpose
• Technology lowers barriers for scaling up interventions

9. Self-Regulated Learning Strategies
Pintrich, 2000; also Zimmerman, 2000; Winne & Hadwin, 1998
Cognitive
Support acquisition, storage, retrieval of information: rehearsal,
organization, elaboration.
Metacognitive
Plan, self-monitor, self-evaluate at various stages of the learning
process.
Resource management
Help seeking and selecting optimal environment for learning.

10. Self-Regulated Learning Strategies
Kizilcec, Pérez-Sanagustin & Maldnado, 2016a, 2016b
Which SRL strategies are effective in MOOCs?
• Correlate self-report SRL with success (N = 4,800)
• Goal setting & planning predict completion
Does merely recommending SRL strategies help?
• Experiment to give feedback (N = 569)
• Perceived as “Very helpful”; but no actual improvement

11. Intervention to Improve Self-regulation
• Randomized experiments in 3 MOOCs (total N = 2,053)
• Brief activity at the beginning of the course
• Expected to improve completion rates for all learners
Plan-Making Intervention
Nickerson & Rogers, 2010; Milkman et al., 2011; 2013 etc.
Yeomans & Reich, LAK 2017

12. Yeomans & Reich (2017),
Planning Prompts Increase
and Forecast Course
Completion in Massive Open
Online Courses Proceedings
of the 2017 Learning
Analytics and Knowledge
Conference, Vancouver,
Canada.
Pre-Registration at
https://osf.io/mky8n/

13. Planning Prompts Increase Completion
Yeomans & Reich, 2017
13.8%
17.7%
Three HarvardX
courses in Business,
Chemistry, and Political
Science
n=2,053

14. Another Intervention to Improve Self-regulation
• Randomized experiments in two MOOCs (N = 18,000)
• 8-minute activity at beginning of course
• Test for cultural differences in effectiveness
Mental Contrasting
Oettingen, 1999
Implementation Intentions
Gollwitzer, 1999
Kizilcec & Cohen, PNAS 2017

15. Balanced Individualist
Cultures
onal Collectivism−Individualism
ental Condition: Control MCII
Collectivist, balanced countries:
|z|s < 0.4, ps > 0.7
32% increase in completion
in individualist countries
z = 2.35, p = 0.019
Kizilcec & Cohen, PNAS 2017

16. All materials available at osf.io/9bacu

17. Interventions in the Scale Up Study
Self-regulation and Wise Interventions
• Long-term Plan-making
• Short-term (1st week) Plan-making
• Mental Contrasting with Implementation Intentions
• Social Accountability
• Value Relevance Intervention*
* Wise intervention

18. Testing Interventions at Scale
Common Guidelines for NSF and IES:
“examines effectiveness in a wide range of populations, contexts, and
circumstances, without substantial developer involvement in
implementation or evaluation.” Earle et al. 2013
Open Science: methods, analyses, results, pre-analysis plans posted online
Research question
1. To what extent will the original (medium to large) effects replicate at scale?
2. What new insights can we glean about where the interventions work?

19. Implementation
Sep 2016 May 2019
Preregistration
Data Collection
Hypothesis Testing
Exploration
Preregistration
Data Collection
Hypothesis Testing
Exploration
Preregistration
Data Collection
Hypothesis Testing
Exploration
Preregistration
Data Collection
Hypothesis Testing
Exploration
Wave 1 Wave 2 Wave 3 Wave 4
First Year Second Year
Coordinate between 3 institutions: Harvard, MIT, Stanford
Evaluated in 247 courses, shown to 269,169 students

20. Open Science Approach
- Posting study materials (i.e. course surveys)
- Pre-registration for each wave with exact analysis script
- Posting raw analysis output for each wave
- Explain any changes to design, analysis, hypotheses in next pre-
registration
- Posting (most) data for reproducibility and extensions
https://osf.io/9bacu/

21. Primary
Expected
Effects at the
start
Planning helps fluent English speakers

22. Primary
Expected
Effects at the
start
Planning helps fluent English speakers
No evidence for our original large- to medium-
effects on completion across interventions
New hypotheses from exploratory data analysis
tended not to be supported in next wave

23. Short-lived
Effect of the
Self-Regulation
Intervention

24. The Social Accountability Intervention
Now is the time to think about who can hold you accountable.
1. Write down the names of one or more friends, co-workers, family
members, acquaintances who could hold you accountable.
2. Plan what you are going to tell them about the course and your goal.
3. Write down how and when you will tell them about this.
Inspired by Hall et al., 2017; Rogers et al., 2018

25. Interested in testing your
interventions in higher ed?
New research infrastructure
Reach out
kizilcec@cornell.edu

26. Small Interventions to Support
Student Self-regulation Online
René Kizilcec @whynotyet
Cornell University
Thank you!