In this presentation to NYU-Learn, I discuss my experience applying data science and machine learning in educational technology and assessment industries. I share tips for thinking about the importance of context and potential of scalability.

1. INTEGRATING RESEARCH &
PRODUCTION LEARNING
ANALYTICS
TIPS LEARNED FROM COMMERCIAL & PUBLIC
SECTOR EXPERIENCES
March 16, 2021| NYU-Learn “Scaling Up”
John Whitmer, Ed.D. (jwhitmer@fas.org | @johncwhitmer)
Senior Fellow, Federation of American Scientists
This work is licensed under a Creative
Commons Attribution 4.0 International
License.
Slides: http://bit.ly/ScalingLA-Whitmer

2. Situating my professional perspective
2

3. IES Data
Science
Fellowship
Goals
3
Implement Data Science
into IES Operations
Expand Data Access
(External & Internal)
Support Development of
IES Data Science
Fellowship Program

4. 4
What’s distinct about
Learning Analytics?

5. 5
Centrality of interventions
Image Credit: Deposit Photos

6. Consider different levels of context relevant to:
1. Early identification of students at-risk of quitting a
class (do we need to know why?)
2. Helping EdTech designers understand if their
"hypotheses" are true.
3. Improve quality of discussion forum feedback from
instructors.
4. Providing students with tips to improve their writing.
5. Identifying the reason an 8th-grade student is
struggling to learn math concepts.
6
Image credit: universetoday.com

7. Low context example: Tool use | Grade outcomes
Source: https://blog.blackboard.com/how-successful-students-use-lms-tools/
Sample Bb. Learn : 601k learners, 18.8k courses, ~927 institutions
Content Grades

8. Medium Context: Linguistic Analysis of Forum Posts
From: https://www.insidehighered.com/news/2017/11/14/
professors-have-mixed-reactions-blackboard-plan-offer-tool-grading-online
From: https://help.blackboard.com/Learn/Instructor/
Interact/Discussions/View_Discussion_Performance

9. Strategy
Tips
1. Understand leadership goals
What is the anticipated value for analytics solutions? These
solutions are expensive (compared to conventional
software development or doing nothing)
2. Organizations are complex
They rarely operate as you anticipate. They are historical,
culturally driven, and rarely operate by explicit rules
3. People will assume you're the expert
Good experts are aware of their limits and ruthlessly
skeptical.

10. • Do you consider yourself an applied researcher?
(Ph.D. / post-doc and beyond)
• Do you have strong tech skills with education data?
(esp. clickstream, NLP, data normalization)
• Are you willing to work on research questions defined
by other people (you lead the methods and approach)?
• Are you patient and willing to help build new practices,
technologies, protocols?
• Are you interested in a position as a term-limited Fellow or
staff position working with U.S. federal data?
If you responded ”yes” to all, ping me with a CV/resume
(jwhitmer@fas.org)
10
Image credit:
Homesteaders of America

11. Thank you!
11
John Whitmer
jwhitmer@fas.org | @johncwhitmer
Slides: Slides: http://bit.ly/ScalingLA-Whitmer

12. Two truths
& a lie
Leadership is a necessary
(but not sufficient) condition
for successful scaling up
Analytics software
development is NOT
different from other software
development
EdTech business leaders
are only in it for the money
Vote for which statement
is the lie using chat
12

13. Methods and processes have better scalability
Source: https://actnext.org/research-and-projects/learning-analytics-constructs-behind-the-clicks/
(broken as of 3/1/2021)