This document discusses methodological innovation in mathematics education research. It begins by introducing the speaker, Dr. Christian Bokhove, and the aim of exploring innovative research methods that can help answer questions in mathematics education. The document then summarizes several of Dr. Bokhove's methodological approaches, including using social network analysis to study classroom interaction and teacher trainee networks, computational analysis of text data from school inspections, and sequence analysis of student log file data from online math tools. Key takeaways emphasized interpreting results, using multiple complementary methods, and moving toward more open and transparent scientific practices.

1. METHODOLOGICAL INNOVATION
FOR MATHEMATICS EDUCATION
RESEARCH
21 October 2021
Maths Education SIG at UCL Institute of Education
Dr Christian Bokhove

2. Who am I?
• Dr Christian Bokhove
• From 1998-2012 teacher mathematics & computer science, head of ICT
secondary school Netherlands
• PhD Utrecht University
‘Use of ICT for acquiring, practicing and assessing algebraic expertise’
• Associate Professor at University of
Southampton
• Mathematics education
• Technology use
• Large-scale assessment (PISA/TIMSS)
• Research methods
• Editorial team Research in Mathematics Education

3. Aim of the talk
• Innovative research methods can help us in answering
research questions for mathematics education.
• Innovation – “a new idea, way of doing something, etc.
that has been introduced or discovered” – maybe not that
grand….standing on the shoulders of giants….
• This talk wants to bring together some methodological
approaches I’ve been using.
• The rationale behind them has always been:
• What is the research question?
• What methodological tools are available to ‘best’ answer
the question?

4. • Classroom interaction
• Communication networks of trainees
• Networks and social creativity
Questions
• Computational Social Science
• Log file data
• Open Science
Questions

5. CLASSROOM INTERACTION

6. SNA for classroom interaction
• Case to use SNA for
classroom interaction
• Making it dynamic
• Classroom interaction
(Moody, McFarland,
& Bender-deMoll, 2005)
• Technological and methodological advances
• Observation apps
• Video recording easier
• Statistical techniques and packages to capture temporal aspects
like Gephi, ERGMs, Rsiena, Statnet, Relevent

7. This project
• Use dynamic social network analysis to describe
classroom interaction
• Data analysis and visualization software
• Gephi 0.8.2 beta
• R and Rstudio with the packages statnet (Handcock, Hunter, Butts,
Goodreau, & Morris, 2008) and ndtv (Bender-deMoll, 2014)

8. Observation apps

9. Data analyses
• Three data analyses approaches
• A: transcripts of TIMSS used ‘as is’ because low effort with existing
transcripts  Gephi
• B: TIMSS videos re-observed
to get more detail  Gephi, Rstudio (statnet and ndtv)
• C: Observations of maths lessons in a secondary school in the
south of the United Kingdom  Using Lesson App, Gephi (incl.
animations)

10. Analysis (US1 only) - ndtv

11. 11

12. Metrics over time
12

13. What might it tell us?
• Teacher student interaction
• Whole class, directionality
• Student interactions
• Groups and cliques
• Individual behaviour
• Help seeking
• Disturbances
• Central students
• Perhaps, patterns over classes, schools, countries
(analogue TIMSS video study, TALIS video/Global Insights)

14. Methodological take-aways
• Observational methods – historically move quantitative to
more qualitative.
• Bridging the approaches with analytical and visual tools
for Social Network Analysis.
• Methods can complement each other.
Bokhove, C. (2018). Exploring classroom interaction with dynamic social
network analysis. International Journal of Research & Method in
Education, 41(1), 17-37.

15. NETWORKS –
TEACHER TRAINEES

16. Context
• Teacher training in UK
• PGCE
– University Led (UL)
– School Direct (SD)
→ NQT
• Secondary Maths and Science
– cohort size (~35)
– R&R – “sink or swim”
– longevity of course

17. Support networks
• Instrumental
– developing teaching
strategies
• Expressive
– friendship

18. Data collection
Time Network Related factors
Peer
(whole)
External
(ego)
Trust Self-
efficacy
1    
2   
3    
4    

19. Network development
j
i
j
i
(Snijders et al , 2010)
RQ1: How do the peer communication networks of pre-
service maths and science teachers develop over time?
Network change

20. Network change – maths T1 & T4
15

21. 21
RQ1: Network structure - triadic closure
Only Maths - not observed among science trainees.

22. RSiena Results
Mathematics T1-T2 T2-T3 T3-T4
outdegree (density) - - -
Reciprocity + + +
transitive triplets + +
transitive recipr. triplets - -
Friend + +
Strategies + +
SD alter + +

23. Methodological take-aways
• Methods for analysing networks.
• There are many different approaches – this also can be
rather confusing!
Bokhove, C., & Downey, C. (2018). Mapping changes in support: a
longitudinal analysis of networks of pre-service mathematics and
science teachers. Oxford Review of Education, 44(3), 383-402.
Brouwer, J., Downey, C., & Bokhove, C. (2020). The development of
communication networks of pre-service teachers on a school-led and
university-led programme of initial teacher education in
England. International Journal of Educational Research, 100, 101542.

24. NETWORKS - CREATIVITY

25. Networks - creativity
Work with Marios Xenos and Manolis Mavrikis in the FP7
MC-Squared project
• Social creativity in communities.
• Communities of interest can foster social creativity
through socio-technical environments.
• How could creativity be measured in online discussion
environments?
• Modelling communities as social networks
Teachers/designers created digital mathematics books and
recorded the design process.

27. Operationalising social creativity with SNA
Fluency – ‘quantity’

28. Flexibility – ‘breadth’
Average out: 0.857
Average out
corrected:
1.20
Modularity: 2
Average out: 0.967
Average out
corrected:
4.83
Modularity: 5

29. Originality – ‘quality’ - self report votes
Elaboration – ‘depth’
Average path length: 1.667
Network diameter: 3
Average path length: 3.163
Network diameter: 8

30. Key nodes or ideas – degree and betweenness centrality

33. Methodological take-aways
• It is possible to use ideas from one discipline in original
contexts.
• Interdisciplinarity.
• Challenge: acceptance of ideas – we have and are
struggling to get these ideas ‘out there’.
Bokhove, C., Xenos, M., & Mavrikis, M. (forever under review). Using
Social Network Analysis to gain insight into social creativity while
designing digital mathematics books .

34. QUESTIONS/DISCUSSION
Interim – network examples

35. COMPUTATIONAL SOCIAL
SCIENCE

36. Computational research methods(*)
Approach that relies on forms of automated analysis of information,
using computers, to answer education research questions.
The methods can include one or more of the following:
• Analysis depends on algorithms, including the use of
• Artificial intelligence (AI) - computers make complex, human-like judgements
• Machine Learning (ML) - computers learn to copy human behaviour
• Data sets are usually large scale, 'Big Data', sometimes millions of
sources are collected and analysed.
• Information already exists, rather than collected specifically for
research.
• 'Scraping' from websites (news, reports, blogs, etc)
• Extraction from databases and archives created for other purposes (eg journal
contents, interactions with a learning platform)
• Social networks (e.g. social media)
• Simulating new data
(*) terminology. 18 October 2021. ‘Educational Data Science’
https://journals.sagepub.com/doi/full/10.1177/23328584211052055

38. For example, Bokhove
(2015) scraped thousands of
OFSTED reports from the
inspection website to answer
the question whether topics
and sentiments in the
reports had changed over
time, so-called ‘sentiment
analysis.
Bokhove, C. (2015). Text mining school inspection reports in England with R. University of
Southampton.

39. Bokhove, C., & Sims, S. (2020). Demonstrating the potential of text mining for analyzing school inspection
reports: a sentiment analysis of 17,000 Ofsted documents. International Journal of Research and Method in
Education. https://doi.org/10.1080/1743727X.2020.1819228

41. Analytical approach
• 3155 documents, classified by judgement
• Outstanding
• Good
• Requiring Improvement
• Satisfactory
• Inadequate
• Lower case, stemming, remove stopwords, remove
punctuation, remove numbers
• “Your corpus now has 3155 documents, 1435 terms and
1767508 tokens.”
• Judgement as covariate.
• Age as covariate.

43. http://www.telegraph.co.uk/education/educationnews/10489675/Reintroduce-
traditional-textbooks-in-schools-minister-says.html

44. Evaluation
• Interpretation is difficult
Table 2: frequencies for the corpora

45. Other examples
• Munoz-Najar Galvez et al. (2019) used text analysis to
study the paradigm wars in graduate research in the field
of education.
• Topic modelling by Inglis and Foster (2018) with the
package MALLET, to study evidence of the ‘social turn’ in
five decades of mathematics education research.
• Marks et al. (2020) analysed all 813 Proceedings of the
British Society for Research into Learning Mathematics
from 2003 to 2018, first using a quantitative corpus-survey
and qualitative thematic coding and, again, independently,
using topic modelling.

46. Methodological take-aways
• Automated text extraction of corpora can provide insight
into social phenomena.
• This also is part of computational social science –
quantitative and qualitative merge more and more.
• Interpretation can be very challenging.
• Different methods should work ‘in tandem’.
• Mathematics testbooks?
Bokhove, C., & Sims, S. (2021). Demonstrating the potential of
text mining for analyzing school inspection reports: a sentiment
analysis of 17,000 Ofsted documents. International Journal of
Research & Method in Education, 44(4), 433-445.

47. LOG FILE DATA

48. Log file data from online maths tools
• Intelligent Tutoring Systems
• Look at trajectories…
• All records of school users in the UK in Years 3-5 who
have at least 100 lesson records in academic year ’18-’19
(N=1799). Number of records ranged from 100 to 4970
entries, totalling 1048575 records from between
December 2010 and January 2019.
• Log file included id, difficulty, questiontime, number of
questions, totalscore, totalhelp, timings, ‘run mode’
• Derived: range, precision, intensity
• Sequence analysis.

49. Analytical approac
• TraMineR is a R-package for mining, describing and
visualizing sequences of states or events, and more
generally discrete sequence data.
• Typology through clustering.
• Discrepancy analysis. With TraMineR, analyze the
discrepancy between sequences and visualize the results.
• Regression tree.

51. Regression tree

52. • Totalhelp and precision, as well as difficulty to a lesser
extent) work together. More help extends the sequence
and extends precision, but up to a point. Furthermore, for
too difficult questions (bottom left in the diagram) help
does not increase the sequence.
• So, well-tailored help helps the student along without
compromising difficulty.
• ‘Sweet spot’ goldilocks?
UCL colleagues from the London Knowledge Lab know
even more about these things…

53. Methodological take-aways
• Software packages generate a lot of data in the form of
log-files.
• Although log-files are well-structured, assumptions re
what ‘clicks’ represent are made.
• However, even these can give insights into learning
patterns and behaviour.
• Interpretation can be very challenging.
• Different methods should work ‘in tandem’.
Bokhove, C. (in preparation). Learning behaviour in an online maths
environment: The role of help-seeking.

54. OPEN SCIENCE

55. Open Science Collaboration (2015). Science, 349.

56. Neuroskeptic (2012). Perspect Psychol Sci, 7, 643-644.
https://journals.sagepub.com/doi/full/10.1177/1745691612459519
(i) Limbo
(ii) Overselling
(iii) Post-hoc storytelling
(iv) P-value fishing
(v) Creative outliers
(vi) Plagiarism
(viii) Non-publication
(viii) Partial publication
(ix) Falsification

57. Carp (2012). Neuroimage, 63, 289-300.
“…nearly as many unique
analysis pipelines as there
were studies in the sample…”

58. So….

59. Publication bias
• Techniques to analyse this as part of metareviews
• Used in review of the metacognition literature
• P-curve analysis: http://www.p-curve.com/
• Funnel plots

60. Open Science Framework
• Data sharing
• Open materials
• Pre-registration (aspredicted.org, OSF)

61. ICT literacy
• Pre-registration at OSF (Open
Science Framework)
• Code published from raw data to
result tables.
• Replicates the study from a 2014
article in Computers & Education
with ICILS data (an international
computer literacy assessment by
the IEA)
• Analytical flexibility determines
much.

62. Methodological take-aways
• Direction of travel is towards open and transparent
science.
• Still…many challenges along the way.
• Also…what does it mean for qual/quant?
• It takes time and mental flexibility to change; culture
change even more (e.g. pre-publications).
• Overall, though, we are hopefully improving science…
• Lead UKRN local network at Southampton, UCL also has
one!
Bokhove, C. (in press). The role of analytical variability in secondary data
replications: a replication of Kim et al. (2014).

63. CONCLUSION

64. Drawing this together
• Quantitative and qualitative methods start to overlap more
and more though computational developments.
• Many, many different tools and analytical approaches –
it’s hard to keep up.
• Interpretation, interpretation, interpretation.
• Tensions with ‘traditional’ approaches.
• But it is exciting and fun to explore ‘new’ paths…
• Countdown
• Direction towards ‘open science’:
• Assumptions
• Analytical choices
• Open data and analysis scripts

65. QUESTIONS/DISCUSSION
C.Bokhove@soton.ac.uk
Twitter: @cbokhove