gsat csec and intervening assessments---policy without numbers?

1. A visual analysis of
relationships between GSAT
Grade 7 & 9 Diagnostic and
CSEC performance in
secondary schools in Jamaica
2008-2011
JTA Education Conference 2012
Robert Philips, Education Specialist
e-Learning Jamaica Company
April 2012
www.e-ljam.net

2. Background
• The e-Learning Jamaica Project is a collaboration
between the Ministry of Science Technology Energy
and Mining and the Ministry of Education to
systematically strengthen inputs to instruction in
high schools.
• Information and communications technologies and
instructor capabilities are being enhanced, with a
view to increasing success of students at the CSEC
level

3. Glossary
• GSAT-- the national terminal • GNDT—Grade 9 diagnostic
examination for the primary test—an examination
curriculum serving as the designed by CXC to provide
placement tool for secondary planning and intervention
school
information to schools and
• PSLE—the equivalent exam in individual students, adopted
Singapore by the MOE for national
• Communication task—the two implementation
part writing component of the
GSAT examination • All-Age Schools—schools
• CXC—Caribbean with some secondary grades
Examinations Council— in addition to full primary
regional authority for school • Primary and Junior High—
examinations another type of school with
• CSEC—CXC exams typically secondary grades in addition
done at the end of 11th grade to full primary
providing a basis for college
matriculation

4. Project Components
• Infrastructure
• Materials
• Assessment
• Remediation
• Training

5. Organisation
• Context
• Vision
• Practice
• Results
• Analysis
• Lessons learned
• Sustainability strategies

6. Context
• GOOGLE search on
“GSAT ANXIETY”
has 79,400 hits
• Search on
“PSLE ANXIETY”
has 46,600 hits

7. Vision
If you want to improve a system over
time you will get surprises if you
don’t have a tracking process

8. Practice
• Ministry of Education agreed to a
National 9th grade Diagnostic
implemented by e-Ljam
• Caribbean Examinations Council contracted to
organise and deliver
– End-of-year multiple choice test with
– Continuous internal assessment
– Voluntary 7th grade assessment

9. The roots of school performance
r=0.99 !!!!
Math Math Math 2009 vs Math 2008 GSAT
School Name 2008 2009
Aabuthnott 120
Gallimore
High 58.88 53.13
100
Math GSAT 2009
80
Albert Town 60
High 46.73 44.73
40
20
Annotto Bay 53.96 50.62
0
Ardenne High 0 20 40 60 80 100 120
School 94.39 93.63 Math GSAT 2008
Ascot High 48.52 45.29

10. A school perception index ?
Math 2009 vs Math 2008 GSAT
• Intake GSAT Math means for two
consecutive years for high schools
120
100 • Explanation invited
80 • Submission that school perception
60 as indexed by selection and/or
placement is the main determinant
Math GSAT 2009
40 of future performance
20 • Visual anomalies are technical
schools which became normal
0 clients of GSAT in 2009
0 20 40 60 80 100 120
• Source :MOE data
Math GSAT 2008

11. A school perception index ?
Math 2009 vs Math 2008 GSAT
120 • Intake GSAT scores are
100 technically independent of
80
the internals of schools
60
Math GSAT 2009
• Schools with lowest intake
40
GSAT scores have lowest
20 levels of academic success
0 • A new rationale for
0 20 40 60 80 100 120 placement may be required
Math GSAT 2008

12. Results of Assessment Component
Growth in participation
2 7000 pilot grade 9 30 schools
3 30000 national grade 9 150 schools
4 64000 grade 9 and grade 7 173 schools *
5 82000 projected 180 schools
*10 Benchmark All-Age and Primary Jnr High included in 2011

13. Scanning Technology
• Examview
• Use scanners supplied by
project
• Local, decentralised solution
• Automatic Document feed
• Powerful reporting suite
• Replaced CXC registration
strategy and MOE
centralized scanning
• 7 day turnaround

14. Performance Trends 2010
• 10% of schools don’t participate
• 11% of students in participating schools
miss exams
• 81% of participants score 45% or
greater on multiple choice test for
English
• 38% of participants score 45% or
greater on Maths test

15. 2010---Math—level 1
• Fairly good performance is seen on statistics
items, with an average of 70%.
• Many students missed items on Number &
Computation such as: order of operations (70%
of students), place value (5-digits (60%),
decimal – hundredths (70%)) (easily Grades 6-7
objectives).
• Students answered correctly, on average, only
one-third of the items on Measurement &
Geometry and Algebra.
• Source Faulkner,D. CXC Exam Consultant
•

16. 2010—math level 1
• Females students performed
better than male students in all
sub-topics except in
Measurement and Geometry.
• The average difference is 2%.
• The largest differences are in
Statistics (7.6%) and Consumer
Arithmetic (4.1%)
• The difference in favour of
males in Measurement and
Geometry is 0.5%.
• 36.4% of female students and
31.6 % of male students scored
45% or more.
• Source Faulkner,D. CXC Exam
Consultant

17. Math 2010—level 2
• All sub-topics, except Algebra have
mean percent scores above 40%.
• Statistics, Consumer Arithmetic and
Relations & Graphs have the
highest mean percent scores.
• Algebra has the lowest scores on
this paper.
• Students need more work in the
following areas: Number and
Computation, Measurement and
Geometry and Algebra.
• Source Faulkner,D. CXC Exam Consultant

18. Select schools 2010
• School A has been closed

19. Select schools 2010
• School B labelled as failing

20. Select schools 2010
School X a classic outlier GSAT scores >95 or +3 standard deviations

21. Select schools 2010
• Ecological validity of the
assessment system
tested using six Catholic
high schools
• English > Math

22. Select schools 2010
• Ecological validity of the
assessment system
tested using six Catholic
high schools
• “posh” vs “poor” *
x,c vs a,b
* after Miller

23. Select schools 2010
• Ecological validity of the
assessment system
tested using six Catholic
high schools
• girls vs boys
y,c vs z

24. Summary results
Grade 9
2011
• Available cohort 41946
• Patricipation 37362
• Scoring less than 45% English 8816
• National Means
– Math 37.06
– English 54.99
– Integrated Science 44.65
– Social Studies 51.53

25. Summary results
Grade 7--2011
• Available cohort est. 45000
• Patricipation 27000
• Scoring less than 45% English 6800
• Scoring less than 45% Math 11900
• National Means
– Math 45.7
– English 55.8
– Integrated Science 50.6
– Social Studies 50.5

26. Grade 9 Math vs English 2011
how schools math and language performance covary at grade 9
90
80
• 140+ of 173 70
schools
test mean for school
60
50
•
math_9_2011
Gap in raw scores 40
eng_9_2011
lowest for outliers 30
20
10
0
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113 120 127 134
school no.

27. GSAT scores 2008 vs Grade 9 2011
• Same cohort after 3
years
• GSAT means of 80
give ca 100%
probability of ninth
grade success for
English
• Math performance
would be interesting to
track

28. English GSAT vs Math GSAT 2010
• Between school Chart Title y = 0.9262x + 5.2538
means of English
2
R = 0.9935
100
and Math 90
80
abnormally linear 70
english gsat 2010
60
egs2010
50
• Explanation invited 40
30
Linear (egs2010)
• Is reading the 20
10
common factor? 0
0 20 40 60 80 100 120
math gsat 2010
• Is there a variable
called test taking?

29. GSAT means 2010 vs Gseven 2011
y = 0.7231x + 10.42
• Low contribution of
gsat 2010 vs gseven 2011 R2 = 0.9327
instruction to the
90.0
80.0
variance after a year in
70.0 high school
60.0
gseven 2011
50.0
40.0
• R (g6 vs g7) > .95
30.0
20.0
10.0
0.0
0.0 20.0 40.0 60.0 80.0 100.0 120.0
• Sources: MOE 2010 GSAT results and e-
gsat 2010 Ljam grade 7 exams 2011

30. GSAT communication task
• Currently the only production test
in GSAT
• Divides schools into two quadrants
• Communication task of 7 and
below associated with lower overall
performance
• Planned for replacement by a book
report
• Sources: MOE 2010 GSAT results and
e-Ljam grade 7 exams 2011

31. GSAT communication task
• Currently the only production test
in GSAT
• Divides schools into two quadrants
• Communication task of 7 and
below associated with lower overall
performance
• Planned for replacement by a book
report
• Sources: MOE 2010 GSAT results and e-Ljam
grade 7 exams 2011

32. GSAT communication task
one year later
• Urgent review of the future of
the communication task is
recommended
• No evidence that book reports
are going to represent
students work given the
behavior with CSEC SBAs
• Sources: MOE 2010 GSAT results
and e-Ljam grade 7 exams 2011

33. Indices of CSEC performance
• Bill Johnson—Ralph Thompson
–Gleaner 2020 index
• Adopted by e-Ljam ,with some
reservations as a meaningful
index
• Simply – school score is 100 if
every student in cohort scores a
Grade 1
• Grade 1 is valued at 4
• Grade 2 is valued at 2
• Grade 3 is valued at 1

34. Tracking variables for six select
schools
tracking variables for selected schools
80
70
60
mgs2008
egs2008
• GSAT
50
math_9_2011
• GNINE
percent or index for gsat
or gnine or csec
40
eng_9_2011
30
20
10
e10
m10
• CSEC english
0
o1 o2 b1 h1 s1 t1
avg_gsat
av_g9
• CSEC math
school

35. Visual analysis—grade 7--2011
schools by comm task score
120
mgs2010
100 egs2010
gsat gseven cxc index
scigs2010
80 socgs2010
math_7_2011
eng_7_2011
60
sci_7_2011
soc_7_2011
40
eng-jt_2007
math-jt_2007
20
e09
m09
0 e10
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 m10
school

36. Visual analysis Grade 7--2011
• CSEC indices m 10—math 2010 e 10---- english 2010
• GSAT mgs 2010 egs 2010
• Grade 7 –math_7_2011 eng_7_2011

37. Visual analysis
• Partially cross-sectional
2010 GSAT vs CSEC
• partially longitudinal
2010—2011 GSAT vs G7
• Intake GSAT scores trend
similarly to CSEC indices
• Floor CSEC results for more
than half the schools
• GSAT placement reinforces
a culture of failure in the
majority

38. Conclusions
• GSAT intake means are a measure of the perception of schools
• GSAT means forecast Grade 7 Grade 9 and CSEC performance with the robustness of physical science measures
• Maths and English gap is evident from grade 7
• Geometry and algebra weaknesses are evident
Contrary to the view that GSAT is too hard, there are not enough moderate to high performance students to share among
the 170+ schools
• Students scoring less than 80 could be placed on geography and some of those above could receive advanced
placement and seek to complete high school in four years
• The GSAT communication task should be reviewed for retention as independent writing is encouraged
• The communication task discriminates and is associated with varying levels of terminal secondary performance
• Data is available to inform policy and strategy choices
• An education system that does not acknowledge the foundations of literacy will have low levels of success

39. Lessons learned
• Jamaican teachers are eager to incorporate
technology into their professional practice,
including assessment
• The consistent placement of weaker students
in the same schools reinforces the culture of
failure
• Assessment at ninth grade indicates that
even at current levels of primary school
output quality, higher levels of success in
CSEC Math and English are achievable

40. Sustainability
• The Ministry of Education has
contracted a supplier for the strategic
overhaul of the lower secondary
curriculum and related assessment
processes
• E-Learning Jamaica has been
requested to liaise with the supplier

41. Question Time
Robert Philips, Education Specialist
www.e-ljam.net