This last spring, I wrote a research proposal for a ph. d. position in "behavioral measurements", focusing on understanding international differences in mathematics understanding as measured by TIMSS and PISA. Recently, I've seen a renewed interest in TIMSS results online, as for example in Michael Pershan's video critique of Khan Academy.
So I'm thinking if I post part of the research proposal here, maybe people will find the "research overview" part interesting and relevant to the times. Parts of the proposal are about Sweden, but from what I understand much is highly relevant for the US as well. Sorry for the sketchy formatting which happened when I copypasted from MsWord.
Oh, and I did get that ph. d. position, it fit me like a glove and I happily accepted. Unfortunately it would have required me to relocate to a different town, and recent family developments made relocation currently impossible. Oh well  I'll always think of this position as "the one that got away."

Specific Objectives and Aims
The overarching aim of this
project is to use existing international TIMSS data to understand the factors
that influence the quality of mathematics education in Swedish schools. Ever since the first international
comparisons of mathematics knowledge in middleschool students, Sweden has
positioned itself at or below the average score of participating nations (Hellerstedt, 2011). Between TIMSS 1995 and TIMSS 2003, the results of Swedish 8^{th}
grade students decreased by 41 points, which is more than any other country
among the 16 that participated in both 1995 and 2003 (Skolverket, 2004), and
then decreased even further by TIMSS 2007 (Skolverket 2008). By contrast, other nations, such as our neighbors
Finland and Russia, have shown consistently higher results in international
comparisons.
Such differences between nations
deserve attention because they signify that mathematics education can be more effective
than is the case currently in Sweden. By identifying the causes behind the
relative successes of highperforming nations, Sweden might be able to emulate
them and thus achieve more efficient use of school finances as well as a more
mathematically literate population. However, what works in one country may not
work in another cultural and economic context. It is therefore necessary to take into
consideration factors that affect mathematics education within Sweden, as well
between Sweden and other nations. While
TIMSS tries to be curriculumneutral, so that it can be applied to all nations,
it can be argued that the mathematics knowledge measures by TIMSS does not
constitute mathematics knowledge in its entirety, that the questions target
only specific aspects of mathematics knowledge such as specific subject areas
or skills. In order to enable research
of these international differences, TIMSS and PISA are accompanied by indepth
data regarding the questions in the test, as well as a wide range of detailed contextual
data about variables at the student, teacher and school levels of the
participating nations.
The three main
objectives of this proposal are:

Overview of the Research Area
The achievement of the above stated objectives
will be made possible by a close analysis of relevant parts of the large amount
of data collected in the TIMSS mathematics reports. This data includes results
for each participating nation on different types of questions in the different
areas of mathematics tested in TIMSS.
Also included is contextual data such as statistics on student, teacher,
school and curriculum variables in each participating nation. Next, we
shall see several research studies that to varying degrees, and with different
aims, make use such contextual data.
One of the most relevant studies
regarding international differences in TIMSS mathematics results is the TIMSS
Videotape Classroom Study (Stigler, 1999a) which was created together with the
1995 TIMSS mathematics study (Beaton, 1996).
Stigler used video recording in order to compare instructional practices
in 8^{th} grade mathematics lessons in Germany, Japan, and the United
States. In a large sample of in total
281 classrooms, chosen to be representative of classrooms in each country, one
lesson per year was randomly chosen and filmed.
Results show that, among many other differences, Japanese classrooms
include more complex problemsolving tasks and higher difficulty mathematical
content than do their German and United States counterparts. In a popular description of this study and
its findings, Stigler and Hiebert claim that it is such differences in
instructional practices which influence some international differences in
mathematics knowledge (Stigler, 1999b).
A related study points in a somewhat
different direction. Leung (2005)
analyzed the data from the larger scale TIMSS video study that was made in
conjunction with the TIMSS 1999 mathematics study (Mullis, 2000). In this larger and more recent video study, 7
countries (Australia, Czech Republic, Japan, Hong Kong, the United States, the
Netherlands, and Switzerland) were included with a total of 638 videotaped
lessons. When comparing the East Asian nations to other nations, Leung observed
that East Asian nations provide students with learning opportunities involving
complex mathematical problems often featuring high level mathematical
difficulty and logical reasoning such as proofs. However, the two East Asian
nations differed from each other in that Hong Kong classrooms are highly
teacherdirected, while Japanese classrooms very much less so. Leung concludes that East Asian classrooms
are highly heterogeneous, and that the success of East Asian nations in
international comparisons must be understood as resulting from interactions of
cultural factors such as perceptions of education and high expectations in the
classroom. These video studies show that although there are differences in
instructional factors between highperforming and lowperforming nations, not
all such differences are causal factors of mathematics knowledge. Also highperforming nations may have some
common and some different strategies to ensure high levels of mathematics
knowledge.
While the video studies have yielded
much valuable data about instructional practices, other research has focused on
psychosocial differences such as attitudes towards oneself in relation to
mathematics. Shen (2008) aimed to
investigate the relationship between selfperception in mathematics and TIMSS
results in 8^{th} graders in the 1995, 1999, and 2003 TIMSS
studies. Using statistical analysis of
relevant TIMSS contextual data, Shen found that within each country, there is a
positive correlation between mathematics results and perceived competence in
mathematics, how much the student likes mathematics, and how easy the student
perceived mathematics to be. Between
countries, however, the relationship is reversed such that the students in the
highest performing nations are the ones who report liking mathematics less,
judge it to be difficult, and have low opinions of their competencies in
mathematics. Shen attributes this surprising relationship to higher academic
standards in highperforming nations, and lower standards in lowerperforming
nations.
Findings such as those in Shen’s study
can be questioned on methodological grounds, since the assumption is that the
surveys used in the collection of TIMSS contextual data are valid for all
participating nations. Eklöf (2007) challenges this assumption by conducting
indepth analysis of the Swedish TIMSS data on mathematics selfconcept and
students valuing of mathematics. While the former of these is shown to be
consistent and correlated to mathematical achievement, this was not true of the
latter. Eklöf’s research, among others,
illustrates the need for careful statistical investigation of the scales used
in the TIMSS contextual data before using the contextual data for secondary
analysis. Eklöf and Shen show that TIMSS data must be analyzed for both
betweencountries and withincountry differences if we wish to understand the
factors that influence mathematics knowledge.
Another area of research into student
variables is illustrated by Boe (2002).
Boe investigated whether student task persistence (a variable not
included in the TIMSS contextual data), as measured by the percent of
background questionnaire questions students completed, was related to
mathematics results on the TIMSS 1995 test. The results indicated a
surprisingly strong correlation (ranging from 0.72 to 0.79 for 7^{th}
and 8^{th} grade students) between task persistence and mathematics
results on a betweennations level of analysis. The relationship between task persistence and
results appeared much smaller at the classroom and student levels, however, and
in total task persistence accounted for about 0.28 of the total variation
between students participating in the TIMSS 1995. Such findings are difficult to interpret.
First, we do not know whether the strong correlations indicate any causal
relationships between the variables. There could be a third factor on a cultural level, such as ability
to delay gratification (Mischel, 1989, shows a moderate correlation between
delayed gratification and SAT scores), or testtaking motivation (Eklöf, 2006,
finds a weak but significant correlation in the Swedish TIMSS 2003 sample),
that causes both increased task persistence and higher mathematical
achievements on the TIMSS tests. Also,
it is strange that the correlation is smaller on the student level than on the
national level. From Boe’s study, it is clear that TIMSS contextual data demands
analysis beyond just looking for correlations, and that the relationships found
require careful interpretation in terms of findings from cultural and
psychological research.
Of special interest for this research
proposal is the research that has been done on mathematics knowledge in Nordic
countries in general, and Sweden in particular. Kjaernsli (2002) investigates
similarities and differences between the Nordic countries, excluding Finland,
and finds that their results on the TIMSS science and mathematics tests are similar
and may be connected to cultural factors such as the reluctance to put academic
pressure on young children. Finland, by
contrast, has seen a dramatic rise in mathematics results as measured by TIMSS
and PISA since 1999. It is of great international interest to determine the
factors behind Finland’s success, and recently much research has been made with
this aim. Välijärvi (2003) aims to present a broad look at factors influencing
Finland’s rise to success. Välijärvi
identifies factors such as educational equity in comprehensive shools, cultural
homogeneity, and highly educated teachers.
Interestingly, Välijärvi also points out that some factors seem to be
more important in Finland than in other OECD nations. The withincountry correlation found by Shen
(2008) between selfperception and mathematics achievement is significantly
higher in Finland than elsewhere. Research such as Välijärvi’s further illustrates
the need for both betweennation and withinnation investigations of factors
influencing mathematics achievement.
TIMSS aims to establish the success of
mathematics education in terms of how well students achieve the educational
goals formulated by their own nations, whether at the state or local levels. It
is therefore very important to investigate to what extent TIMSS questions are
aligned with the Swedish curricula, both in terms of the subject matters
covered (geometry, algebra, etc.) and the skills which students are meant to
develop (reasoning, application of procedures, etc.). However because of the
loosely formulated goals in the governmentissued curriculum documents, we
should be wary of using those documents to understand the implemented curricula
in the Swedish schools. Instead, it makes better sense to analyze teacher
responses about their intentions and expectations within their implemented
curricula (Skolverket, 2004).
Teacher responses to TIMSS
questionnaires indicate that students have received relatively more instruction
in arithmetic and measurement, and less in algebra and geometry, compared with
students in other nations as well as compared with the proportions that each
subject area has in the TIMSS examinations (Skolveket, 2004). Also, Alger (2007) finds that Swedish
teachers compared to teachers in other nations report using a larger proportion
of class time on independent practice with mathematics exercises, and less time
going over homework and lecturing.
Lindström (2006) considered differences between Swedish national tests
(though he used an old test from 1992) and TIMSS 2003 and PISA 2003. He found
that the exercises are about equal in difficulty level, but the Swedish test
had much less emphasis on reasoning and on applications, and more emphasis on
identifying and carrying out procedures.
One major limitation of Lindström’s study is that the Swedish national
tests have changed considerably since 1992. However, Lindström’s results find
support in a more recent indepth analysis (Skolverket, 2009) of student
responses to the TIMSS 2008 Advanced and the Population 3 responses to TIMSS
1995. This analysis is based on the
patterns of correct and incorrect solutions, and frequent mistakes, made by
Swedish students and reveals that Swedish upper secondary school teachers since
1995 have increasingly focused on procedural knowledge rather than conceptual
and reasoning based understanding of mathematics. Together, these studies indicate that there are variables at the
instruction level that affect Swedish students’ results on the TIMSS
assessments.
On a school level, there are other
important variables identified in the responses from teachers and principals. Overall
Swedish students in grades 4 and 8 receive substantially fewer instructional
hours compared to the OECD average, this difference in grade 8 is approximately
25% and is even larger (closer to 40%) in grade 4. In addition, few Swedish
students receive extra mathematics instruction outside of school and report
much less frequent homework and less time spent on homework compared with OECD
averages (Skolveket, 2004). Thus, it is reasonable to assume that
schoollevel factors also play a role in determining students’ mathematical
knowledge.
Research questions
In light of the background research
presented above, the main research question in this research proposal is:
Research question:
To what extent do school and teacherlevel variables influence Sweden’s
mathematics results on international tests?
In order to investigate
the main research question, it is necessary to consider several related
questions:
1.
What types of
mathematical knowledge is measured by TIMSS?
2.
Are the TIMSS
measurements reliable and valid?
3. What statistical
analysis methods are relevant for studying TIMSS secondary data in search of
potential causal relationships?
4.
To what extent are the
aforementioned statistical methods valid?
Potential
causal factors can be divided into several categories:
5.
What instructional
factors influence mathematics achievement within Sweden?
6.
What instructional
factors influence mathematics achievement differences between Sweden and other
nations?
References
Alger, S. (2007).
Svenska skolans lärare och
undervisning i matematik och NO i ett internationellt perspektiv: Några resultat från TIMSS 2003.
(BVM 32:2007). Umeå: Umeå universitet, Department of Educational
Measurement.
Beaton,
A., Martin, M. O., Mullis, I., Gonzalez, E. J., Smith, T. A., & Kelley, D.
L. (1996). Mathematics
Achievement in the Middle
School Years: IEA's Third International Mathematics and Science
Study. Chestnut Hill,
MA: Boston College.
Boe, E. E., May, H., & Boruch, R. F. (2002). Student task persistence in the Third International Mathematics and
Science Study: A major source of achievement differences at the national,
classroom, and student levels (Research Rep. No. 2002TIMSS1).
Philadelphia, PA: University of Pennsylvania, Graduate School of Education,
Center for Research and Evaluation in Social Policy.
Eklöf, H. (2006). Motivational
Beliefs in the TIMSS 2003 Context. Theory, Measurement and Relation to Test
Performance. Doctoral dissertation.
Umeå: Umeå University, Department of Educational Measurement.
Eklöf, H. (2007). Selfconcept and valuing of mathematics in TIMSS
2003: Scale structure and relation to performance in a Swedish setting.
Scandinavian Journal of Educational Research, 51(3), 297313.
Hellerstedt, L., & Reistad, H. (2012, January 22). Matteutveckling för miljarder. Origo, 1.
Retrieved March 22, 2012, from
http://www.lararnasnyheter.se/origo/2012/01/22/matteutvecklingmiljarder
Leung, F. (2005). Some characteristics of East Asian mathematics
classrooms based on data from the TIMSS 1999 video study. Educational Studies in Mathematics, 60, 199215.
Lindström, J. (2006). Med
fokus på matematik och naturvetenskap: en analys av skillnader mellan
internationella jämförande studier och nationella kursplaner. Stockholm:
Skolverket.
Mischel, W., Shoda, Y., & Rodriguez, M. I. (1989). Delay of gratification in children. Science,
244, 933938.
Mullis, I.V.S., Martin, M.O., Gonzalez, E.J., Gregory, K.D., Garden,
R.A., O’Connor, K.M., Chrostowski, S.J. & Smith, T.A. (2000) TIMSS 1999 International Mathematics Report:
Findings from IEA’s Repeat of the Third International Mathematics and Science
Study at the Eighth Grade. Chestnut
Hill, MA: Boston College.
Nyström, P.
(2006). TIMSS fixpunkter. En analys av
vad elever med olika resultat i TIMSS
2003 vet och kan göra. (BVM 20:2006). Umeå: Umeå universitet, Department of
Educational Measurement.
Nyström, P., & Lind, A.
(2009). Hur samstämmiga är svenska styrdokument och nationella prov med
ramverk och uppgifter i TIMSS Advanced 2008. Stockholm: Skolverket.
Reinikainen, P. (2004). Explanatory variables of
science achievement in Finland: Cultural approach. In C. Papanastasiou (Ed.), Proceedings of the IRC2004. IEA
International Research Conference (Vols. 1–4).
Kjaernsli, M., & Lie, S.
(2002). Timss science results seen from a nordic perspective. In D. Robitaille
& A. Beaton (Eds.), Secondary Analysis of the TIMSS Data (pp. 193208). New York: Kluwer
Academic Publishers.
Shen, C., & Tam, H. P. (2008). The
paradoxical relationship between student achievement and selfperception: A
crossnational analysis based on three waves of TIMSS data. Educational Research and Evaluation, 14, 87–100.
Skolverket.
(1996). TIMSS 1995: Svenska 13åringars kunskaper i matematik och
naturvetenskap i ett internationellt perspektiv. Stockholm: Skolverket.
Skolverket.
(2004). TIMSS 2003: Svenska elevers kunskaper i matematik och naturvetenskap
i skolår 8 i ett nationellt och internationellt perspektiv. Stockholm: Skolverket.
Skolverket.
(2008). TIMSS 2007: Svenska grundskoleelevers kunskaper i matematik och
naturvetenskap i ett internationellt perspektiv. Stockholm: Skolverket.
Skolverket.
(2009). Svenska elevers kunskaper i
TIMSS Advanced 2008 och 1995: en djupanalys av hur eleverna i gymnasieskolan
förstår centrala begrepp inom matematiken. Stockholm: Skolverket.
Stigler, J. W. (1999a). The TIMSS Videotape Classroom Study: methods
and findings from an exploratory research project on eighthgrade mathematics
instruction in Germany, Japan, and the United States. Washington, D.C.:
U.S. Dept. of Education, Office of Educational Research and Improvement.
Stigler, J. W., &
Hiebert, J. (1999b). The teaching gap: best ideas from the world's
teachers for improving education in the classroom. New York: Free Press.
Välijärvi, J., Linnakylä, J., P., Kupari, P., Reinikainen, P., &
Arffman, I. (2002). The Finnish success in PISA  and some reasons behind
it: PISA 2000. Jyväskylä: Institute for Educational Research, University of
Jyväskylä.
No comments:
Post a Comment