TIMSS research plan

This last spring, I wrote a research proposal for a ph. d. position in "behavioral measurements", focusing on understanding international differences in mathematics understanding as measured by TIMSS and PISA. Recently, I've seen a renewed interest in TIMSS results online, as for example in Michael Pershan's video critique of Khan Academy.

So I'm thinking if I post part of the research proposal here, maybe people will find the "research overview" part interesting and relevant to the times. Parts of the proposal are about Sweden, but from what I understand much is highly relevant for the US as well.  Sorry for the sketchy formatting which happened when I copy-pasted from MsWord. 

Oh, and I did get that ph. d. position, it fit me like a glove and I happily accepted. Unfortunately it would have required me to relocate to a different town, and recent family developments made relocation currently impossible. Oh well - I'll always think of this position as "the one that got away."

---------------------------------------------------------------------------------------------

Specific Objectives and Aims

The overarching aim of this project is to use existing international TIMSS data to understand the factors that influence the quality of mathematics education in Swedish schools.  Ever since the first international comparisons of mathematics knowledge in middle-school students, Sweden has positioned itself at or below the average score of participating nations (Hellerstedt, 2011). Between TIMSS 1995 and TIMSS 2003, the results of Swedish 8^th grade students decreased by 41 points, which is more than any other country among the 16 that participated in both 1995 and 2003 (Skolverket, 2004), and then decreased even further by TIMSS 2007 (Skolverket 2008).  By contrast, other nations, such as our neighbors Finland and Russia, have shown consistently higher results in international comparisons.  

Such differences between nations deserve attention because they signify that mathematics education can be more effective than is the case currently in Sweden. By identifying the causes behind the relative successes of high-performing nations, Sweden might be able to emulate them and thus achieve more efficient use of school finances as well as a more mathematically literate population. However, what works in one country may not work in another cultural and economic context.  It is therefore necessary to take into consideration factors that affect mathematics education within Sweden, as well between Sweden and other nations.  While TIMSS tries to be curriculum-neutral, so that it can be applied to all nations, it can be argued that the mathematics knowledge measures by TIMSS does not constitute mathematics knowledge in its entirety, that the questions target only specific aspects of mathematics knowledge such as specific subject areas or skills.  In order to enable research of these international differences, TIMSS and PISA are accompanied by in-depth data regarding the questions in the test, as well as a wide range of detailed contextual data about variables at the student, teacher and school levels of the participating nations.

The three main objectives of this proposal are:

To identify what aspects of mathematics knowledge are targeted by the TIMSS questions and how they are related to the aspects of mathematics knowledge valued in Sweden To analyze TIMSS contextual data to determine in what relevant ways Sweden differs from nations with higher TIMSS results To analyze TIMSS contextual data to determine what factors cause differences in mathematical knowledge within Sweden

Overview of the Research Area

The achievement of the above stated objectives will be made possible by a close analysis of relevant parts of the large amount of data collected in the TIMSS mathematics reports. This data includes results for each participating nation on different types of questions in the different areas of mathematics tested in TIMSS.  Also included is contextual data such as statistics on student, teacher, school and curriculum variables in each participating nation.   Next, we shall see several research studies that to varying degrees, and with different aims, make use such contextual data.

One of the most relevant studies regarding international differences in TIMSS mathematics results is the TIMSS Videotape Classroom Study (Stigler, 1999a) which was created together with the 1995 TIMSS mathematics study (Beaton, 1996).  Stigler used video recording in order to compare instructional practices in 8^th grade mathematics lessons in Germany, Japan, and the United States.  In a large sample of in total 281 classrooms, chosen to be representative of classrooms in each country, one lesson per year was randomly chosen and filmed.  Results show that, among many other differences, Japanese classrooms include more complex problem-solving tasks and higher difficulty mathematical content than do their German and United States counterparts.  In a popular description of this study and its findings, Stigler and Hiebert claim that it is such differences in instructional practices which influence some international differences in mathematics knowledge (Stigler, 1999b).

A related study points in a somewhat different direction.  Leung (2005) analyzed the data from the larger scale TIMSS video study that was made in conjunction with the TIMSS 1999 mathematics study (Mullis, 2000).  In this larger and more recent video study, 7 countries (Australia, Czech Republic, Japan, Hong Kong, the United States, the Netherlands, and Switzerland) were included with a total of 638 videotaped lessons. When comparing the East Asian nations to other nations, Leung observed that East Asian nations provide students with learning opportunities involving complex mathematical problems often featuring high level mathematical difficulty and logical reasoning such as proofs. However, the two East Asian nations differed from each other in that Hong Kong classrooms are highly teacher-directed, while Japanese classrooms very much less so.  Leung concludes that East Asian classrooms are highly heterogeneous, and that the success of East Asian nations in international comparisons must be understood as resulting from interactions of cultural factors such as perceptions of education and high expectations in the classroom.  These video studies show that although there are differences in instructional factors between high-performing and low-performing nations, not all such differences are causal factors of mathematics knowledge.  Also high-performing nations may have some common and some different strategies to ensure high levels of mathematics knowledge. 

While the video studies have yielded much valuable data about instructional practices, other research has focused on psycho-social differences such as attitudes towards oneself in relation to mathematics.  Shen (2008) aimed to investigate the relationship between self-perception in mathematics and TIMSS results in 8^th graders in the 1995, 1999, and 2003 TIMSS studies.  Using statistical analysis of relevant TIMSS contextual data, Shen found that within each country, there is a positive correlation between mathematics results and perceived competence in mathematics, how much the student likes mathematics, and how easy the student perceived mathematics to be.  Between countries, however, the relationship is reversed such that the students in the highest performing nations are the ones who report liking mathematics less, judge it to be difficult, and have low opinions of their competencies in mathematics. Shen attributes this surprising relationship to higher academic standards in high-performing nations, and lower standards in lower-performing nations.

Findings such as those in Shen’s study can be questioned on methodological grounds, since the assumption is that the surveys used in the collection of TIMSS contextual data are valid for all participating nations. Eklöf (2007) challenges this assumption by conducting in-depth analysis of the Swedish TIMSS data on mathematics self-concept and students valuing of mathematics. While the former of these is shown to be consistent and correlated to mathematical achievement, this was not true of the latter.  Eklöf’s research, among others, illustrates the need for careful statistical investigation of the scales used in the TIMSS contextual data before using the contextual data for secondary analysis.  Eklöf and Shen show that TIMSS data must be analyzed for both between-countries and within-country differences if we wish to understand the factors that influence mathematics knowledge.

Another area of research into student variables is illustrated by Boe (2002).  Boe investigated whether student task persistence (a variable not included in the TIMSS contextual data), as measured by the percent of background questionnaire questions students completed, was related to mathematics results on the TIMSS 1995 test. The results indicated a surprisingly strong correlation (ranging from 0.72 to 0.79 for 7^th and 8^th grade students) between task persistence and mathematics results on a between-nations level of analysis.  The relationship between task persistence and results appeared much smaller at the classroom and student levels, however, and in total task persistence accounted for about 0.28 of the total variation between students participating in the TIMSS 1995.  Such findings are difficult to interpret. First, we do not know whether the strong correlations indicate any causal relationships between the variables. There could be a third  factor on a cultural level, such as ability to delay gratification (Mischel, 1989, shows a moderate correlation between delayed gratification and SAT scores), or test-taking motivation (Eklöf, 2006, finds a weak but significant correlation in the Swedish TIMSS 2003 sample), that causes both increased task persistence and higher mathematical achievements on the TIMSS tests.  Also, it is strange that the correlation is smaller on the student level than on the national level.  From Boe’s study, it is clear that TIMSS contextual data demands analysis beyond just looking for correlations, and that the relationships found require careful interpretation in terms of findings from cultural and psychological research.

Of special interest for this research proposal is the research that has been done on mathematics knowledge in Nordic countries in general, and Sweden in particular. Kjaernsli (2002) investigates similarities and differences between the Nordic countries, excluding Finland, and finds that their results on the TIMSS science and mathematics tests are similar and may be connected to cultural factors such as the reluctance to put academic pressure on young children.  Finland, by contrast, has seen a dramatic rise in mathematics results as measured by TIMSS and PISA since 1999. It is of great international interest to determine the factors behind Finland’s success, and recently much research has been made with this aim. Välijärvi (2003) aims to present a broad look at factors influencing Finland’s rise to success.  Välijärvi identifies factors such as educational equity in comprehensive shools, cultural homogeneity, and highly educated teachers.  Interestingly, Välijärvi also points out that some factors seem to be more important in Finland than in other OECD nations.  The within-country correlation found by Shen (2008) between self-perception and mathematics achievement is significantly higher in Finland than elsewhere.  Research such as Välijärvi’s further illustrates the need for both between-nation and within-nation investigations of factors influencing mathematics achievement.

TIMSS aims to establish the success of mathematics education in terms of how well students achieve the educational goals formulated by their own nations, whether at the state or local levels. It is therefore very important to investigate to what extent TIMSS questions are aligned with the Swedish curricula, both in terms of the subject matters covered (geometry, algebra, etc.) and the skills which students are meant to develop (reasoning, application of procedures, etc.). However because of the loosely formulated goals in the government-issued curriculum documents, we should be wary of using those documents to understand the implemented curricula in the Swedish schools. Instead, it makes better sense to analyze teacher responses about their intentions and expectations within their implemented curricula (Skolverket, 2004).  

Teacher responses to TIMSS questionnaires indicate that students have received relatively more instruction in arithmetic and measurement, and less in algebra and geometry, compared with students in other nations as well as compared with the proportions that each subject area has in the TIMSS examinations (Skolveket, 2004).  Also, Alger (2007) finds that Swedish teachers compared to teachers in other nations report using a larger proportion of class time on independent practice with mathematics exercises, and less time going over homework and lecturing.  Lindström (2006) considered differences between Swedish national tests (though he used an old test from 1992) and TIMSS 2003 and PISA 2003. He found that the exercises are about equal in difficulty level, but the Swedish test had much less emphasis on reasoning and on applications, and more emphasis on identifying and carrying out procedures.  One major limitation of Lindström’s study is that the Swedish national tests have changed considerably since 1992. However, Lindström’s results find support in a more recent in-depth analysis (Skolverket, 2009) of student responses to the TIMSS 2008 Advanced and the Population 3 responses to TIMSS 1995.  This analysis is based on the patterns of correct and incorrect solutions, and frequent mistakes, made by Swedish students and reveals that Swedish upper secondary school teachers since 1995 have increasingly focused on procedural knowledge rather than conceptual and reasoning based understanding of mathematics. Together, these studies indicate that there are variables at the instruction level that affect Swedish students’ results on the TIMSS assessments.

On a school level, there are other important variables identified in the responses from teachers and principals. Overall Swedish students in grades 4 and 8 receive substantially fewer instructional hours compared to the OECD average, this difference in grade 8 is approximately 25% and is even larger (closer to 40%) in grade 4. In addition, few Swedish students receive extra mathematics instruction outside of school and report much less frequent homework and less time spent on homework compared with OECD averages (Skolveket, 2004).  Thus, it is reasonable to assume that school-level factors also play a role in determining students’ mathematical knowledge.  

Research questions

In light of the background research presented above, the main research question in this research proposal is:

Research question: To what extent do school- and teacher-level variables influence Sweden’s mathematics results on international tests?

In order to investigate the main research question, it is necessary to consider several related questions:

1.       What types of mathematical knowledge is measured by TIMSS?

2.       Are the TIMSS measurements reliable and valid?

3.   What statistical analysis methods are relevant for studying TIMSS secondary data in search of potential causal relationships?

4.       To what extent are the aforementioned statistical methods valid?

Potential causal factors can be divided into several categories:

5.       What instructional factors influence mathematics achievement within Sweden?

6.       What instructional factors influence mathematics achievement differences between Sweden and other nations?

References

Alger, S. (2007). Svenska skolans lärare och undervisning i matematik och NO i ett internationellt perspektiv: Några resultat från TIMSS 2003. (BVM 32:2007). Umeå: Umeå universitet, Department of Educational Measurement. 

Beaton, A., Martin, M. O., Mullis, I., Gonzalez, E. J., Smith, T. A., & Kelley, D. L. (1996). Mathematics

Achievement in the Middle School Years: IEA's Third International Mathematics and Science

Study. Chestnut Hill, MA: Boston College.

Boe, E. E., May, H., & Boruch, R. F. (2002). Student task persistence in the Third International Mathematics and Science Study: A major source of achievement differences at the national, classroom, and student levels (Research Rep. No. 2002-TIMSS1). Philadelphia, PA: University of Pennsylvania, Graduate School of Education, Center for Research and Evaluation in Social Policy.

Eklöf, H. (2006). Motivational Beliefs in the TIMSS 2003 Context. Theory, Measurement and Relation to Test Performance. Doctoral  dissertation. Umeå: Umeå University, Department of Educational Measurement. 

Eklöf, H. (2007). Self-concept and valuing of mathematics in TIMSS 2003: Scale structure and relation to performance in a Swedish setting. Scandinavian Journal of Educational Research, 51(3), 297-313.

Hellerstedt, L., & Reistad, H. (2012, January 22). Matteutveckling för miljarder. Origo, 1. Retrieved March 22, 2012, from http://www.lararnasnyheter.se/origo/2012/01/22/matteutveckling-miljarder

Leung, F. (2005). Some characteristics of East Asian mathematics classrooms based on data from the TIMSS 1999 video study. Educational Studies in Mathematics, 60, 199-215.

Lindström, J. (2006). Med fokus på matematik och naturvetenskap: en analys av skillnader mellan internationella jämförande studier och nationella kursplaner. Stockholm: Skolverket.

Mischel, W., Shoda, Y., & Rodriguez, M. I. (1989). Delay of gratification in children. Science, 244, 933-938.

Mullis, I.V.S., Martin, M.O., Gonzalez, E.J., Gregory, K.D., Garden, R.A., O’Connor, K.M., Chrostowski, S.J. & Smith, T.A. (2000) TIMSS 1999 International Mathematics Report: Findings from IEA’s Repeat of the Third International Mathematics and Science Study at the Eighth Grade. Chestnut Hill, MA: Boston College.

Nyström, P. (2006). TIMSS fixpunkter. En analys av vad elever med olika resultat  i TIMSS 2003 vet och kan göra. (BVM 20:2006). Umeå: Umeå universitet, Department of Educational Measurement. 

Nyström, P., & Lind, A. (2009). Hur samstämmiga är svenska styrdokument och nationella prov med ramverk och uppgifter i TIMSS Advanced 2008. Stockholm: Skolverket.

Reinikainen, P. (2004). Explanatory variables of science achievement in Finland: Cultural approach. In C. Papanastasiou (Ed.), Proceedings of the IRC-2004. IEA International Research Conference (Vols. 1–4).

Kjaernsli, M., & Lie, S. (2002). Timss science results seen from a nordic perspective. In D. Robitaille & A. Beaton (Eds.), Secondary Analysis of the TIMSS Data (pp. 193-208). New York: Kluwer Academic Publishers.

Shen, C., & Tam, H. P. (2008). The paradoxical relationship between student achievement and self-perception: A cross-national analysis based on three waves of TIMSS data. Educational Research and Evaluation, 14, 87–100.

Skolverket. (1996). TIMSS 1995: Svenska 13-åringars kunskaper i matematik och naturvetenskap i ett internationellt perspektiv. Stockholm: Skolverket.

Skolverket. (2004). TIMSS 2003: Svenska elevers kunskaper i matematik och naturvetenskap i skolår 8 i ett nationellt och internationellt perspektiv. Stockholm: Skolverket.

Skolverket. (2008). TIMSS 2007: Svenska grundskoleelevers kunskaper i matematik och naturvetenskap i ett internationellt perspektiv. Stockholm: Skolverket.

Skolverket. (2009).   Svenska elevers kunskaper i TIMSS Advanced 2008 och 1995: en djupanalys av hur eleverna i gymnasieskolan förstår centrala begrepp inom matematiken. Stockholm: Skolverket.

Stigler, J. W. (1999a). The TIMSS Videotape Classroom Study: methods and findings from an exploratory research project on eighth-grade mathematics instruction in Germany, Japan, and the United States. Washington, D.C.: U.S. Dept. of Education, Office of Educational Research and Improvement.

Stigler, J. W., & Hiebert, J. (1999b). The teaching gap: best ideas from the world's teachers for improving education in the classroom. New York: Free Press.

Välijärvi, J., Linnakylä, J., P., Kupari, P., Reinikainen, P., & Arffman, I. (2002). The Finnish success in PISA - and some reasons behind it: PISA 2000. Jyväskylä: Institute for Educational Research, University of Jyväskylä.