Projects 2014
The head of the project: M.Karnoy, I.D. Frumin
Direction 1. Factors of educational quality
The main goal was the development of test tools for the assessment of engineering students’ skills and knowledge in Russia and China. Tests for two areas (academic knowledge and critical thinking) were developed. Academic knowledge is measured by mathematics and physics tests. To create those tests we used experts’ oriented approach, which allows creating tests with the existing item pools. The item banks were provided by China, Russia and USA. Items for mathematics and physics were translated into all three languages, and then evaluated by experts in each country. The critical thinking test was provided by ETS (Educational Testing Service); it was translated and adapted to the needs of the project.
The clinical pilot study was conducted in order to check the testing procedure and item functioning. Based on the results of clinical study we corrected the procedure for the main piloting. The main pilot study was conducted in the beginning of November 2014 and included more than 2600 students in 10 universities. Sample was randomly divided into 2 groups, 1/3 were assigned critical thinking test and 2/3 were assigned general academic knowledge tests in mathematics and physics. The results of the piloting are very encouraging – the tests can be considered as good measurement toolkits in psychometric terms. Most items demonstrate good psychometric quality. We also find it possible to construct the common scale between grades and countries.
Following toolls were developed and piloted:
1. The general academic knowledge (physics and mathematics) and critical thinking tests for the 1 grade students were developed and piloted;
2. The general academic knowledge (physics and mathematics) and critical thinking tests for the 3 grade students were developed and piloted;
3. Contextual questionnaires for students, teachers and college administration were developed and piloted.
Direction 2. Relation of educational contexts and teaching practices to students’ outcomes
This research direction consists of three parts. One part had rather methodological focus. We were interested in how different approaches to family socio-economic status operationalization relate to discrepancies in research results. Other two parts were devoted to different factors driving secondary school students’ educational outcomes and trajectories in Russia. In the second part we investigated factors of secondary school students’ self-efficacy in a subject. Finally, in the third part we focused on effect of the whole class and referent classmates on high school students’ performance and after-school educational trajectory choice.
Different data and methods were used depending on the research questions. The first part of the research was based on deep literature review. We analysed approaches to family SES operationalization. For the second part we used data of Russian longitudinal panel study of educational and occupational trajectories (TIMSS 2011 and PISA 2012 waves). Regular OLS series of logistic regressions. We used the data of the same longitudinal survey (fall 2013 wave).
The results of the first part of the study have shown that there is quite a wide range of SES proxies used in social science research. The most frequently used operands are parental occupation and education, indicators of family financial capital (income), cultural capital indicators, such as books in the home and other cultural possessions. Our analysis shows that question about parental occupation has the most parents-child agreement. Question about parental education, on the contrary, has low parents-child agreement and is more tended to be missed by the children. All variables need additional conversion to become appropriate for use in comparative international studies. Number of books in home is a widespread indicator of cultural capital or family educational resources. This question has quite high level of agreement and low non-response rate especially when its categories grouped in bigger ranges. Concerning practical needs of research it can be said that researchers should pay more attention to parents’ occupation rather than education when they operationalize SES.
While doing the second part of the research we found that self-efficacy in math is correlated with some out-of-school factors such as gender and family SES. Among in-school factors the most important is average class achievement in particular subject. When low performing student is in a high-performing class this decreases his or her self-esteem. The same time if teacher uses different math tasks for students with different level of abilities in math this will strengthen big-fish-in-little-pond effect for lower-performing students.
The third part of the study was devoted to peer-effect on performance and educational trajectory choice. We found that student’s achievements in algebra are more related to the whole class average performance, rather than to the referent classmates’ outcomes. However in physics student’s achievements show statistically significant correlation with both class average scores and referent classmates average scores. High performance in physics increases a chance of STEM choice, while high performance in math does not. It could be so probably because performance in math may reflect general ability rather than interest to STEM. Proportion of classmates who are going to enter STEM department in college turned out to be not significantly related to student trajectory choice. However if some referent classmates choose STEM and these classmates are central persons in class network this increases the probability for a student to go to STEM as well. These inferences demonstrate that studying peer-effect only by estimation of average performance in class can lead to biased estimates. Therefore it is crucial to include in-class network characteristics in analysis when peer-effect is in interest.
Direction 3. Comparative Study of Secondary School Mathematics Teachers’ Beliefs and Practices
The research revealed that differences between the teachers of mathematics in Russia, Latvia and Estonia were statistically significant on all the scales analyzed. Moreover, the teachers from the Russian-language (minority) schools in Estonia and Latvia were in most dimensions somewhere between teachers from Russia and teachers from Estonian and Latvian-language schools. High level of constructivism is typical for teachers in Russia, as compared to the teachers in Latvia and Estonia. At the same time, a considerable percentage of Russian teachers are traditionalists, which proves that teaching mathematics as a collection of rules, formulas and procedures is still quite popular in Russia.
We can assume that the Estonian and Latvian teachers use both approaches at the same time in their practices, because the percentage of teachers with average results for both constructivism and traditionalism scales is rather high. We can suggest that these teachers teach to develop conceptual understanding of mathematics among learners, at the same time considering the instrumental views of mathematics and focusing on rules and procedures, provided that classes of learners are heterogeneous in their levels of training. Maybe, this approach of compromising conceptions of teachers has led to better performance of Latvian and Estonian students in PISA study.
These teachers’ conceptions of good teaching might combine understanding of teaching as knowledge construction and as knowledge transfer. Conventional teaching focused on procedures and modern constructivist teaching methods developing conceptual understanding of the material are seen not as opposed but rather as complementary. The proportion of teachers with low levels of traditionalist beliefs is higher in Latvia and Estonia than in Russia. Apparently, the Baltic teachers integrated into the European community are trying to get rid of obsolete and routine teaching methods.
Thus, the research has shown that different approaches to education reforms used in Russia, Latvia and Estonia resulted in significant differences in mathematics teachers’ beliefs about teaching mathematics. The analysis of cross-cultural differences in teachers’ beliefs provides essential information about teachers’ classroom practices and choice of teaching strategies. This data allows to evaluate more accurately the situation in general education school and to predict its further development, which is especially relevant in the light of the education reforms. Furthermore the methodology to measure the differences in lower-secondary teachers’ beliefs in cross-cultural context has been created.
Direction 4. Solving of contextual tasks: a role of subject knowledge, general abilities and teachers’ factors
The aim of this qualitative study was to understand whether the difficulties of Russian students in solving PISA math problems are specific and caused by some particular characteristics of these problems or the difficulties are general, but revealed themselves because of a specific, uncommon design of these problems. The results suggest that partly failures in problem solving could be explained by difficulties in modeling process, that is they were specific for these particular problems. However the vast majority of mistakes reflected the poor general problem solving skills. We found that an inability to visualize clearly a disposition of the key task elements has a detrimental effect on performance. Also the need to develop a solving strategy, validate their own solution made difficulties for students.
The other study was dedicated to developing of a principal framework for word math problems that are intended for assessment of a competence to apply mathematics to real life situations. In this study we proposed one possible theoretical model, based on the key characteristics of real everyday situations where math knowledge would be applicable. We have considered the following model parameters as obligatory: 1) everyday semantic; 2) novelty, untypical for a problem solver; 3) consistency between a question and a context (a situation) of a problem; 4) moderate structuring; 5) requiring some curricular math knowledge for solving. We have shown how the proposed model parameters can be applied to distinguish between different math contextual problems and develop appropriate problems.
Finally, there was conducted analysis of math teacher practices in classes with poor and excellent PISA results as it was suggested that specific teaching practices are correlated with students’ results in PISA. At the first stage, the most and the less effective schools were identified by regression analysis with PISA and TIMSS scores on participants of longitude survey. However, further statistical analysis in those different of classes hasn’t revealed the difference in math teacher practices, and therefore it was suggested to start a series of video-observation in math classes and in-depth interviews with those math teachers.
Publications:
Тюменева Ю. А., Вальдман А. И., Карной М. Что дают предметные знания для умения применять их в новом контексте? Первые результаты сравнительного анализа TIMSS-2011 и PISA-2012, проведенных на одной и той же выборке российских учащихся. // Вопросы образования. 2014. № 1. С. 8-24.
Tyumeneva Y. A., Kuzmina Y., Kardanova E. IRT Analysis And Validation Of The Grit Scale: A Russian Investigation / Working papers by NRU Higher School of Economics. Series PSY "Psychology". 2014. No. WP BRP 24/PSY/2014.
Кузьмина Ю. В. Метод разрывной регрессии и метод отбора подобного по вероятности для оценки эффекта одного года обучения: опыт применения на примере данных PISA 2009 // Социология: 4М. 2014. № 38. С. 7-37.
Тюменева Ю. А. Задания на "перенос" знаний: теория и практика // Математика в школе. 2014. № 10. С. 3-9.
Khavenson T., Solovyeva A. Relationship between the Unified State Exam and Higher Education Performance / Пер. с рус. // Educational studies. 2014.
Карданова Е. Ю., Пономарева А. А. Исследование убеждений и представлений учителей математики об обучении математике в основной школе // Качество образования в Евразии. 2014. № 2. С. 115-131.
Zakharov A., Carnoy M., Loyalka P. Which teaching practices improve student performance on highstakes exams? Evidence from Russia // International Journal of Educational Development. 2014. Vol. 36 (May 2014). P. 13-21.
Карданова Е. Ю., Пономарева А. А., Осин Е. Н., Сафуанов И. С. Сравнительное исследование убеждений и практик учителей математики основной школы в России, Эстонии и Латвии // Вопросы образования. 2014. № 2. С. 44-81.
Carnoy M., Khavenson T., Costa L., Fonseca I., Marotta L. Is Brazilian Education Improving? A Comparative Foray Using PISA and SAEB Brazil Test Scores / Working papers by NRU Higher School of Economics. Series EDU "Education". 2014. No. 22.
Ларина Г. С., Кузьмина Ю. В. Стало ли российское образование эффективнее за восемь лет? (на данных международного исследования PISA) // В кн.: XIV Апрельская международная научная конференция по проблемам развития экономики и общества: в 4-х книгах. Книга 4 / Отв. ред.: Е. Г. Ясин. М. : Издательский дом НИУ ВШЭ, 2014. С. 565-574.
Carnoy M., Вальдман А. И. How well do you need to know it to use it? / Working papers by NRU Higher School of Economics. Series EDU "Education". 2014.
Carnoy M., Froumin I., Loyalka P. K., Tilak J. B. The concept of public goods, the state, and higher education finance: a view from the BRICs // Higher Education. 2014. Vol. 68. No. 3, Published online: 25 February 2014. P. 359-378.