Projects 2013

The head of the project: M.Karnoy, I.D. Frumin

1. Comparative study of the quality of training in mathematics and physics engineering students of universities in the BRIC countries (Project Supertest)

 

Research object: engineering education in BRIC countries.
Research purpose: to conduct a comparative research of preparedness of BRIC school graduates for university (particularly graduates who are entering engineering majors), and the quality of their education after two years of university. The aim in 2013 year was to develop a proposal of the research of the quality of engineering students preparedness in physics and mathematics in Russia and the same students in other BRICs. The main challenge for 2013 was the analysis of engineering education in BRIC countries, the quality of engineering graduates (including school and university graduates) in order to understand whether it is necessary and possible to develop one test to assess the quality of engineering graduated in BRIC, and one test to assess the quality of engineering education. Consequently, the research of 2013 year is the first step in a big research designed for 2014-2017 years.

The present research continues previous works of laboratory researchers that were connected with the analysis of higher education (particularly, engineering) in BRIC.

Empirical base of research:
1. Were analyzed data from more than 400 universities in Russia that provide education in engineering majors
2. 4 mathematics lecturers from 4 different Russian universities and 176 first-year students of the same universities were surveyed by the questions on mathematics from JEE Main test (India) and CEE (China) in order to understand whether they can solve such tasks.


Research results:
1. Analysis of literature revealed main characteristics of higher education in BRIC countries: differentiation of universities, big number of private universities and high role of government in higher education.
2. Methods for choosing engineering graduates differ between BRIC in syllabus and configuration
3. Research on USE scores showed that there are a lot universities that take very weak students on engineering majors (average USE score is less than 55), and there are only 30 universities that take good students (average USE score is more than 70)
4. Expert (mathematic lecturers) and student assessment showed that such tasks can be solved by Russian school graduates and can be used as a base for future test for BRIC countries.

Implementation of research results: the results of this research will be used as a base for the development of international test for the assessment of engineering education quality. 

2. Communication of the educational environment and educational contexts and practices of learning outcomes

Part 1

Research object: resilient and “antiresilient” students.
Research purpose is to study factors which drive resilience and “antiresilience”.
Empirical base of the research is  PISA and TIMSS students’ and schools’ data.
Research results: Compared to other countries, there is an average number of both resilient and “antiresiliet” students in Russia (about 3%). Practices that effect resilience are reading for joy, practices of control in learning used by students, practices of support used by teachers and teachers’ stimulation of reading. Finally we find that memorization used by students supports “antiresilience”


Part 2

Research object: USE's predictive validity
Research purpose: To analyze the relationship between USE scores and a student’s further academic achievement in university. Through that relationship, we will examine the predictive validity of the USE as a tool for university admission.
Empirical base of the research: The sample consisted of students from five universities across different Russian regions. Four of the universities provided information about all students enrolled during 2009-2011, and one provided information only about the two largest departments within the same cohort. Overall, the sample data included 65 departments and more than 19,000 students.

Research results: the total USE score predicts further performance accurately enough to consider this exam as a valid tool for selecting applicants. The regression models resulted in a percentage of variance explained ranging from 15 to 35% in different departments. The predictive ability of specific subject scores is approximately the same, but the examinations in Mathematics and Russian language are most likely to be the best predictors of academic performance. The association between the USE results and academic performance during the second year and higher could be observed only indirectly through the first year performance.
Implementation of the research results: suggested methodological approach can be used as a tool for monitoring survey for any university.

Part 3


Title: «The characteristics of the educational environment and learning outcomes in primary school (based on SAM - Student's Achievement Monitoring)».

Research object: the characteristics of the educational environment in primary school.
Research purpose: to identify and to describe the characteristics of the educational environment related to the educational results in primary school in the context of Vygotsky's theory.
Empirical base of the research: the results of student’s testing in grade 4 (in one region of the Russian Federation, 4406 students), and questionnaire for their teachers.
Research results: The results of SAM test in the region were described. The scales for constructive and traditional teachers’ practices were composed and their reliability and validity were confirmed. The characteristics of the educational environment associated with successful test results were identified and analyzed.
Implementation of the research results: The results can be taken into account by the regional and federal authorities when decision making and policy formation

Part 4


Research object: shadow education effects on high school students’ academic outcomes.
Research purpose was to provide rigorous evidence about the causal impacts of participating in shadow education on college preparation.
Empirical base of the research: to estimate the impact of participating in shadow education on student achievement in Russia, we rely on data from a large-scale, representative survey. The survey was conducted in May 2010 in three Russia regions: Pskovskaya and Yaroslavskaya oblasts and Krasnoyarsky krai. The schools in the dataset were sampled using a stratified random sample design. In each sampled school, all students in the 11th grade were surveyed. Altogether, the dataset contains information on 2,938 final-year (grade 11) students in 127 schools. Besides students three more groups of respondents were surveyed within each school: Russian language and math teachers, and school principals. Finally, in the summer of 2010, after USE test results were released, each student’s individual USE scores in math and Russian language were collected.
Research results: we find that participating in shadow education positively impacts high-achieving students but not low-achieving students. Participating in shadow education further does not lead students to substitute time away from other out-of-school studies. Instead, the results suggest that low-achieving students participate in low-quality shadow education, which, in turn, contributes to inequality in college access.
Implementation of the research results: policymakers that are concerned about inefficiency in the provision of shadow education (in helping low-achieving students) may wish to find ways of providing this information to low-achieving students.

 

3. Cognitive skills and national achievements in international studies of education quality

The project was implemented within the framework of the HSE Basic research program.
Research object: transfer of learning.
Research purpose: to combine PISA and TIMSS data collected on the same sample of students to examine the link between the mastery level of subject knowledge in math and knowledge transfer into out-of-subject context.
Empirical base of the research: russian students (n= 4241) from 229 schools who took part in both TIMSS-2011 and PISA-2012. There were 49.8 % girls and 50.2% boys with a mean age of 15.9 years (in 2012) (SD=0.5). The sample was representative for Russian 14-15 year-old students in terms of school size and school location.
Research results: we found a positive relation between the mastery level of a subject and the ability to transfer learnt math to out-of-subject context. The better the mastery level of mathematics, the higher probability that learnt math will be transfer to contextualized situations. This link was not linear though: only the highest mastery level contributed significantly to the transfer. The other levels only slightly differentiated performance in transferring.
Implementation of the research results: the impact of formal subject knowledge on its transfer is discussed on frame of theory of transfer as well as reinterpreting achievement testing.

 

 4. Characteristics of teachers and their impact on the educational outcomes of students

Part 1


Title: “Mathematics teachers’ professional profile”
The project was implemented within the framework of the HSE Basic research program.
Research object: mathematics teachers of ISCED 2 level.
Research purpose: To identify and describe the psychological and professional features of ISED2 mathematics teacher in Russia and comparing the professional profile of Russian math teacher with the results of cross-cultural international studies.
Empirical base of the research: To estimate the socio-demographical feachers the data from the website of Federal Agency of Education were used. To describe professional knowledge of mathematics teachers the TEDS-M database was used. To estimate and describe school climate, teachers’ beliefs and practices the field trial of NorBA questionnaire was conducted.
The data for field trial were collected in Latvia, Estonia and Russia. The target group was secondary school math teachers (in grades 7-9). In Latvia 385 mathematics teachers from different regions participated in the survey. In Estonia 327 teachers took part in the survey. In Russia research was conducted in one big region in Siberia. 1223 teachers took part in the survey. It is 40% of the general population and the sample is representative relative to population of the region. 

 Research results:


1. 98 % of teachers - women, 96 % have higher education;
2. 25% are teaching in big cities (over 200 thousands citizens), 30% - in small towns (less than 100 thousands), 29% - in villages and 14% in urban settlements; 37% of teachers have more than 26 students in a class, 15% - less than 10.
3. The average age of teachers was 48 years, while there are only 4% of teachers younger than 26 years, and 20% is over 55 years (retirement age). Thus, we can talk about the general "aging" of teachers and a small tributary of the new specialists. This is also confirmed results of TEDS-M: only 5% of graduated students of pedagogical universities are confident that they will work in school, while 73% do not consider teaching as promising.
4. Graduates of Russian pedagogical universities, future teachers of mathematics, have a high level of both fundamental mathematics knowledge and  pedagogical knowledge according to TEDS-M.
5. Differences between math teachers’ beliefs in different countries are statistically significant. Russian teachers are high in their constructivism then teachers from Latvia and Estonia.
6. Most of teachers in Estonia compromise both approaches: their views about good teaching include construction of knowledge and accept also transmission of knowledge in combination with it.
7. Russian teachers are strongly oriented towards systematic approach to math teaching.  It supports the claim that in Russia traditions of high quality mathematics education are still strong; emphasis on rigorous proofs, logic, exact definitions and a precise use of the mathematical language is a characteristics of Russian mathematics education.
8. Teachers’ beliefs are realized in their classroom practice.
9. Implementation of the research results: The research results will help to evaluate school mathematics education in Russia in comparison with other countries and to predict its development. 

International partners:

1)  Markku Hannula, PhD, University of Helsinki,Teacher Education, Faculty Member,http://helsinki.academia.edu/MarkkuHannula
2)  Madis Lepik, PhD, Tallinn University, Institute of Mathematics and Natural Sciences ; Associate Professor, https://www.etis.ee/portaal/isikuCV.aspx?PersonVID=38069&lang=&lang=en 

Part 2


The project was implemented within the framework of the HSE Basic research program.
The project was implemented within the framework of the HSE Basic research program.
Research object: impact of teachers’ characteristics on students’ performance in easiest and hardest mathematical tasks
Research purpose: describe the national achievements in international assessments programs in terms of performance in easiest and hardest mathematical tasks;determine an impact of teachers’ characteristics on students’ performance in easiest and hardest mathematical tasks.  
Empirical base of the research: national sample of TIMSS-2011 (8 Grade) 5008 students человек (49,2% females и 50,7% males): 211 schools, 230 classes и 231 math teachers.
Research results: Teachers’ practices can be divided into three groups: asking students to solve hardest (advance level) tasks, asking students to solve routing tasks; asking students to do math under teacher supervision;

- practicing with hardest tasks associates with higher performance in TIMSS, practicing with routing tasks associates with lower performance; 
- inappropriateness  of TIMSS data to planned type of analysis led to shifting method: dividing  the students into six TIMSS-groups according  to their performance in TIMSS; identifying 10 (and 20) the most difficult PISA items (based on Rasch Model); determining a percentage of the most difficult PISA items answered correctly in every TIMSS group. This percentage served as a measure of the ability to transfer school math knowledge to out-of-subject and apply it to out-of-subject context. In details see Direction 3 in this report. Implementation of the research results: conducted analysis can be consider as a preliminary stage for the following research.