Evaluating Effects of Digital Transformation on School Outcomes

Experts at IOE and Yandex have reported findings from a one-of-a-kind massive joint study that they carried out in association with Stanford and the University of California to evaluate whether and how engaging in practices of e-learning contributes to academic performance in primary school. Completing more assignments online can be specifically of aid in catching up those early-graders who fall behind on math literacy, the study suggests.    

The study was carried out by the International Laboratory for Education Policy Analysis and the Centre for Psychometrics and Measurements in Education together with Yandex.Uchebnik, with the support of leading experts from Stanford and the University of California.

Over 6,000 third-graders and their teachers from 343 schools in Novosibirsk and Altay regions of Russia took part in the study.

This research project was the first in contemporary Russia to use the ‘gold standard’ for assessing the impact on school education, namely a large-scale experiment. The participants were randomly placed in different experimental conditions. In ordinary interim assessments, the effect in question cannot be ‘cleared’ from many other side factors, and as a result, it is often difficult to determine the real reasons behind certain results. Random distribution helps solve this problem.

In the experimental groups, school students did homework in math and Russian using Yandex.Uchebnik (Yandex.Textbook), which includes over 35,000 assignments for elementary school, on smartphones and computers. The control group did homework in a more traditional way. Psychometricians from the Institute of Education compared the results of both groups in two rounds of tests with iPIPS+, a complex of modern tools to benchmark learning attainment in early-graders.

What researchers have found debunks some myths about the digitalization of schools.

‘First, completing assignments online has an upward effect on academic performance, and the school students maintain a higher interest in their studies. Second, the new technology may be most useful for underachieving students, in whose case the greatest positive correlation has been found. Third, we have not monitored any additional workload for teachers due to the introduction of online assignments: they did not spend more time on preparing for lessons and checking the homework. At the same time, they started using some other digital resources in their work more often, in addition to those suggested in the experiment,’ said Andrey Zakharov, Head of the International Laboratory for Education Policy Analysis.

This study is unique in nature and highly significant. Such complex audacious experiments help us understand what really works in education and what does not work. The importance of this research is particularly obvious in light of the fact that large-scale changes in education like digitalization have an impact on all families and are often costly. This means that decisions here should be made using evidence-based educational policy. Together with our colleagues from Kazan Federal University, we are discussing the launch of an information platform about ‘reliable’ innovations in education. 

Isak Froumin 
Head of HSE Institute of Education

The HSE Institute of Education has become one of the world’s leading hubs in psychometrics – a research field that helps properly design assessment tools, find the best forms of their implementation, and validly interpret their results. The Institute runs a Master’s programme and a research centre in this field that are headed by Professor Elena Kardanova.

The Institute also carries out comprehensive research on the development, prospects, and problems of education digitalization in Russia.  Particularly, this year, a new Master’s programme, Digital Transformation in Education, which has been launched by the Laboratory for Digital Transformation of Education, is enrolling the first students.

Digitalization of school education is entering a new stage as prominent IT players get more involved. Since its launch, Yandex.Uchebnik has planned not only developing its product, but also carrying out fundamental research together with leading research teams.

‘Importantly, the outcomes of this effort will be useful for all parties involved—teachers, researchers, and the Uchebnik team—for its further development,’ said Evgeny Lurie, Product Director of Yandex.Uchebnik.

According to Evgeny Lurie, real field research is hard work. ‘The project involved dozens of professionals from our two organizations, as well as local coordinators and education development institutions.’