Въпроси на преподаването
EVOLUTION OF THE CONTENT AND QUALITY OF STANDARDIZED TESTS IN MATHEMATICS IN UKRAINE
Резюме. The evolution of the content and quality of written state standardized final and entrance testing in mathematics in Ukraine, from the first attempt to implement in 1994 until now, is considered in the article. We describe the features and stages of development of Ukrainian standardized tests in mathematics (external independent assessment and state final attestation), analyze the causes of successes and failures and also prospects for their further improvement. Based on our own statistical experiment, we compare the quality of mathematical preparation of Ukrainian graduates who entered the pedagogical universities of Ukraine in 1994, 2005 and 2020. According to this experiment, we draw conclusions about the existing problems of ensuring the proper quality of mathematics education in Ukraine and give recommendations on possible solutions to these problems.
Ключови думи: mathematical education; state final attestation; external independent assessment; entrance exams to the university; the quality of mathematical preparation
Introduction
Using of standardized testing in education remains one of the most discussed and debatable topics in modern pedagogical science. There are many scientific publications on this topic – from journal articles to monographs (see Berliner 2011, Kuncel, 2018; Lingard,2013; Miner, 2000, etc.). At the same time, the tone of discussions on the need for such tests, ways how they should be organized and the impact of them on society in general and on the teaching community in particular varies from enthusiastic and positive to critical and negative (see Claiborn, 2009; Garrisson, 2009; Johnson, 2010; Sahlberg, 2015; Wagner & Dintersmith,2015, etc.).
We keep a moderate position and believe that such a complex phenomenon as standardized testing has its own individual characteristics in the educational system of each country, and therefore, its assessment should be balanced and based on a fundamental investigations, including statistical. In this article we consider the evolution of the system of standardized tests in mathematics in Ukraine, as well as on the basis of the author’s statistical experiment we draw conclusions about its quality and suggest possible ways to improve this system.
Regarding standardized tests in mathematics, we focused on written nationwide final examinations for basic and senior school courses and on entrance examinations to the country’s universities. These trials have a great importance in society of all countries in the world, and hence in Ukraine as well. And this is natural, because the final exams determine the level of students’ achievement over a significant period of time (9 years for basic school and 11 years – for senior school), and entrance examinations, in fact, determine the future fate of the graduate, his (her) life path. Therefore, the way of their organization and quality are always in the center of public attention, and therefore, research in this direction is relevant.
Evolution of standardized testing in Ukraine
Ukraine began to develop its own system of students’ achievements assessing after gaining independence in 1991. Until then, being part of the USSR, the laws of the union state, including in the field of education, were in force on its territory. Graduation attestation in mathematics were carried out in the form of written exam conducted by school examination commissions. Such commissions traditionally consisted of: the chairman – a representative of the school administration; the examiner – a mathematics teacher who taught the children in the graduating class; and the assistant – another teacher of mathematics.
The tasks for the mathematics exam were determined at the state level, they were developed by the Ministry of Education. Completion of the tasks required students to provide a detailed written answer with a full explanation. In the 1970-s and early 1980-s, letters with such assignments were sent to schools and opened just before the exam in the presence of students.
In the late 1980-s instead of letters began to use collections of exam tasks, which were printed by state publishers. From these collections of tasks, competent employees of the Ministry of Education selected the numbers of examination tasks, which were announced on television immediately before the exam. Completing of these tasks also needed a detailed answer with a full explanation. After the exam (traditionally it lasted 180 minutes), the examiner and the assistant checked the correctness of answers under the supervision of the chairman, who approved the final results in the form of a protocol.
In the Soviet period written entrance examinations in mathematics in Ukraine were conducted by university similarly to final exams. The only peculiarity was that such examinations were no longer conducted centrally by the state, their content was developed by specially created examination commissions headed by the rector of each individual university. This method of conducting of high stakes testing has caused much criticism in Soviet times due to opacity and high levels of corruption (see Kovaleva, 2016; Shvets et al., 2020, etc.). The authors of this paper are aware of the facts when the content of entrance exam tasks became known to some students in advance, and therefore, putted them in unequal conditions with others.
After gaining independence in Ukraine, the Soviet scheme of holding final and entrance exams in mathematics used for some time. In particular, Ukrainian specialists have developed collections of examination tasks in mathematics, similar to the Soviet ones. However, the Ministry of Education almost immediately decided to change the system of entrance exams, replacing them with national standardized testing. The first such test in mathematics took place in the summer of 1994 on the basis of materials developed under the guidance of Vasyl Shvets.
However, after public discussions and discussion of the results of testing in 1994 at a meeting of the Verkhovna Rada of Ukraine, the introduction of standardized testing as an alternative to entrance examinations was postponed (the reasons for this are analyzed in detail in Shvets et. al. 2020). Therefore, starting from 1995, final examinations were again conducted according to the collections of examination tasks (see Lytvynenko et. al. 1997a; Lytvynenko et. al. 1997b, etc.), and the Ministry of Education returned to the idea of national standardized testing only in 2005, when the Ukrainian Center for Educational Quality Assessment (UCEQA) was established. With the assistance of the International Foundation “Renaissance” and the Institute of Open Society of J. Soros, with the involvement of domestic experts and foreign experts, during 2006-2008 the Center developed and implemented a new type of standardized testing – external independent assessment (EIA), which included a math test.
External independent assessment in mathematics, in contrast to the 1994 final test, replaced not only university entrance examinations but also the state final attestation (SFA). The feasibility of such a combination was questionable and due primarily to economic factors. The stages of formation of external independent assessment in mathematics are described in detail in the monograph (Shkolnyi, 2015). Here we only note that currently the EIA test in mathematics is a mandatory test for SFA of Ukrainian graduates, and is used by all state universities for competitive selection of applicants.
Features of Ukrainian standardized tests in mathematics
It should be noted that although more than 10 years have passed between the 1994 final mathematics testing and the start of math testing in EIA format, the structure of both tests is rather similar. The 1994 test consisted of 30 tasks, divided into 3 parts according to the level of difficulty (10 tasks in each part, which represented the main content lines of the school of course mathematics). The tasks of parts 1 and 2 of the test were presented in the form of multiple choice question (MCQ type, choosing one correct answer from 4 or 5 proposed), and the tasks of part 3 required a detailed answer with a full explanation. The 2020 EIA math test (see the UCEQA website www.testportal.gov.ua) contained 35 tasks: 20 tasks had MCQ type, 8 tasks required short answer (SA type, answer should be given in form of integer number or decimal fraction), 4 tasks needed matching (finding logical pairs) and 3 tasks were with a full explanation.
All tasks of the 1994 test were checked manually by pre-established commissions, which included university professors and teachers of mathematics. For the EIA 2020 test, manual checking by teachers was used only for the last three tasks, the rest of the tasks were checked automatically using a special scanner (test participants entered their answers to a special paper form).
As we can see, the ideology of conducting tests in mathematics in Ukraine since 1994 and to this day, in fact, is preserved. In contrast to standardized tests in the United States, Ukrainian standardized tests in mathematics have retained the open part of the test, which tests students’ ability to reason logically, substantiate statements, describe mathematical transformations in detail, and so on. This still introduces a certain element of subjectivity in the evaluation of student work, but allows to avoid the negative impact of student orientation technologies solely on obtaining the correct answer instead of understanding and development of cognitive skills.
One of the reasons for the above continuity for entrance testing in mathematics in Ukraine is the preservation of the tradition of conducting between 1994 and 2006 a written SFA in mathematics in 9th and 11th grade. As already mentioned, these exams were conducted according to the collections of tasks approved by the Ministry of Education. From the beginning of the 21st century, these collections did not containe just individual tasks grouped by topic, but parallel variants of tests (40-60 variants), the structure of which was similar to the 1994 entrance test and the current EIA math test. For example, collections of problems for the SFA in mathematics in 9th and 11th grade, edited by Zinaida Sliepkan (Sliepkan, 2004; Sliepkan, 2006) also contained tasks of four levels of difficulty, the simplest of which were presented in MCQ type. On the base of these SFA tests the first EIA test in mathematics in 2006 was built.
However, it should be noted that there are some differences between the 1994 and 2020 tests. First of all, as already mentioned, tasks MCQ and SA types to the EIA test are checked automatically, in contrast to the 1994 test. Second, for all 10 tasks with a full explanation of the 1994 test, each graded at 3 points, did not have clearly defined grading schemes. In the EIA test in mathematics for each of the three such tasks, which were evaluated at 4, 4 and 6 points respectively, the evaluation scheme was created before the test. Moreover, typical alternative solutions were provided, for each of which an evaluation scheme was written separately. Finally, thirdly, the content of the 1994 test tasks differed significantly from the content of the EIA 2020 test tasks. Actually, during these years both the mathematics program and the general learning paradigm have changed – from focused on the formation of knowledge and skills to the competence oriented. Although the main content lines of the school mathematics course were quite fully reflected in both tests, but, for example, the probabilistic and statistical components, which were absent in the Soviet school curriculum of the late 1980s, in the 1994 test were not presented. In the test of 1994 also were almost absent practical tasks that became very important and significant in the modern Ukrainian school.
The Table 1 below shows the distribution of the number of tasks of both tests by content lines and forms of test tasks. Data for the EIA 2020 test have taken from the website of UCEQA www.testportal.gov.ua.
Table 1. Distribution of the test tasks by the content lines and forms of rep resentation.
The differences between the tests presented in the Table 1 are natural. Indeed, the 1994 test focused on students who had spent most of their lives in a Soviet school. However, despite these differences, the basic knowledge and skills tested by both tests remained essentially unchanged. Moreover, the distribution of the number of tasks according to the content lines remained approximately the same. Thus, the 1994 test can be used to compare the quality of mathematical preparation of graduates of different years. This we will do and describe later in this article.
The problem of assessing the quality of mathematics education in Ukraine.
Usually, to the problem of ensuring the proper quality of education in general and mathematics education in particular in most countries of the world is paid much
attention. Ukraine is no exception, so both periodicals and professional scientific journals constantly publish research on this topic (see Liashenko & Rakov 2008, Dvoretska 2015, Protasova et. al. 2012, etc.).
An essential reason for considering the quality of mathematics education was the results of Ukraine’s participation in the international comparative study PISA 2018 (Mazorchuk et. al., 2019). They largely prompted the presidential decree to declare the 2020 – 2021 school year the year of mathematics education in Ukraine. In the context of this decree, the Cabinet of Ministers approved an action plan, and the Ministry of Education decided on a mandatory SFA in mathematics for Ukrainian graduates in the form of EIA, starting in 2021 (see MES of Ukraine 2019). UCEQA has developed and tested a new structure of the EIA in mathematics, which includes a two-level SFA for graduates who study mathematics at the standard level and at the profiled level (see site of UCEQA https://testportal.gov.ua/matematyka-2021/).
All the above changes indicate an increase of interest to the quality of mathematics education. However, it would be naive to assume that only the strengthening of state control over the results of standardized testing of graduates will lead to a relatively rapid improvement in their level of academic achievement. Moreover, many researchers, including Finland’s education reform developer Pasi Sahlberg, warn that over-strengthening the role of standardized testing could have the opposite effect (Sahlberg, 2015). The emphasis in learning on the results of such tests inevitably leads to a gradual narrowing of the curriculum and reorientation of the community of teachers, students and their parents from developing the child’s ability to getting a high score for the test (see also Wagner & Dintersmith, 2015).
Therefore, we believe that standardized tests in mathematics should play, although important, but not a dominant role in the development of the child’s abilities and in the formation of his (her) life and professional competencies. The emphasis in teaching mathematics should be on the development of teaching methods and technologies, not just control. Control activities should only help to correlate the achieved results with the expected ones, to improve the existing and developing new forms, methods and means of teaching mathematics. It is important to understand the trends and dynamics of student results over a long period of study. This understanding should be based on the results of a statistical experiment. It will allow the teacher to make the necessary adjustments to the learning process, focusing on the main things more than additional. This is exactly the experiment we describe later in this article.
Description of the statistical experiment
To track the dynamics of academic achievement of mathematics graduates over the past 25 years, we compare the results shown by participants in the final test in mathematics in 1994 and graduates in 2005 and 2020, who were also asked to complete the final test in mathematics of 1994. All 415 participants in the 1994 experiment were entrants to the Faculty of Physics and Mathematics of National Dragomanov Pedagogical University (NDPU). In 2005, 228 first-year students of the Faculty of Physics and Mathematics of the NDPU took part in the testing and in 2020 the same did 107 first-year students of mathematical specialties at the pedagogical universities of Kyiv, Vinnytsia, Chernihiv, Zhytomyr and Poltava. Unfortunately, the number of students of mathematical specialties of pedagogical universities has recently decreased significantly, which is why the number of test participants in 2020 was less than in previous years.
The content of test tasks for participants in the experiment in 1994, 2005 and 2020 was identical. The full text of the proposed test variant can be found in the article (Shvets, 2006). Here are just a few examples of tasks from each of the three levels of difficulty.
Samples of first level tasks.
2. Perform multiplication: \(\tfrac{a^{2}-a b}{a} \cdot \tfrac{a b+b^{2}}{b}\).
10. Point \(P\) is remoted from each of quadrate side on the distance equals to \(\sqrt{2}\), and from the plane of quadrate – on the distance equals to 1. Find the length of the quadrate side.
Samples of second level tasks.
11. Cucumbers contain an average of \(95 \%\) water. How many kilograms of water are contained in 20 kg of cucumbers?
20. The points \(A(1 ; 0 ; 1), B(-1 ; 1 ; 2), C(0 ; 2 ;-1)\) are given. Find the point \(D\) on the \(z\) axis so that vectors \(\overrightarrow{A B}\) and \(\overrightarrow{C D}\) are perpendicular.
Samples of third level tasks.
25. Solve the inequality \(2^{x^{2}} \gt \left(\tfrac{1}{2}\right)^{2 x-3}\).
27. Find the extremums of function \(f(x)=x \cdot(\ln x)^{2}\).
28. If in a quadrilateral the two sides are parallel and equal, then this quadrilateral is a parallelogram. Prove it.
30. The basis of a straight parallelepiped is a rhombus, the areas of diagonal sections are M and N. Determine the area of the side surface of the parallelepiped.
In 1994, 2005 and 2020, students were given 180 minutes to complete all 30 test tasks. The use of calculators and textbooks was not allowed.
The results of statistical experiment Table 2 shows the percentage of the total number of test participants who correctly solved a test task (task success).
Table 2. Complete results of testing.
The data on the success according to the content lines are interesting for further analysis. They are shown in the Table 3.
Table 3. Results of testing according to the content lines.
Data on the distribution of test scores on a scale from 0 to 60 points for each of the years are shown in Figure 1. The table in the chart shows the percentage of the
total number of participants who scored the number of points from the corresponding interval.
Figure 1. Results of testing by the interval scale.
The average test score in 1994 was 38 points, in 2005 – 26 points, and in \(2020-\) 22 points. Note also that the average score of the EIA test in mathematics on a scale of 100 – 200 points for the participants of the experiment in 2020 was 159, and the correlation coefficient between the results of the EIA test and the experimental test for these participants is 0.463.
Analysis of experiment results and some conclusions. The results of the experiment show that the average level of mathematical training of entrants to pedagogical universities is gradually declining. Indeed, the best results were shown by graduates of 1994. Although the success of solving for some tasks related to content lines “Numbers and Expressions”, “Equations and Inequalities” and “Functions” from 2005 to 2020 did not deteriorate (sometimes even was higher), it is still significantly lower than in 1994.
The situation is especially sad with geometry problems that require justification. In particular, none (!) of the first-year students of mathematical specialties of pedagogical universities in 2020 was able to prove correctly the criterion of parallelogram (task #28). Note that this theorem is studied in Ukrainian school as mandatory at the standard level. In our opinion, one of the reasons for this phenomenon is the extremely small number of tasks on proving statements in the SFA and EIA tests in mathematics. As a result, some teachers often simply omit proofs during explaining new material in geometry and do not focus students on the ability to prove statements, limiting themselves to solving computational problems.
The distribution of test points on a scale from 0 to 60 in 2005 and 2020 is close to normal (testing of the statistical hypotheses according to K. Pearson’s criterion confirms this), in contrast to the distribution of test points of test participants in 1994. This suggest that in 1994 some of the best entrants went to pedagogical universities, and the teaching profession was quite prestigious, in contrast to the current state of affairs. Such situation should lead to the need activities from society and the government to improve the status of teachers in Ukraine and to pay more attention to the need for proper mathematical training of Ukrainian graduates.
The rather high average score of the EIA math test for the participants of the 2020 experiment shows that now, fortunately, the entrants of pedagogical universities in Ukraine are not the worst graduates. The positive correlation between the results of the experimental test and the EIA test for participants in 2020 also demonstrates that, in general, the results of the EIA test really reflect the current level of mathematical preparation of graduates. However, the test in mathematics in 1994 and the EIA 2020 math test differ quite significantly in content and form, consequently correlation is weak. Indeed, in 26 years, both the mathematics program and the requirements for the professional competencies of graduates have changed. The fundamental component of mathematical preparation, which was emphasized in the 1994 test, gave way to a larger applied orientation of the mathematics course, more typical of the tests of EIA and SFA in 2020.
However, the development of abstract thinking, in our opinion, is still important, especially for those graduates who plan to make mathematics a field of their future professional realization and enter mathematical specialties. Therefore, the Ministry of Education decision to renew from 2021 the two-level test of external examination in mathematics is natural. The need for this we have emphasized even after the first unsuccessful attempt in 2015 (see Shkolnyi 2017). This will allow us not to lose characteristic to Ukrainian school tradition of fundamental mathematical education.
The results of the experiment show that both the content and quality of mathematics education of school graduates should be in the focus of the project “New Ukrainian School” (see NUSh 2017), which provides for the formation of an education system that is adequate to modern Ukrainian society. During this evolution, it is important, without losing the fundamental component of mathematical training, to develop applied aspects of the school course of mathematics, preparing students for successful activities in real life.
Acknowledgement. This work is partially supported by Scientific Research Grant №RD-111/02.02.2021г.of Shumen University “Bishop Konstantin Preslavsky” (Bulgaria).
REFERENCES
Berliner, D. (2011). Rational responses to high stakes testing: The case of curriculum narrowing and the harm that follows. Cambridge Journal of Education. 41.3, 287 – 302.
Claiborn, C. (2009). High Stakes Testing. Encyclopedia of Giftedness, Creativity, and Talent. SAGE Publications.
Dvoretska, L.P. (2015). Zovnishnye nezalezhne otsinyuvannya z matematyky v ukrayini cherez pryzmu svitovoho dosvidu [External independent assessment in mathematics in Ukraine through the prism of world experience]. Naukovi zapysky CSPU, 8 (III), 15 – 25. Retrieved from https://core.ac.uk/reader/228637440 [in Ukrainian].
Garrison, M. J. (2009). A Measure of Failure: The Political Origins of Standardized Testing. Albany: State University of New York.
Johnson, R. (2010). Standardized Tests. Encyclopedia of Educational Reform and Dissent. SAGE Publications.
Kovaleva, A. (2016) Shkol’nye gody sovietskie [Soviet school years]. Gazeta.ru. Retrieved from https://www.gazeta.ru/ science/2016/06/11_a_8298359.shtml [in Russian].
Kuncel, N. & Sackett, P. (2018). The Truth About the SAT and ACT. The Wall Street Journal. Retrieved from https://www.wsj.com/articles/thetruth-about-the-sat-and-act-1520521861.
Liashenko, O.I., Rakov, S.A. (2008). Testovi tekhnolohiyi i monitorynh v systemi osviti Ukrayiny: stan i perspektyvy rozvytku [Test technologies and monitoring in the education system of ukraine: state and prospects of development]. Visnyk TIMO, 11 – 12, 67 – 70 [in Ukrainian].
Lingard, B., W. Martino, and G. Rezai-Rashti. (2013). Testing regimes, accountabilities and education policy: Commensurate global and national developments. Journal of Education Policy 28.5: 539 – 556.
Lytvynenko, G.M., Fedchenko L. Ya., Shvets, V.O. (1997). Zbirnyk zavdan’ dlia ekzamenu z matematyky na atestat pro seredniu osvitu. Chastyna 1. Algebra I pochatky analizu [Collection of problems from the exam in mathematics for the certificate of secondary education. Part 1. Algebra and beginnings of analysis]. L’viv: NTL Publishing. [in Ukrainian].
Lytvynenko, G.M., Fedchenko L. Ya., Shvets, V.O. (1997). Zbirnyk zavdan’ dlia ekzamenu z matematyky na atestat pro seredniu osvitu. Chastyna 2. Geometriya [Collection of problems from the exam in mathematics for the certificate of secondary education. Part 2. Geometry]. L’viv: NTL Publishing. [in Ukrainian].
Mazorchuk, M. et. al. (2019). Nacionalnyi zvit PISA 2018 Ukraine. https:// testportal.gov.ua/wp-content/uploads/2019/12/PISA_2018_Report_ UKR.pdf [in Ukrainian].
MES of Ukraine. (2019). Deyaki pytannya provedennya v 2021 rotsi zovnishnʹoho nezalezhnoho otsinyuvannya rezulʹtativ navchannya, zdobutykh na osnovi povnoyi zahalʹnoyi serednʹoyi osvity [Some issues of conducting in 2021 an external independent evaluation of learning outcomes obtained on the basis of complete general secondary education]. Retrieved from http://search.ligazakon.ua/l_doc2.nsf/link1/ RE33821.html?fbclid=IwAR1ZNLfb_K9-VHlw9010Tr9LT1IDvFmxDhtiZaXYOoL4xMTD8BojZ6bp70 [in Ukrainian].
Miner, B. (2000). Standardized Minds: The High Price of America’s Testing Culture and what we can do to Change it / Contradictions of School
Reform: Educational Costs of Standardized Testing. The Progressive. 64, 40 – 43.
NUSh. Oficiynyi sayt. [The New Ukrainian School. Official cite]. (2017). Retrieved from http://nus.org.ua.
Protasova, N.G., Luhovyi, V.I., Molchanova, Yu.O. and others. (2012). Education reform in Ukraine: public administration aspect. Lviv: National Academy of Public Administration Publishing [in Ukrainian].
Sahlberh, P. (2015). Finnish Lessons 2.0: What Can the World Learn from Educational Change in Finland? Second Edition. New York. Teachers College Press.
Sliepkan’, Z.I. (2004). Zbirnyk zavdan’ dlia DPA. Algebra. IX klas. [Collection of tasks for SFA. Algebra. Grade 9]. Kharkiv: Himnaziya. [in Ukrainian].
Sliepkan’, Z.I. (2006). Zbirnyk zavdan’ dlia DPA z matematyky. XI klas. [Collection of tasks for SFA in mathematics. Grade 11]. Kharkiv: Himnaziya. [in Ukrainian].
Shkolnyi, O.V. (2015). Osnovy teoriyi ta metodyky ociniuvannia navchal’nyh dosiagnen’ z matematyky uchniv starshoyi shkoly v Ukrayini [The basis of the theory and methodology of educational achievement evaluation for mathematics in senior school students in Ukraine]. Monograph. Kyiv: Dragomanov NPU Publishing. [in Ukrainian].
Shkolnyi, O.V. (2017). Prospects for a two-level assessment of the quality of knowledge in mathematics in Ukraine. Evaluarea în sistemul educaţional: deziderate actuale. Chişinău: Institutul de Ştiinţe ale Educaţiei Publishing, 194 – 197.
Shvets, V.O. (2006) Do pytannia pro yakist’ matematychnoi osvity. [On the question about math education quality]. Matematyka v shkoli. 4 (60), 22 – 25. [in Ukrainian].
Shvets, V.O., Bevz, V.G., Shkolnyi, O.V., Matyash, O,I. (2020). Ukraine: School Mathematics Education in the last thirty years. In A. Karp (Ed.), Eastern European Mathematics Education in the Decades of Change. Cham: Switzerland. Springer.
Wagner, T., Dintersmith, T. (2015) Most Likely to Succeed. Preparing Our Kids to the Innovation Era. New York. Scribner.