https://doi.org/10.53656/voc24-1-06

Резюме. In this article, the author continues the argument, that he has previously put forward (Budgell 2023), for an open and transparent use of data that already exists but is not collected, collated and analysed appropriately within the education system in Bulgaria. Currently, School Directors have at their disposal: 1. a longitudinal or trend analysis; 2. a comparative analysis within the school; 3. a comparative analysis across schools; and 4. an ipsative analysis, the assessment of pupils’ progress or value-added by the school. However, given: a. the nature of the pupil population; and b. the structure and organisation of the school system these techniques are of limited value without a detailed analysis of the State Matriculation Examinations. The author uses the prior attainment scores and the results from the State Matriculation Examinations for the pupils who left a Mathematics Grammar School in 2023 to illustrate the limitations of the techniques available to School Directors. Finally, the author proposes, in detail, the type of analysis that would take account of: i. the actual pupils who attend an individual schools; and j. the differences between the results of the subjects for which there is a State Matriculation Examination. He concludes that, it is only with this type of detailed analysis, that it is really possible for School Directors, the teachers and parents to know how well their school is performing.

Ключови думи: education system; Bulgaria; key performance indicators; variables; input; output; intervening

Introduction

“Even though measuring the impacts of the reforms on education quality is important, a national standardized test for this purpose does not exist. The national external examinations are not designed to measure progress in student learning over time. The national examinations are not comparable over time and the most important ones – grades 7 and 12 – are for selection into higher levels of schooling.

A national standardized test could be used for measuring the impacts of the reforms on quality. This may require improvements in the national tests to ensure comparability over time and publication of results. An alternative would be to create a separate national standardized test for this purpose. In either case the results should be public, disseminated, analysed, used for policy and strategy, and comparable over time and grade. The strengthened assessment system should be aligned with the accountability and autonomy framework.

National assessments are needed to measure progress, school value-added, and to determine the ability of parents and local authorities to hold principals accountable. Existing national assessments could be made more suitable for monitoring changes in quality resulting from the reform”.

The World Bank: A Review of the Bulgaria School Autonomy Reforms

At an awards ceremony held on Tuesday 20th June, a School Director presented the Klac XII pupils with their Diplomas of Secondary Education. As part of the presentation, the School Director told the pupils and their parents that the pupils had achieved an average of 5.68 in the Diploma of Secondary Education and an average of 4.97 in the State Matriculation Examinations.

The awards ceremony was an occasion for the pupils to celebrate their success and to say goodbye to their teachers; it was not the right time to ask hard questions about how high the standards of achievement were in 2023. However, these are important questions for School Directors, teachers and the parents of younger children in a municipality who will be thinking about which is ‘the best secondary school’ for their children to attend.

If a School Director wants to determine how good the standards of achievement are, a number of approaches are available:

5. a longitudinal or trend analysis;

6. a comparative analysis within the school;

7. a comparative analysis across schools; and

8. an ipsative analysis, the assessment of pupils’ progress or value-added by the school.

A Longitudinal or trend analysis

Because schools already have the data, a Longitudinal or Trend Analysis is the most straight forward exercise for School Directors to undertake. The results for the last 10 years are illustrated in Table 1 and Figure 1.

On initial inspection, Figure 1 suggests that there has been:

a. a gradual increase in the average Diploma of Secondary Education results; but b. a decrease in the State Matriculation Examination results.

However, the Diploma of Secondary Education and the State Matriculation Examinations raise questions of validity and reliability.

– The validity of an assessment procedure is the idea that it measures what it is intended to measure. Validity is concerned with the connection between the purpose of the assessment procedure and which data the education system chooses to quantify that purpose. The Diploma of Secondary Education is intended to assess the pupils’ knowledge, skills and understanding across the whole curriculum; but it is quantified primarily from unmoderated assessment by the teachers who taught the pupils. This leaves doubt about the reliability of the upward trend in the Diploma of Secondary Education

– The reliability of an assessment procedure is not at all concerned with intent; it is concerned with whether the data collected by the assessment procedure produces accurate results. In this context, accuracy is defined by consistency (whether the results could be replicated). If, for example, pupils sat for the School Matriculation Examination in English Language on a different day, in a different setting, marked by a different examiner would they get the same results? However, given that the results in the State Matriculation Examinations are determined by one examination, there is serious doubt about whether it can assess pupils’ knowledge, skills and understanding across the curriculum; i.e., doubt remains about the validity of the decrease in results of State Matriculation Examinations.

These questions of validity and reliability are further emphasised by Table 2; which breaks down the Diploma and Examination results into individual subjects. Are the results in the State Matriculation Examination in Mathematics really (almost) a grade worse than the results in English Language or is this an indication that the issues raised by the World Bank (2012)

“The national external examinations are not designed to measure progress in student learning over time. The national examinations are not comparable over time”. 1 still need to be addressed? The relationship between the Diploma results and the Examination results will be returned to in the next section, suffice it to say at this stage that the relationship between validity and reliability is characterised in Table 2, in which, the average Diploma results are only for those pupils who actually took the Examination in that subject. What is certain at this stage is that School Directors must remain unsure whether there is a real increasing or decreasing trend over time.

A comparative analysis within the school

Again, because the data already exists within the school, the next most straightforward analysis for School Directors to undertake is a Comparative Analysis within the School.

Figure 3 illustrates two measures of the overall attainments of the pupils: the Diploma of Secondary Education and the Average Results in the State Matriculation Examinations. Overall, the results illustrated in Figure 3 reflect the results presented by the School Director at the Diploma Awards Ceremony: i.e., that the pupils achieved an average of 5.68 in the Diploma of Secondary Education and an average of 4.97 in the State Matriculation Examinations.

Figure 3 is also summarised in Table 3 which shows that although 82 pupils were awarded a Diploma Score of 5.50 or above, only 22 pupils attained an Average Score of 5.50 or above in the State Matriculation Examinations. In addition to validity and reliability, this marked difference in the distribution of the results raises a range of serious issues.

1. If it is intended that both the Diploma of Secondary Education and the State Matriculation Examinations assess pupils’ knowledge, skills and understanding across the curriculum, they are obviously assessing very different things. Does the Diploma assess breadth but not depth, while the Examinations assess depth but not breadth?

2. Are the assessment criteria used by the teachers in quantifying the Diploma set too low: i.e., is it too easy, for example, for a pupil to be awarded a 6.0 by the teachers?

3. Alternatively, with so many pupils (more than 30%) being awarded Diploma Scores of 5.90 or above, does the system differentiate between sufficiently between the good pupils and the (truly) excellent pupils?

4. If the Diploma of Secondary Education is intended to be used by Higher Education to select the best pupils, how do they differentiate between good and excellent pupils?

Is it surprising therefore, that:

a. Higher Education does not trust the Diploma of Secondary Education; and b. in order to select the best students, the more prestigious universities set their own entrance examinations?

The discussion so far has concentrated on overall measures of the standards of achievement:

1. the overall results in the Diploma of Secondary Education; and

2. the average results in the State Matriculation Examinations;

Figure 4 illustrates the results in individual subjects in both the Diploma and the Examinations. In common with Table 2, the Diploma results in Mathematics, for example, includes only the pupils who sat the Examination in Mathematics.

Figure 4 indicates that there is no significant difference across the subjects in the Diploma Scores. However, this is not true for the Examination Scores. In Mathematics, for example, the Average Examination Score was 4.47 while the Average Diploma Score was 5.36 for the same pupils. At the same time, while the Average Examination Score in Mathematics was 4.47, the Average Examination score in English Language was 5.36.

Figure 5 presents the data about the standards of achievement in a different format. Rather than the average (mean) illustrated in Figure 4, Figure 5 illustrates the minimum, the lower quartile, the median, the upper quartile and the maximum. Figure 6, is derived from the same data set but summarises the data previously illustrated in Table 2 and Figure 4 as a scatter graph.

This discussion about standards of achievement across subjects raises an additional range of important issues.

1. Why are the Diploma Scores so high in Bulgarian Language and Literature (5.81) and so low in Mathematics (5.56)?

2. Why is the different between the Diplomas and the Examination Scores so low in English Language (0.33) and so high in Mathematics (1.09)?

3. Given that the school has 2 Profiled classes in Informatics; why did only 3 pupils elect to take Informatics as their second State Matriculation Examination?

Questions about the standards of achievement across subjects in the State Matriculation Examinations cannot be answered within the school, they require the Ministry of Education to regularly and reliably publish national information about standards of achievement across subjects. If the national Average Score in the State Matriculation Examination in Mathematics is 0.89 below the national Average Score in the State Matriculation Examination in English Language, it provides a partial answer to questions about the difference within the school.

A comparative analysis across schools

The fact that the Ministry of Education and Science does not regularly publish reliable data about standards of achievement makes it hard for School Directors to undertake any comparative analysis across schools. The data presented in this section is reliable but it is not for the 2018 – 2023 cohort of pupils.

Figure 7 illustrates the Average Score in the State Matriculation Examination in Bulgarian Language and Literature for all the schools in The Municipality. For the year in question, a School Director could be sure that the pupils achieved the highest Average Score in The Municipality. However, there is an important caveat that will be returned to in the next section. The school is able to select the pupils with the highest scores in the Year 7 National Tests; that is to say, the school can select the pupils with the highest prior attainment in The Municipality. Therefore, everything else being equal, they ought to achieve the highest Examination Scores. The question remains, however, of the extent to which the school is achieving that highest Average Score. It is at least theoretically possible that the school could achieve the highest Average Score and still, comparatively, be under-achieving.

Figure 8 illustrates the Average Score in the State Matriculation Examination in Mathematics for all the Mathematics Grammar Schools in Bulgaria. Again, for the year in question, School Director could be sure that the pupils achieved one of the highest Average Scores in Bulgaria. However, there is an analogous caveat. The Mathematics Grammar Schools can select the pupils with the highest prior attainment in their municipality, but the schools are of different sizes in municipalities with very different pupil populations. It is unclear therefore what proportion of the most able pupils an individual school can select. Therefore, question of the extent to which the school is achieving one of the highest Average Scores remains; only in this case, it is at least theoretically possible that in achieving one the highest Average Scores, the pupils are also making the most progress in the country.

The caveats raised in this section cannot be addressed fully until the Ministry of Education and Science publishes annual, reliable information about standards of achievement.

An ipsative analysis, the assessment of pupils’ progress or value-added by the school

The previous sections have presented uncontextualized performance data: that is to say, they made no reference to the pupils’ socio-economic circumstances or their prior attainment. The following sections, take into account pupils’ prior attainment and, in doing so, attempt to assess the progress made by the pupils and the value added by the school.

The school selects its pupils using an algorithm that calculates a score derived from:

– 3* the score in the Year 7 National Test in Mathematics;

– the score in the Year 7 National Test in Bulgarian Language and Literature;

– the score in the subjects in the Primary Certificate of Education that correspond with the Profile; and

– the score in the Bulgarian Language and Literature.

Figure 9 illustrates the scores on the Admission Criterion for the same pupils that were admitted to the school in 2018 and were awarded their Diploma of Secondary Education in 2023. The minimum score was 398.50, the mean was 427.30 and the maximum was 481.75 – a range of 83.25.

In Figure 10 the results in the State Matriculation Examination in Bulgarian Language and Literature2 are plotted against the Scores on the Admission Criterion. The ‘Regression Line’ included in Figure 10 reinforces the immediate impression that there is a strong relationship between the pupils’ prior attainment and their final examination results.

Figure 11 is derived from exactly the same data as Figure 10, but the ‘Box and Whisker’ plot emphasises that strong relationship between prior attainment and final examination results.

That Figures 10 & 11 illustrate a strong relationship between the State Matriculation Examination in Bulgarian Language and Literature and prior attainment is important because all pupils in the school are entered for this examination.

Figures 12 & 13 Illustrate the analogous information for the ‘Second Subject’ in the State Matriculation Examinations. In Figure 12, the results in the State Matriculation Examination in the ‘Second Subject’ are plotted against the Scores on the Admission Criterion. The ‘Regression Line’ included in Figure 12 reinforces the immediate impression that there is a strong relationship between the pupils’prior attainment and their results in their ‘Second Subject’. Figure 13 is again derived from exactly the same data as Figure 12, and again the ‘Box and Whisker’ plot emphasises that strong relationship between prior attainment and final examination results.

Figures 12 & 13 have not distinguished between the ‘Second Subjects’ in the State Matriculation Examinations. Figure 14, on the other hand, clearly identifies the individual subjects illustrated in Figure 12. In particular, Mathematics is highlighted; the subject that was identified earlier in Figures 4, 5 & 6. There appears to be at least two clearly identifiable clusters of pupils:

– those who are admitted to the school with high scores on the Admissions Criterion and do very well in the State Matriculation Examination in Mathematics; and – those who are admitted to the school with much lower scores on the Admissions Criterion and do badly in the State Matriculation Examination in Mathematics.

This is reinforced by Figure 15, in which the data points are replaced by separate ‘Regression Lines’ for each subject. Again, the ‘Regression Line’ for Mathematics is anomalous and reinforces the idea that pupils with high prior attainment do well in Mathematics but pupils with low prior attainment do badly.

Figure 16 focusses on the pupils’ prior attainment and illustrates quite clearly that the pupils who were entered for the State Matriculation Examination in Mathematics had the widest range in prior attainment with a heavy loading of pupils with low prior attainment.

Figures 14, 15 & 16 provide an alternative perspective to that suggested by Figures 4, 5 & 6; rather than raise questions about the quality of teaching, they raise questions about the pupils’ decision-making processes. Only 12 pupils from 12а, the Mathematics Profile class with the highest prior attainment chose to enter the State Matriculation Examination in Mathematics; i.e., just less than 50% the class. At the same time, these 12 pupils made up less than 40% of those entered for State Matriculation Examination in Mathematics. To reinforce this point:

– the 12 pupils from 12а had an average score on the Admissions Criterion of 446.92 and an average of 5.29 in State Matriculation Examination in Mathematics;

– the 10 pupils from 12б had an average score on the Admissions Criterion of 418.75 and an average of 3.85 in State Matriculation Examination in Mathematics;

– the 9 pupils from 12в had an average score on the Admissions Criterion of 412.03 and an average of 4.35 in State Matriculation Examination in Mathematics.

The discontinuity illustrated in Figure 14, and the steep ‘Regression Line’ illustrated in Figure 15 are features of Mathematics that are common to Education Systems across the world. Being good at the end of one stage; for example, in the Year 7 National Tests in Mathematics in Bulgaria, is no guarantee that you will be successful at the end of the next stage; for example, in the State Matriculation Examinations in Mathematics.

You have to be very good pupil at the end of stage ‘m’ to succeed at stage ’m+1’. Pupils still have the right to choose, but before they make their choice, they have the right to know that a score of about 440 is needed on the Admission Criterion in order to do well in the State Matriculation Examinations in Mathematics.

A comparative analysis across schools – revisited

The previous section illustrated just how important, at an individual pupil level, the prior attainment scores are in determining the outcomes in the State Matriculation Examinations. In revisiting a Comparative Analysis Across Schools, this section analyses the possible impact of prior attainment across a municipality.

Figure 17 illustrates the results in Year 7 National Tests of Bulgarian Language and Literature of the pupils entering Secondary Education: the dark area shows the results for the pupils entering the Mathematics Grammar School and the light area shows the results for the pupils entering all the other schools in The Municipality. Two things are immediately apparent from Figure 17:

1. the Mathematics Grammar School admitted a very high proportion (almost 45%) of the pupils with the highest prior attainment scores in Bulgarian Language and Literature; and

2. 272 pupils in The Municipality (almost 17%) failed to score on the Year 7 National Tests in Bulgarian Language and Literature.

Figure 18 illustrates the analogous information for the National Tests of Mathematics; again, two things are immediately apparent:

1. the Mathematics Grammar School, not surprisingly, admitted an even higher proportion (over 75%) of the pupils with the highest prior attainment scores in Mathematics; but

2. 441 pupils in The Municipality (almost 27%) failed to score on the Year 7 National Test in Mathematics.

It was pointed out earlier that the school admits pupils on the basis of a score derived from the National Test data and the Primary Certificate of Education. Figure 19 illustrates the result of applying that algorithm to all the pupils in the Municipality; and again, two analogous things are immediately apparent:

1. the Mathematics Grammar School admitted 104 pupils (over 45%) with the highest scores on the Admissions Criterion;

2. almost 250 pupils (15%) barely registered on that Admission Criterion.

Between them, Figures 17, 18 & 19 issue a warning on over-interpreting the data on relative performance across The Municipality that is illustrated in Figure 7. Certainly, the pupils at the Mathematics Grammar School achieved the highest Average Score in Bulgarian Language and Literature. However, given the data on prior attainment just presented, should the pupils have made more progress or was their progress higher than would have been predicted?

The same arguments apply to the relative performance of all the Mathematics Grammar Schools illustrated in Figure 8; however, without the Ministry of Education and Science publishing comparative data for the whole country, School Directors cannot really know how well the school is performing.

A detour into low attaining schools

The reviews presented above have concentrated on the relative performance of high attaining pupils in high attaining schools. Similar reviews of uncontextualized performance data inevitably favour the highest attaining schools and frequently describe them as ‘the best or most desirable’ schools. Like similar reviews, little attention was paid above to low attaining pupils in low attaining schools. This short detour into low attaining schools will resist the temptation to label them ‘the weakest schools in Bulgaria’ (Nikolov 2023) with the implication that they are failing schools, with failing school directors and failing teachers.

Figures 17, 18 and 19 provide different measures of the prior attainment of pupils across The Municipality. The structure and organisation of schools within The Municipality will ensure that some schools, predominantly vocational schools, will have to admit a very high proportion of very low attaining pupils.

Figure 20 is derived from the same data set as Figure 10, but the ‘Regression Line’ has been extrapolated to provide an indication of the likely results of pupils with very low prior attainment in Bulgarian Language and Literature. That is to say, it suggests that it is very unlikely that pupils with less than 150 on the Admission Criterion will achieve above 2.0 in the State Matriculation Examination in Bulgarian Language and Literature.

Figure 21 is similarly derived from the same data set as Figure 12, with a similar extrapolation of the ‘Regression Line’. In this case, it suggests that it is very unlikely that pupils with less than 200 on the Admissions Criterion’ will achieve above 2.0 in the State Examination in Mathematics.

Figures 20 & 21 suggest that, if some schools have to admit a very high proportion of pupils with very low prior attainment, those schools will have very low Average Results in the State Matriculation Examinations. Although it is inappropriate to label these schools ‘weak schools’, Nikolov (2023) is correct in concluding that these ‘schools in Bulgaria highlight the systemic failure of education’. Furthermore, it is inappropriate to hold individual professionals accountable for systemic failure.

The same arguments apply to the lower attaining Mathematics Grammar Schools in Figure 8, but lack of any national data on prior attainment makes it impossible to construct those arguments rigorously.

The way forward

There is a wide range in the ability of young people in Bulgaria. This, in turn, means that:

1. with a wide range in their size and socio-economic characteristics, there is a wide range in the average prior attainment of similar schools; e.g., the Mathematics Grammar Schools in different municipalities;

2. the structure and organisation of the school system within municipalities means that there is also a wide range in the average prior attainment of schools within each municipality; and

3. there is even a wide range of ability within individual schools.

Similarly, there is no clear evidence on:

1. whether the standards in the State Matriculation Examinations are consistent over time; or

2. whether the standards in the State Matriculation Examinations are even consistent across subjects.

This means that none of the strategies currently available to School Directors: a) the longitudinal or trend analysis;

b) the comparative analysis within the school;

c) the comparative analysis across schools; and d) the ipsative analysis, the assessment of pupils’ progress or value-added by the school can provide a clear indication of whether the standards of achievement are low, average or high.

When it publishes anything, the Ministry of Education and Science only publishes uncontextualized performance data. Therefore, this all that School Directors and the local and national media have to rely on.

In order to address the issues raised above, the first task would be to construct a Common Measure of Prior Attainment for which the Ministry of Education and Science already has all the data necessary.

Its function differs from that of the Admission Criterion used by Mathematics Grammar Schools and it must be the same for all pupils and all schools, for example:

– 2* the score in the Year 7 National Test in Mathematics;

– 2* the score in the Year 7 National Test in Bulgarian Language and Literature;

– the score in Mathematics in the Primary Certificate of Education; and

– the score in the Bulgarian Language and Literature in the Primary Certificate of Education.

Once this Common Measure of Prior Attainment has been developed, the Ministry of Education and Science must agree on Grade Boundaries for the whole range of prior attainment; for example:

<100, 100-132, 133-165……366-399, 400-432, 433-465, ≥466

On an annual basis, it must determine the Average Grade in every subject for which there is a State Matriculation Examination for the pupils who scored; for example: 366-399, 400-432, 433-465 and ≥466 on the Common Measure of Prior Attainment.

In this way, the Ministry of Education and Science must complete and publicise the information in Table 4 for the full range of Prior Attainment and for all subjects for which there is a State Matriculation Examination. Only with such information available can School Directors compare the progress of pupils in their school with the average progress nationally of pupils of similar prior attainment in each subject.

Table 5 presents simulated national data alongside real data for the school. If the national data were real data, the School Director would be able to conclude that:

1. the results in the State Matriculation Examination in Bulgarian Language and Literature of the pupils with lower prior attainment were above the national average; but

2. the results of the pupils with higher prior attainment were below the national average.

At the same time, the School Director would be able to conclude that:

1. the results in the State Matriculation Examination in Mathematics for pupils with higher prior attainment were in line with the national average; and

2. although the results for pupils with lower prior attainment were well below the results of pupils with higher prior attainment, they were in fact above the national average for similar pupils.

If the results illustrated in Table 4 were available for:

a. all subjects for which there is a State Matriculation Examination; and b. for pupils across the range of prior attainment

School Directors would finally be able to determine the strengths and weakness in their school.

As the system develops, the Average Grade in Table 5 could be replaced with the Minimum, Lower Quartile, Median, Upper Quartile and Maximum Grade. The School Director might then be able to determine that, not only was the school above average; but for pupils with very high prior attainment, it was in the top quartile of similar schools in the country.

Only then, will it be “really possible for School Directors to know how well their school is performing”.

Using this approach, the results should be public, disseminated, analysed, used for policy and strategy, and comparable over time and grade. This strengthened assessment could measure progress, school value-added, and to determine the ability of parents and local authorities to hold principals accountable.

NOTES

1. Nor, so it would seem, across subjects!

2. The pupil who did not turn up for the examination has been excluded from the analysis.

REFERENCES

BUDGELL, P., 2023. Key performance indicators–input, intervening and output variables. Vocational Education, vol. 25 no 2, pp. 120 – 143.

NIKOLOV, A., 2023. The weakest schools in Bulgaria highlight the systemic failure of education. Institute of Market Economics. 13-07-23.

WORLD BANK, 2012. A Review of the Bulgaria school Autonomy Reforms. (Report No. 54890-BG). Washington DC: World Bank.