Резюме. According to the outbreak of COVID-19 pandemic situation, the educational system was one of the sectors that had to move very quickly and adapt the learning process, and millions of young people to complete their education in an online environment, others to move to a higher educational level, and of course there are thousands of specialists who ask the question “was this education really effective?”. The transition to entirely online form of education was necessitated by the anti-epidemic measures in connection with COVID-19, and due to the uncertainty in controlling the virus, will probably continue in 2020/21 academic year. The aim of this study is to check whether there is a change in the opinion of experts from the National Sports Academy “Vassil Levski” (NSA) regarding the indicators for assessing the effectiveness of the online learning at the moment of online education, compared to 2016 theoretical model. 73 teachers from NSA gave their opinion on the criteria presented in a theoretical model developed and evaluated in 2016, in a previous study. Various statistical methods were used to analyze the results. The results that were derived after the processing showed a significant rearrangement of the factors (indicators) for evaluating the effectiveness and greater concentration and attention on the selected criteria. In conclusion, it could be summarized that the factors influencing the evaluation of effectiveness are several and they are all significant and of particular importance. There is a need a lot of work to be done to improve the communication in the online platform, especially for online video conferencing with many participants, which should be integrated into the platform so as to create a convenient, easy and with many functional possibilities for work and study in a virtual environment.

Ключови думи: COVID-19; experts; quality; effectiveness; indicators

Introduction

With the emerging coronavirus pandemic (COVID-19), the education system was one of the sectors that had to move very quickly and adapt the learning process for both schools and universities to comprehensive online learning. The rapid spread of the disease across a wide geographic area prompted the World Health Organization1) to declare COVID-19 a pandemic on 11 March 2020. Most schools from basic to universities have shut down their doors and students have returned home to their parents and together self-quarantined2) . According to UNESCO, over 1.5 billion learners in 165 countries are attracted by COVID-19 school closure. This translates to 87% of the world’s student population. Technological literary is one of the required skills in the current knowledge-based society and ICT and e-learning have a great impact on the educational processes and systems, researches and learning initiatives, especially within higher education institutions. (Pavel, Fruth & Neacsu, 2015)

In this regard, the evaluation of the effectiveness of the education will be a main topic, which will be on the agenda for a lot of educational institutions for at least a few more years. Millions of young people have had to complete their education in an online environment, others have to go to a higher educational level, and there are many specialists who ask themselves the question “Was this training really effective?”. In an analysis by the World Health Organization3) (2020b), the transition to this form of education was necessitated by the anti-epidemic measures in relation to COVID-19, which also affected the education system worldwide, and due to uncertainty in controlling the virus is likely to continue during the 2020/21 academic year. Similar is the conclusion of Laura E. Rumbely (2020), who says that “Clearly, we are in the first stages of a situation with no obvious resolution in the near term”.

This also requires updating the criteria for evaluating the effectiveness, gaining a clearer vision of how the global education system is doing and what benefits could be generated.

In terms of the effectiveness of online learning and its evaluation, many working evaluation models could be presented, as well as many models that present a general picture of evaluation of effectiveness. Evaluation of online learning has been the subject of study for decades, analyzing not only learning but also teaching. Some reports provide a good number of reasons as to why students are likely to learn effectively through online studies. According to the reports, students have more control over their studies and have more opportunities at their disposal for reflection. It is reported that successful online students tend to be organized and are self-starters who can accomplish their work without close supervision (Picciano, 2017; Wang, Pi & Hu, 2019).

When we talk about the quality of e-learning, it is the quality of the learning process itself, its management and the professional realization of the learners. According to the standard ISO / IEC 9000 “Quality is the degree to which a set of inherent characteristics meets the requirements”4) .

According to the legal definition in the Bulgarian legislation (the additional provisions of the Law on the National Audit Office), the effectiveness is the level of achievement of the objectives when comparing the actual and expected results of the activity. This is very close to the international standard ISO 9000, which says, “the degree to which the planned activities are implemented, and the planned results are achieved”. Effectiveness is related to the appropriateness of actions. It answers the question of whether the “right” things are being done.

There are many organizations such as British quality Assurance Agency, Norwegian Association for Distance Education and Flexible Education, Council for Higher Education Accreditation, European Association of Distance Learning, International Network for Quality Assurance Agencies in Higher Education, European Foundation for Quality in eLearning, UNESCO, that develop systems, standards and methods for assessing the quality of e-learning, distance, web-based and mobile learning in higher education.

The efficiency of the functioning of the e-learning system is its ability to achieve the set goal with a predetermined quality and with the available resources. Efficiency is the measure that characterizes the process of functioning of the interconnected elements in the e-learning system for achieving the set goal, determined by the desired result of the functioning of the system. One of the most important goals of higher education is to achieve the highest possible quality of education. It does not consider the costs (energy, labor, financial resources), but only the achievement of the goal. The higher the degree of achievement of a goal is, the more effective the actions and activities are (Halachev, 2012).

Based on a study of a number of models for evaluating the effectiveness, in 2016 at the National Sports Academy “Vassil Levski” was developed a theoretical model to assess the effectiveness of the learning process according to the web-based platform of NSA “Vassil Levski”, in which experts in the field of sports education defined criteria for evaluating the effectiveness of online training for NSA (Kuleva, 2017).

In the presented model, 8 indicators were identified, which in the opinion of the specialists from the National Sports Academy “Vasil Levski” could assess the importance of the factors influencing the effectiveness. The basis of the present study is exactly this theoretical model for evaluation, which was developed in 2016 in a previous study (Kuleva, 2016), and our main goal is to compare the factors/ indicators identified by experts in 2016 compared to those that were derived in 2020, during the online training due to COVID-19 outbreak.

Methodology

Participants

The research was done among the academic staff from the National Sports Academy “Vassil Levski”, as the questionnaire was sent online to 80 experts, of which 74 questionnaires were completed, of which 73 were valid. The gender distribution of the experts was 26 men and 47 women, aged between 28 and 68 years. The completion of the questionnaire was anonymous and voluntary and took place between 4 and 29 May 2020. The consultation period was chosen to be realized at the end of the second semester, as the goal was for the maximum number of experts to work with the distance learning system of NSA, as well as with other online video learning systems or applications, and to gain work experience in an online environment with students.

Statistical analysis

The results from the study were subjected to mathematical-statistical processing with SPSS 23; depending on the tasks of the research the following statistical methods were applied:

– Descriptive statistics – for defining average values, normality of distribution and variety of indicators under study;

– Correlation analysis between the indicators for the 2020 survey;

– Factor analysis – calculating the ranking with weight factors (3, 2 and 1);

– Comparative analysis by t-criterion of Student for independent samples – for checking the reliability of differences between average levels of indicators under study between both groups. The critical value of the t-criterion, at high statistical reliability (Pt≥95%), is t-critical=1.98. (Gigova, 2002);

– Comparative analysis of two independent samples with Man Whitney U-criterion on quantitative indicators with distribution of values different from normal.

We compared the defined criteria from 2016 and those that were derived in 2020 and the results were managed using the abovementioned statistical methods. To define the indicators, the calculated sum of the frequencies was multiplied by the corresponding weight factor (3, 2 or 1), depending on the ranking order of the proposed indicators (Levy, 2006).

Results

Table 1 presents the comparison of the indicators specified by the experts in the two studied years – 2016 and 2020. There are some shifts in the choice of indicators, but there are also indicators that have kept their position from 4 years ago. In the survey conducted in 2020, 6 out of a total of 15 indicators received most of the points, and there is a clear between the number of points of the first 6 indicators and the 7th and 8th in the ranking (8th and 13th indicators). Rather, they remain close as a result to the next ranked criteria. This highlighting of only 6 indicators could mean that the experience gained over the years, as well as the active use of distance learning in the period February – May 2020 by experts, has greatly changed their opinion and criticism of this form of learning.

The change in the derived criteria to determine the effectiveness of distance learning in NSA “Vassil Levski” from 2020 could be confirmed by Table 2.

The analysis of the results from 2016 shows, that the observed differences of the experts’ opinions, the dispersion of the individual estimates around the average for each indicator is relatively stable. The same could be said for the results of 2020.

Indicator 1 – “Students' satisfaction with the learning process in the platform” takes its place in the first six important indicators for the evaluation of efficiency for 2020, compared to its place (outside the eight criteria) in 2016. This is also confirmed by the value of the t-criteria (t= 4.01, P=99.99).

Indicator 2 – „Usefulness of the acquired knowledge according to the students” also undergoes a change in the position, as from 7th position in the ranking in 2016, it is determined to be the most important among the indicators in 2020 with 70 points. This change of position is also confirmed by the value of the t-criteria (t= 2.65, P=99.07).

Indicator 7 – „Online communication between student and lecturer” also increases its importance from 7th to 2nd place among the indicators. This is one of the indicators that is especially important for many students and teachers and to some extent is related to Indicator 9 – “Activity of students in the platform”, with a strong correlation between them r = .625 **, at α = .00

It is noteworthy that Indicator 6 – “Presentation of educational materials in the platform” does not keep a high position in 2020, but it’s placed almost to the last position. This result of the indicator could be due to the fact that quite a small number of teachers were able to conduct online video lectures during the training period in a completely online environment. Most of them used a mixed model, sending materials, work assignments and less video communication. Although indicators 9, 10, 11 and 14 show statistically significant differences from the two studies, they generally retain their positions from 4 years ago, with an increase in the degree of importance, but are less preferred in the grading of the indicators.

Indicator 10 – “Attrition from the learning process” remains unchanged and rather it remains misunderstood and underestimated, but in fact of great importance for the overall learning outcome. По тематиката, която разглежда възможно най-удачната форма за обучение на студенти, които са спортисти пише и Galimov, et al, 2019. They present an algorithm of content integration of educational process and sport activity for a certain student-athlete in the framework of a certain academic course. (Galimov, et al, 2019)

Looking at indicator 14 – “Theoretical focus of the material” it is noted that the average values are in favor of the study of 2020, explaining the fact that many teachers still rely on good theoretical presentation of the study material to students can understand the theory of the studied subject (U=2.41l α=.02).

Discussion and conclusion

The highlighting of the 6 indicators for 2020, compared to the 8 from 2016, shows that distance learning is getting more understandable and accessible to teachers and they have a clearer opinion, as the choice of criteria with the greatest weight are more focused, without distraction between the indicators. This could be due to the fact that teachers were teaching without an active face-to-face communication with their students and the three top indicators that were selected by the teachers, confirm that fact very clearly. According to them, it is of paramount importance that students are satisfied with the acquired knowledge and their usefulness, and to maintain active communication between the two parties – teacher-student. A study conducted by Borukova and Kotev, 2019, related to long-term goals and prospects with students from Bulgaria and Serbia, found that the communication is a very important factor. (Borukova, Kotev, 2019). Moreover, according to Doncheva, (2015, 2016), the use of distance studies platforms and different web-based courses offer an innovative teaching method which increases students’ participation motivation and enhances their interest towards the particular subject.

The indicators indicated in the theoretical model refer only to the educational process conducted at the National Sports Academy “Vassil Levski”. It could not be said without further research that this is a complete list of indicators that are universal or appropriate for assessing the effectiveness of any educational program. Rather, in this case, we focus on those that are applicable at the level of the educational program that could be taught through the NSA distance learning platform. These indicators could be useful to experts responsible for program management or those responsible at the highest level for monitoring individual programs, the quality of which must be maintained, , but it is advisable to first check whether they could to be adapted to the relevant online learning system.

Performance evaluation indicators provide a basis for evaluating the quantitative efficiency of a system. Although they are based on the same data that serve as management information, they are evaluation criteria clearly related to the objectives of the program or specialty being evaluated.

The results obtained, whether weak or significant, should provoke further studies. Professional evaluation of all data must dominate. Only in this way the quality and the effectiveness of the education, which are so important for the future learners, could be guaranteed. Marinov & Uzunov, (2014) has reached a similar conclusion in their article, concerning the online education and its management.

The topicality of the curriculum, the quality of teaching and the platform itself are also factors with significant weight, which gives reason to believe that teachers sustain a good level of high quality teaching and the choice of teaching platform (in this case, nearly 60% of teachers have chosen the system for distance learning of NSA “Vassil Levski” as the main) is a good attestation for the online education.

In conclusion, it could be summarized that the factors influencing the evaluation of efficiency are several and they are all significant and of particular importance. There is a need the communication in the online environment, especially through online video conferencing applications with many participants to be improved. Such integration into a learning platform (including the online learning platform of NSA) will create a prerequisite for creating a convenient, easy and with many functional possibilities for work and study online platform for distance learning.

Whether this COVID-19 pandemic was crucial to a change in the higher education system remains to be seen, but it could be said that solid foundations have been laid for its development. From now on, the direction is rather clear - online learning will be more common and preferred, not only by students but also by teachers themselves. The experience and confidence that teachers have generated in these few months is significant and it would not be good to lose this accumulated experience, but rather to develop it to wider horizons, with or without a pandemic.

NOTES

1. World Health Organization. (2020a). WHO Director-General’s opening remarks at the media briefing on COVID-19 – 11 March 2020. Retrieved on 13 June 2020 from https://www.who.int/dg/speeches/detail/who-director-general-sopening-remarks-at-the-media-briefing-on-covid-19---11 -march-2020

2. UNESCO. (2020). Global Education Coalition-290-million students out school due-COVID-19. Retrieved July 10,2020 from https://en.unesco.org/ news/290-million-students-out-school-due-covid-19-unesco-releases-firstglobal-numbers-and-mobilizes

3. World Health Organization. (2020b). Report of the WHO-China Joint Mission on Coronavirus Disease 2019 (COVID-19). The WHO-China Joint Mission on Coronavirus Disease 2019, February 16-24. Retrieved from https://www.who. int/docs/default-source/coronaviruse/who-china-joint-mission-on-COVID-19final-report.pdf

4. International standard ЕN ISO 9000. (2015). Quality management systems – Fundamentals and vocabulary.

REFERENCES

Borukova, M. & Kotev, Vl. (2019). Research of the motivation and long-term objectives of Bulgarian and Serbian students, Pedagogical Almanac, VTU, issue 2, volume 27 pp. 273 – 278, http://journals.uni-vt. bg/almanac/bul/vol27/iss2/18

Doncheva, L. (2015). An implementation of a hybrid form of education in foreign language acquisition. Activities in Physical education and Sport. International Journal of Scientific and Professional Issues in Physical Education and Sport. Vol. 5.

Doncheva, L. (2016). Izsledvane na efektivnostta na inovativnata Sistema za prepodavane na chujd ezik sred student-sportisti. VIII International Scientific Conference “Modern Trends of Physical Education and Sport”, “St. Kliment Ohridski”, pp.230

Galimov, A., Botova, L., Nazarenko, A. & Galyautdin, M. (2019). Model of integration of formal and informal education for student-athletes, Journal of Applied Sports Sciences, Vol.2, DOI: 10.37393/jass.2019.02.8

Halachev, P. (2012). Predicting the Effectiveness of e-Learning by Neural Networks. Cybernetics and Information Technologies, Vol. 12, No 2.

Kuleva, M. (2016). Upravlenie na interaktivna platforma za obuchenie na specializirani sportni kadri. Dissertation, Sofia. [In Bulgarian]

Kuleva, M. (2017). Determining Criteria for Evaluating the Efficiency of the Education in a Distance Learning Platform, Proceeding book of the International Scientific Congress “Applied Sports Sciences” 1 – 2 December, pp.424 – 427, doi:10.37393/icass2017/87

Marinov, G., Uzunov, St. (2014). Capabilities and Development of The System for Distance Learning in Security and Defence Logistics at G. S. Rakovski National Defence College. Proceeding book from Scientific Conference “Improvement of higher education management systems”, Veliko Tarnovo, 13 – 14 Nov, pp.160 – 165.

Pavel, A., Fruth, A. & Neacsu, M. (2015). ICT and E-Learning – Catalysts for Innovation and Quality in Higher Education. Retrieved July 15, 2020, from https://www.sciencedirect.com/science/article/pii/ S2212567115004098

Picciano, A. G. (2017). Theories and frameworks for online education: Seeking an integrated model. Online Learning Journal, 21(3). https:// doi.org/10.24059/olj.v21i3.1225

Rumbley, E. L. (2020). Coping with COVID-19: International higher education in Europe. The European Association for International Education (EAIE).

Wang, H., Pi, Z. & Hu, W. (2019). The instructor’s gaze guidance in video lectures improves learning. Journal of Computer Assisted Learning, 35(1), 42 – 50. https://doi.org/10.1111/jcal.12309