Образователни технологии
LEARNING ANALYTICS TOOL FOR BULGARIAN SCHOOL EDUCATION
https://doi.org/10.53656/math2021-4-2-lea
Резюме. Nowadays, schools use many information systems to automate their activities for different stakeholders’ groups – learning management systems, student diary, library systems, digital repositories, financial management and accounting systems, document processing systems, etc. The huge amount of data generated by the users of these systems, led to increased interest in the collection and analysis of data to encourage students to achieve higher results, teachers to provide personalized support and school managers to make data-driven decisions at all levels of school, and stimulates research into the application of Learning Analytics (LA) in schools. The paper presents a LA model and a software prototype of the LA tool designed for the needs of Bulgarian school education from the perspective of different stakeholder groups (students, teachers, class teachers, parents, school managers, inspectors from evaluation agencies), aiming to improve school methods of approaching and analyzing learning data. The tool allows stakeholders to track data for students’ learning or training for different purposes, e.g. monitoring, analysis, forecast, intervention, recommendations, etc., but finally to improve the quality of learning and teaching processes. Research and experiments with the model and the LA tool under consideration are conducted based on the information infrastructure of a typical Bulgarian school.
Ключови думи: learning analytics; intelligent data analysis; learning data; learning analytics models; data collection; data analysis; software prototype; school education; educational improvement
Introduction
In recent years, extraction and analysis of data, produced by participants in learning processes, have become increasingly important and has led to the emerging of a new research field, called Learning Analytics (LA). LA refers to the process of collecting, evaluating, analyzing, and reporting organizational data for decision making (Sclater 2017) to improve learning processes (Campbell et al. 2007), learning efficiency and effectiveness in primary, secondary and post-secondary education (Siemens et al. 2011).
Modern schools use many software systems to automate their activities for different stakeholders’ groups (e.g. learning management systems – LMS, student diaries, etc.), which collect a huge amount of data (incl. about students and their achievements). For example, when conducting e-learning process, data, generated by students’ activities in the system (e.g. assignment submissions, answering quizzes, participation in group discussions, etc.) are stored in the LMS database. The teacher records each student’s personal data, all current marks, term marks and annual marks, absences. These data can help schools to improve the quality of courses and teaching methods (Avella et al. 2016), track students’ performance, identify students who need support, help managers with data-driven decision-making (Conde & Hemandez-Garcia 2015), promote the continuous improvement of schools’ educational processes (Andrade e Silva & Camanho2017). LA tools provide stakeholders (students, teachers and decision-makers) with actionable insight to classroom and course level activities, better information and deep insight into the factors within the learning process that contribute to learner success (Siemens et al. 2011).
Many schools worldwide have already used LA tools. Successful attempts for implementing LA tools have been conducted in Australia (Varanasi et al. 2018), the USA (Macon et al. 2016), France (Brun et al. 2019), Spain (Sancho et al. 2015), Denmark (Ferguson et al. 2016; Andersen 2017), Uruguay (Macarini et al. 2019), Netherlands and Norway (Ferguson et al. 2016). Many of these tools are developed for the needs of students and teachers and provide them with improved indicators to measure the effectiveness of teaching methods, learners’ engagement in the LMS, and the effectiveness of the learning process using technology. Less attention has been paid to their use by other stakeholders (e.g. school managers, inspectors form evaluation agencies, etc.).
The paper is devoted to the application of LA Techniques in Bulgarian school education. It presents a LA model and a software prototype of the LA tool designed for the needs of Bulgarian school education from the perspective of different stakeholder groups (students, teachers, class teachers, parents, school managers, inspectors from evaluation agencies), aiming to improve school methods of approaching and analyzing learning data. The tool allows stakeholders to track data about students’ learning or training for different purposes, e.g. monitoring, analysis, forecast, intervention, recommendations, etc., but finally to improve the quality of the learning and teaching processes. Research and experiments with the model and the LA tool under consideration are conducted, based on the information infrastructure of a typical Bulgarian school.
LA Model for Bulgarian School Education
The model for LA, appropriate to the needs of Bulgarian School Education, is built after a comprehensive literature review and thorough analysis of requirements for quality evaluation of learning in school education. The full process of building the model is presented in a previous study (Gaftandzhieva et al. 2021). Herein, it is summarized for the sake of completeness of the presented study.
On the first step, as the needed information depends on an individual’s role and responsibilities in the school (Pardo et al. 2016), before designing the LA tool, different stakeholder groups are identified (students, teachers, class teachers, parents, school managers, inspectors from evaluation agencies).
After this, it was done a deepen literature review of related studies to determine whether LA tools can be useful for each stakeholder group, results of which are presented in Table 1.
Table 1. Benefits for stakeholders
In the third step, the software systems, used in a typical Bulgarian school and stored data, were analyzed:
– Software systems for timetable development – data about the weekly schedule (subjects, days and hours of classes, teacher, classroom number);
– LMSs – data described students’ activities, such as assignment submissions, answering quizzes, participation in group discussions, reads of learning resources, etc., are stored in the LMS database;
– Electronic diary – students’ current grades, term marks and annual marks in each subject, absences, etc.;
– Student diary – personal data of each student, current grades, term marks and annual marks in each subject, absences, etc.
In the last step, a review of the requirements of all normative documents, which regulate the need to collect data on the learning process, was performed. The most important document is the Inspection criteria of the National Inspectorate of Education (NIE), which require evaluation, based on the data, generated during the learning process.
After these four steps, a model with a set of indicators for data collection that serve as a business logic basis of the developed LA tool (see Section 3) is built.
The model defines what type of data should be collected from the institutional information infrastructure that relevant stakeholders can use to improve the quality of education and teaching. It consists of measurable indicators, which allow the stakeholder to track data about students’ learning or training for different purposes, e.g. monitoring, analysis, forecast, intervention, recommendations, etc., and finally to improve the quality of education and teaching. The model is built as hierarchies of measurable indicators of two levels. The model contains five Indicators at Level 1, which represent the activity/subject to which the collected and aggregated data relate – 1. Students’ activity, 2. Teachers’ activity, 3. Control of the school schedule, 4. Students’ success, 5. Quality of learning. These indicators group together a set of forty indicators at Level 2 that allow the relevant stakeholder to track data in specific activities of that type. Each indicator at Level 2 contains a set of measurable attributes at Level 3, whose values are extracting from the software system, used in the school (e.g. number of visits of school materials and activities, assignments submitted, test results, grades, etc.). Figure 1 presents the LA model and its indicators at Level 1 and Level 2.
Figure 1. LA Model
Figure 1 shows the groups of indicators, and which indicator could be monitored by the members of each of the identified stakeholder groups. Table 2 lists the numbers of the specific indicators for which each stakeholder group can obtain values. The data for each indicator’s evaluation, which was gathered, is different for each stakeholders’ group (e.g. different data are gathered for Indicator 1.1 for each stakeholder group – student, teacher, class teacher, parent, inspector).
Table 2. Indicators for stakeholders
LA Tool Description
Based on the verified model (Gaftandzhieva et al. 2021), a corresponding software system, called Learning Analytics Tool for School Education (LASchool), was designed and implemented.
As a result of an analytical review of software solutions for extracting, analyzing and visualizing data from various information sources, the technologies and tools for software development were selected. The LASchool is developed by the integration of existing software solutions, namely JasperReport Server, Jaspersoft ETL and JasperSoft Studio tools1), and the client application.
The JasperSoft Studio provides a rich set of instruments for designing report templates that can be filled out with data, retrieved from a variety of data sources (relational databases, big data sources, or other types of database systems). Along with JasperReport Server, it can be used to create powerful report publishing workflows.
JasperReport Server provides opportunities for organizing structured repositories, accessing data collections in different types of organization (incl. custom ones – DB, XML, CSV, Hibernate, POJO) and using them as data sources for the needs of JasperSoft Studio when generating reports, storing reports and presenting them in the preferred by the user form. The server also offers powerful tools for integration with various software applications through shared web services.
Jaspersoft ETL (extract, transform, and load) is the most flexible, powerful, and affordable open-source tool for data integration requirements. It is designed to support developers while scaling to the highest levels of data volumes and process complexity. JasperSoft ETL is easy to deploy and it is used to extract data from a transactional system to create a consolidated data warehouse or data mart for reporting and analysis. It allows developers to design graphically, schedule, and execute data movements and transformations for business intelligence projects, such as loading an Operational Data Store (ODS), Data Mart, or Data Warehouse.
The architecture of the LASchool (see Fig. 2) follows the standard type of 3-tier architecture with well-known three layers – Presentation, Application and Data layers.
The Client application allows users to request the generation of a report by a chosen template and to view the result of the request (visualized report). The Client application (using XML Parser and Style Control Module functionalities) allows users (students, teachers, class teachers, managers, parents, inspectors), through predefined conditions, to modify some view attributes such as color, font size, etc., to visualize the report in the web browser in a user-friendly way.
By the report templates design tool JasperSoft Studio, it is implemented the core functionality of the Application Layer of LASchool and its business logic.
Figure 2. LASchool Architecture
The key element of this functionality is the modelling of the developed LA model for the needs of each stakeholder group (see Section 2) and acquisition of values for the model’s indicators at different levels from digital footprints, left by students and/or teachers during training in each course in LMS, electronic diaries, student diary and/or other systems of the school information infrastructure.
Therefore, in the first stage, the institutional information infrastructure (incl. school digital repository, databases of school software systems, LMS, etc.) of a typical Bulgarian school has been analyzed. The analysis has been done in terms of its use as a data source (about the training, the results achieved, etc.) when forming values of the indicators from the proposed model. Then, all data sources are integrated through JasperSoft ETL.
In the second stage, templates of reports were designed using JasperSoft Studio based on the proposed model as sets of indicators (see Figure 1 and Table 1) for the needs of each stakeholder group. All developed templates of reports have been stored on the JasperReport Server. JasperReport Server plays an intermediate role between the three architectural layers:
– The Client Application requests the REST services of JasperReports Server to run a chosen template and generate a report through the Service Client;
– The JasperReports Server Web Service interface responds to HTTP requests by the client application.
Data Layer of the LASchool includes various databases of the school information infrastructure (Moodle LMS, electronic diaries, student diary, etc.) as well as the JasperReports Server repository itself. JasperReport Server addresses them to retrieve the necessary data when generating reports.
The benefits of LASchool allows all participants with a certain interest in learning (students, teachers, class teachers, managers, inspectors, parents) to follow the training process and to take timely corrective measures that can improve the quality of the training (even before the completion of the student’s education). LASchool allows, for each indicator of the proposed model, reports to be generated with values retrieved from the information systems depending on the user’s role in the tool.
In this way, the generated report for different stakeholder’s group provides different data (retrieved from software systems) depending on the group to which the user belongs (student, parent, class teacher, teacher, manager, inspector). This is possible as the indicators at Level 1 and Level 2 are the same for different stakeholder groups, but they differ at the lower levels and this is embedded in the designed report templates in LASchool. For example, for the Indicator 4.3. Tendencies in students’ success, the related data sources for acquisition of values of the indicators of Level 3 and the indicators/values themselves for each user role will be different (see Table 3).
Table 3. Indicators of Level 3 for each user role
As a result, the generated reports for different stakeholder groups provide different data (retrieved from the information systems) depending on the user’s role in LASchool (Teacher, Class Teacher, Manager, Parents, Inspector as mentioned above). Generated reports contain tables and diagrams and allow users to perform various analysis on the retrieved data. For example, LASchool allows managers to monitor and assess the quality of the learning process and the results achieved by students and teachers in all school subjects from all study programmes, as:
– Monitor teachers’ activity in their communication with students and parents (Indicator 2.1);
– Monitor timely feedback from teachers (Indicator 2.2);
– Monitor teachers’ preparing and publishing curriculum (Indicator 2.3);
– Monitor teachers’ preparing current assessment schedules (Indicator 2.4);
– Monitor teachers’ timely marking of students’ absences (Indicator 2.5);
– Monitor teachers’ marking positive or negative feedback for students in the system (Indicator 2.6);
– Monitor students’ absence from school due to unconvincing reasons (indicator 3.6) and take measures to prevent it;
– Monitor the current average result of students from all classes for all subjects and take measures to improve the results of that class whose GP A (Grade Point Average) is under the GPA of all classes in the class of the year; (Indicator 4.2); – Monitor results from a course of study of students in each speciality offered by the school (Indicator 4.3) and identify study programmes in which students do not show satisfactory results;
– Identify students with unsatisfactory results at the earliest possible stage and compare their results to the average results of students who dropped from school as well as to take measures to additionally support them to complete their course of study (Indicator 4.4);
– Monitor tendencies for assessment of students for each studied subject and compare to results of students from the previous class of the year (Indicator 4.5) and take measures in case the average mark from the current year is lower than the average from a previous year;
– Forecast and improve the degree of graduation (Indicator 4.6);
– Compare the GPA of students when graduating their course of study at the end of each school year (Indicator 4.7);
– Identify the most desired specialities (Indicator 4.8);
– Monitor students’ graduation rate and take measures if results are decreasing (Indicator 4.9);
– Monitor students’ results from competitions (Indicator 4.10);
– Monitor tendencies of students’ success in competitions during different school years (Indicator 4.11);
– Compare class rank according to GPA (Grade Point Average) of students from the class of the year to that of the previous class of the year (Indicator 4.12) and take measures in case the GPA of students in the current year is lower than the GPA of students during the previous year;
– Monitor success of students when graduating their course of study for a subject at the end of each academic year and take measures in case the GPA of students in the current year is lower than the GPA of students in the previous year (Indicator 4.13);
– Assess the quality of learning materials and course of education based on students’ participation and results achieved (Indicator 5.1);
– Assess the variety of learning activities and resources included in the course of study (Indicator 5.3);
– Monitor overloading of students with tasks (Indicator 5.4) and of teachers (Indicator 5.5).
Figure 3. Part of a generated report for Indicator 4.9
Based on the generated reports, Managers will be able to comment whether the adopted measures for students’ detention are efficient and sustainable, to use the results from analyzing the assessment and appraisal of teachers’ staff and to take measures for increasing the quality of education and improving the achieved results.
Figure 3 presents a generated report through the LASchool for Indicator 4.9. Tendencies in graduation rate by the Manager input value System programming. The report shows the statistically processed data – the average graduation rates of students training in System programming study programme for the last 5 years. Thus, e.g. if there is a significant decrease in the average student achievement during years (as in the presented report where average grade decrease from 5.25 to 4.25 and the percentage of excellent students from 15 to 10), the Manager can take timely measures to improve the quality of teaching and thus increase student success rate.
Conclusion
The paper presents the second step from the ongoing study dedicated to the implementation of LA tools in Bulgarian School education – the development of appropriate models. It presents the LA model and corresponding LA Tool designed for the needs of six stakeholder groups in Bulgarian school education (students, teachers, class teachers, parents, school managers, inspectors from evaluation agencies).
The proposed LA tool is the first LA tool that meets the needs of Bulgarian Secondary education in the full degree. LA tool will be provided for real-time testing at schools. Based on the results, users from different stakeholder groups will take measures to improve the quality of training and students’ achievements. Feedback from all users will be taken into account in the development of the final version.
Acknowledgements
The paper is supported within the National Scientific Program “Young scientists and Post-doctoral students” in accordance with Appendix No. 11 of Council of Ministers Decision No. 577 of 17 August 2018.
NOTES
1. The tools are developed by TIBCO Software, https://www.tibco.com/.
REFERENCES
Abo, R., Koga, T., Horikoshi, I., Yamazki, K., & Tamura, Y., 2016. Data Visualization Framework for Learning Analytics, The International Workshop on Learning Analytics and Educational Data Mining (LAEDM 2016).
Admiraal, W., Vermeulen, J., & Bulterman-Bos, J., 2017. Learning Analytics in Secondary Education: Assessment for Learning in 7th Grade Language Teaching, ECER 2017, https://eera-ecer.de/ecer programmes/ conference/22/contribution/39935/.
Andrade e Silva, M, & Camanho, A., 2017. Using Data Analytics to Benchmark Schools: The Case of Portugal, In B. Vanthienen, & K. Witte (Eds.), Data Analytics Applications in Education, New York: Auerbach Publications.
Andresen, B., 2017. Learning Analytics for Formative Purposes, Tomorrow’s Learning: Involving Everyone. Learning with and about Technologies and Computing, Springer International Publishing.
Avella, J., Kebritchi, M., Nunn, S, & Kanai, T., 2016. Learning Analytics Methods, Benefits, and Challenges in Higher Education: A Systematic Literature Review. Online Learning, 20(2), 13 – 29.
Brun, A., Bonnin, G., Castagnos, S., Roussanaly, A., & Boyer, A., 2019. Learning analytics made in France: the METAL project, International Journal of Information and Learning Technology, 36(4), 299 – 313.
Campbell, J., De Blois, P., & Oblinger, D., 2007. Academic analytics: A new tool for a new era, Educause Review, 42(4), 40 – 57.
Conde, M., & Hernandez-Garcia, A., 2015. Learning analytics for educational decision making. Comput Hum Behav, 47, 1 – 3.
Davis, S. K., Edwards, R. L., Miller, M., & Aragon, J., 2018. Considering Context and Comparing Methodological Approaches in Implementing Learning Analytics at the University of Victoria. Proceedings 8th International Conference on Learning Analytics & Knowledge (LAK18), 1 – 4.
Dehler, J., Bodemer, D., Buder, J., & Hesse, F. W., 2011. Guiding knowledge communication in CSCL via group knowledge awareness. Computers in Human Behavior, 27(3), 1068 – 1078.
Ebner, M., & Schön, M., 2013. Why Learning Analytics for Primary Education Matters!, Bulletin of the IEEE Technical Committee on Learning Technology, 15(2), 14 – 17.
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R., 2016. Research Evidence on the Use of Learning Analytics – Implications for Education Policy. R. Vuorikari, J. Castaño Muñoz (Eds.). Publications Office of the European Union.
Gaftandzhieva, S., Docheva, M. & Doneva, R., 2021. A comprehensive approach to learning analytics in Bulgarian school education. Educ Inf Technol 26, 145 – 163.
Guo, J., Huang, X., & Wang, B., 2017. My COS Intelligent Teaching Assistant, 392 – 393.
Hylen, J., 2015. The State of Art of Learning Analytics in Danish Schools, http://www.laceproject.eu/blog/the-state-of-art-of-learning-analytics-indanish-schools/.
Jivet, I., Scheffel, M., Specht, M., & Drachsler, H., 2018. License to evaluate: Preparing learning analytics dashboards for educational practice. Proceedings of the 8th International Conference LAK, 31 – 40. ACM.
Khine, M., 2018. Learning Analytics for Student Success: Future of Education in Digital Era, The European Conference on Education 2018.
Long, P., & Siemens, G., 2011. Penetrating the Fog: Analytics in Learning and Education, EDUCAUSE Review, 46(5), 31 – 40.
Macarini, L., Santos, H., Cechinel, C., Ochoa, X., Rodes, V., & Casas, A., 2019. Towards the implementation of a countrywide K-12 learning analytics initiative in Uruguay, Interactive Learning Environments, 28(2), 166 – 190.
Macon, B., Macon, L., & Phillips, N., 2016. Learning Analytics: Case Studies at a Large Urban Community College, 2016 International Conference on Computational Science and Computational Intelligence (CSCI), IEEE, Las Vegas: NV, 258 – 263.
McKay, E., 2019. Digital literacy skill development: Prescriptive learning analytics assessment model, Australian Council for Educational Research, Research Conference 2019, 22 – 28.
Meyers, E., Cahill, M., Subramaniam, M., Stripling, B., 2016. The Promise and Peril of Learning Analytics in P-12 Education: An Uneasy Partnership?, iConference 2016.
Mouri, K., Yin, C., & Uosaki, N., 2018. Learning Analytics for Improving Learning Materials using Digital Textbook Logs, Information Engineering Express International Institute of Applied Informatics, 4(1), 23 – 32.
NIE, National Inspectorate of Education, Inspection Criteria, https://nio. government.bg/инспектиране/критерии/.
Pardo, A., Dawson, S., Gašević, D., Steigler-Peters, S., 2016. The role of learning analytics in future education models, https://www.telstra. com.au/content/dam/tcom/business-enterprise/industries/pdf/tele0126_ whitepaper_5_spreads_lr_notrims.pdf.
Papamitsiou, Z., & Economides, A. A., 2015. Temporal learning analytics visualizations for increasing awareness during assessment. RUSC. Universities and Knowledge Society Journal, 12(3), 129 – 147.
Sancho, M., Cañabate, A., & Sabate, F., 2015. Contextualizing learning analytics for secondary schools at micro level, 2015 International Conference on Interactive Collaborative and Blended Learning (ICBL), Mexico City, 70 – 75.
Siemens et al., 2011. Open Learning Analytics: an integrated & modularized platform Proposal to design, implement and evaluate an open platform to integrate heterogeneous learning analytics techniques, https:// solaresearch.org/wp-content/uploads/2011/12/OpenLearningAnalytics. pdf.
Sergis, S., & Sampson, D.G., 2016. School Analytics: A Framework for Supporting School Complexity Leadership. In: Spector J., Ifenthaler D., Sampson D., Isaias P. (eds) Competencies in Teaching, Learning and Educational Leadership in the Digital Age. 79 – 122.
Sclater, N., 2017. Learning Analytics Explained, Routledge, 1 edition, 290 pages.
Varanasi, M. R., Fischetti, J. C., & Smith, M. W., 2018. Analytics Framework for K-12 School Systems. In E. Mense, & M. Crain-Dorough (Eds.), Data Leadership for K-12 Schools in a Time of Accountability (206 – 233). Hershey, PA: IGI Global.