https://doi.org/10.53656/ped2024-9.03

Резюме. The advent of ICT in education has allowed teachers to introduce the use of learning resources in new formats, such as video. While this has many advantages, some of the information about how students learn that is obtained by instructors when teaching face to face is lost with the new format. In this paper, we present a visualization and analysis tool to minimize this loss, collecting and then visualizing the digital footprint left by students while learning from video resources, pointing to the existing evidence found in the literature about such data. We also provide an instrument to collect feedback from teachers and other stakeholders regarding both the data and the provided evidence presented to them by the tool. We carry out a limited test to validate its working and present a proposal to expand the analysis and improve on its design. We conclude that the developed application presents a clear picture of the available data, together with interpretations from the literature about what is happening in the learning process, which can lead to improvements in the process. Additionally, the feedback collected from the tool presents opportunities to complement the available knowledge in the field, and thus lead to a better understanding of how students learn from resources in video format and how we can improve on their design and creation.

Ключови думи: learning analytics; digital resources; video learning; data visualization; teacher dashboards

Introduction

The advent of ICT in education has allowed teachers to introduce the use of learning resources in formats different to those which were used in the past. In the particular case of video, while it had already been used often in the classroom before the popularization of the internet, first, and then online streaming, these two facts have led to the current situation, in which learning resources in video format are being used extensively across multiple scenarios: not only in fully distancebased learning, but also in conventional teaching situations (e.g. using flipped classroom), and even in informal learning. This has many advantages, but it also has disadvantages: for example, the feedback instructors get by watching how students react to their teaching is lost. While it is not common to use it in analysing the teachinglearning process, students do leave a digital footprint, in the form of a clickstream, when they watch a video. There have been some studies regarding this subject, mostly linked to the use of MOOCs, involving large amounts of data and machine learning techniques, such as Banerjee et al. (2001), Sinha et al. (2014a, 2014b), Li et al. (2015), or Brinton et al. (2016).

Previous research (Córcoles et al. 2021) shows that even for smaller scale learning scenarios, when a video has been watched one or two hundred times, the technology acceptance model shows instructors find it useful to have access to that clickstream data and to be able to visualize it, even without the help of automatic analysis tools, pointing to an opportunity affecting a wider range of applications for the use of learning analytics. In that research, though, the presentation of the clickstream does not result in conclusions being extracted and interventions suggested to improve learning. Informal evidence points to the possibility that this was due to the offered visualization for the data not being clear enough, on one hand, and on the other, to the fact that, until now, there are few evidences in the literature connecting clickstream data to aspects such as perceived difficulty or others that may affect the learning process, Brinton et al. (2016), Sinha et al. (2014a, 2014b).

In the present study, we develop a novel visualization and analysis tool that intends to help solve the aforementioned problems, by presenting a clear visualization and pointing the existing evidence found in the literature, first, and then providing an instrument to collect feedback from teachers and other stakeholders regarding the connections between student behaviour and interventions to be made. We also carry out a limited test to validate its working, collect early feedback and present a proposal to expand the analysis and improve on the design of the tool. We also conclude that the developed application allows, on one hand, to present a clear picture of the available data about how students learn from learning resources in video format, together with interpretations from the literature about what is happening in the learning process, which can lead to improvements in the process, especially regarding the learning resource itself. On the other hand, the feedback collected from the tool presents opportunities to complement the available knowledge in the field, and thus lead to a better understanding of how students learn from resources in video format and how we can improve on their design and creation.

This paper is organized as follows. First, we present a literature review of the work that has been done in the field until now. Then follows a section where we describe the followed methodology. Next, we have two sections devoted to the two iterations in the design and development and analysis of our tool. Finally, we have the last two sections, devoted to the presentation of the results and the associated discussion.

Literature review

Several authors highlight the value of data visualization in the educational context. On one hand, the importance of exploring learning through tracking generated data is emphasized (Siemens 2013). On the other hand, the utility of visualizing learning processes and outcomes to enhance analysis and understanding is pointed out (Gómez-Aguilar et al. 2010). Applications like learning analytics dashboards are perceived as useful means to assess the effectiveness of a course in the preparation, execution, and evaluation phases of learning (Hooshyar et al. 2020). It is presented as essential for students to obtain information about their actions, for teachers to be aware of the interactions they generate, and for researchers to identify patterns and communicate them to users (Klerkx et al. 2017). Henrie et al. (2015) review the existing approaches to measure engagement in technology-mediated learning, identify their strengths and limitations, and outline potential approaches to improve the measurement of student engagement. Another such review, focused on visual tools, such as the one we present in the present work, is found in Vieira et al. (2018). Jovanović et al. (2017) show the use of learning analytics in a flipped classroom scenario to try to understand the types of learning strategies that students employ in that situation. Hernández-Leo et al. (2018) present a framework for the use of learning analytics to inform learning design.

As for learning analytics applied specifically to measure aspects related to learning resources in video format, Seidel et al. (2017) present a literature review on analytics on video-based learning. Dodson et al. (2018) present a framework to describe students' behaviour while learning from video. The first effort in the literature focusing on clickstream analytics is found in Banerjee et al. (2001), applying clustering techniques. Sinha et al. (2014a, 2014b) construct a “quantitative information processing index” based on video clickstream data, which can aid instructors to better understand MOOC hurdles and reason about unsatisfactory learning outcomes, and try to predict student attrition, respectively. Li et al. (2015) carry out similar research and find a relationship between video watching behaviour and perception of difficulty. Brinton et al. (2016) follow on that path to try to find a relationship between video watching behaviour and quiz performance. A similar effort is carried out by Hasan et al. (2020), as do Yoon et al. (2021). All of these efforts rely on clickstream data collected from videos that have been viewed a large number times, often in MOOC style environments.

Regarding the visual presentation of clickstream data to visualize student behaviour when learning from learning resources in video format, Kim et al. (2014) carry out a large-scale analysis of in-video dropout and peaks in viewership and student activity using data from numerous videos used in MOOC environments and show a relationship between longer videos and bigger dropout rates, and show that different types of videos and learning situations result in different viewing behaviour. Kleftodimos et al. (2016) and Giannakos et al. (2018) design and present an analytics system for use in learning scenarios where students are offered learning resources in video format. The first ones introduce a framework of “active viewing”, inserted in the concept of active learning, while the second ones, in their analysis, show evidence of a relationship between video navigation and the level of cognition/thinking required for a specific video segment. Wang et al. (2016) present a different visualization for clickstream data, that makes use of animation to illustrate video viewing behaviour. Min et al. (2019) present a similar system to gather and visualize viewing data. The visualization they present is similar to the one our previous research used (Córcoles et al. 2021).

To our knowledge, the example of work in the literature that is closest to the works we present in this paper is the one seen in Mubarak et al. (2021), presenting a learning analytics and analysis solution sharing some principles with ours. Theirs, though, is oriented to videos being viewed many more times than our study, and applies machine learning techniques to try to extract conclusions that, while compelling and powerful, cannot be applied to the small-scale learning scenarios this work focuses on.

It must be noted that even before the advent of comprehensive analytics solutions, studies have been carried out to analyse whether the use of digital videos and the possibilities they afford, such as interactivity. For an example, see Merkt et al. (2011). Another example, now with mature learning analytics infrastructure available, of research made possible by digital video resources is the analysis of students’ behaviour and perceptions regarding complementary videos in online environments found in Pérez-Navarro et al. (2021).

Hu et al. (2020) propose analysis methods of students’ video watching behaviour in MOOCs platform and put them to test with real data. Their data collection schema is similar to ours. They use big data techniques to process and distil the data into human processable information.

Córcoles et al. (2021) show evidence that even for small-scale learning scenarios, instructors find it useful to have access to a presentation of the clickstream data for the different videos they have used in their classrooms in a distance learning context. The proposed visual presentation, though, didn’t allow for an easy interpretation of said navigation patterns. In particular, instructors were keen to see if there were instances in which a high enough number of learners skipped from a certain point in the video to another point, and the distribution and length of pauses. This work also documents the data collection schema that has been used in the present work. Similar schemas can be found in Kim et al. (2014), Min et al. (2019) or Hu et al. (2020).

Methods

This research follows a mixed research methodology (Figure 1) that combines an action research methodology with a design and creation approach (Oates 2005).

First, the problem is detected and shared (in our case, an absence of information about how students learn from learning resources in video format). Secondly, a solution in the form of an artifact (in our case, this artifact is a software application that visually presents the clickstream data left by students when watching said learning resources, together with the available interpretations that the literature offers) is suggested. Thirdly, the artifact is gradually implemented and tested in real scenarios following an iterative cycle of plan-act-reflect. After each cycle, an evaluation is done according to performance measures. Depending on the results, changes in the artifact are introduced, causing a new cycle until obtaining the final artifact. The research we present constitutes two cycles of that process, conducted on the learning resources in video format used on a course on electronic circuits from a four-year degree in telecommunications engineering, and it can be considered an iteration on the methodology from our previous research (Córcoles et al. 2021).

We are then faced with how to present said data in an easy to digest format for instructors, who we cannot assume are experts in the analysis of the presented data, ideally with minimal onboarding.

Participants

In essence, we collect all events resulting from events launched by user interaction (such as students clicking play or pause, scrubbing the video timeline to skip backwards or forwards, or clicking on an index item to move to a particular point in the video’s timeline). The collected and analysed data corresponds to four different videos associated to a course in Electronic Circuits, offered yearly in a degree in Telecommunications Engineering, describing the workings of two devices, a tester and a protoboard, and the measurements that can be taken using both devices. The course is preceded by another one, Circuit Theory, which also contains the videos as learning resources. As a consequence, students already proficient in the use of the tester and the protoboard are not expected to watch the videos in this second course. The data corresponds to editions of the course between 2014 and 2020. The total enrolment for those courses is of 452 students, or an average of 65 students per course. After discarding playback sequences where the collected data presents doubtful data, the total number of analysed playbacks for the four videos is 135, 118, 130 and 81. The videos range in length from four minutes and three seconds to seven minutes and three seconds. In the indicated period, three instructors were in charge of the course. Two were available for interviews, and two semistructured interviews were carried out with each one.

Procedures

As corresponds to the methodology we propose to follow, we proceed in an iterative process. First, we carry out a first design and development stage of the application. Then, we present the application to some of the stakeholders and carry out a semistructured interview to obtain their validation and feedback, which is integrated in a second stage of design and development. Then, the tool is presented more extensively to stakeholders to carry out a second validation, again taking the form of a semistructured interview. Finally, the results obtained from that phase are presented and discussed, and we close with some future lines of work derived from there, and the conclusion.

First stage of design and development

We take a similar visual approach to the one seen in Mubarak et al. (2021), although their focus is on quantitative data on performance, which is of course essential, while ours centres on instructors’ impressions on student understanding based on navigation patterns, and whichever conclusions can be obtained from those patterns according to the existing literature. Regarding their visualization, we combine aspects from their “learner performance view” and their “video statistics view” and allow for segmentation and annotation. Our purpose is different from theirs in that we intend to present instructors and other stakeholders with a visualization that makes it easy to notice and observe emerging patterns, annotate them and allow for intervention if the pattern occurs again in the future (with one possible intervention being the interrogation of future students presenting the same pattern). It is our assumption that in small-scale learning scenarios the available data are not enough to make use of most quantitative techniques present in the literature, as the available amount of data is not enough to achieve statistical significance, or to apply artificial intelligence techniques such as machine learning, while more individualized, case-by-case attention is possible.

As we focus on small-scale learning scenarios, where a few hundred visualizations are the norm, rather than thousands or more, we opt for a visualization that can combine both backwards and forwards skipping with pausing behaviour, providing an option so that users can choose to display only one or the other, as the visual clutter resulting from presenting larger amounts of data does not take place, and the combination allows for a more efficient reading of the data.

Evidence and data from other research, such as Li et al. (2015), Brinton et al. (2016), Hasan et al. (2020) or Yoon et al. (2021), that have shown an association between different types of viewing behaviour and aspects such as perception of difficulty or differing results in academic performance, should also be presented to instructors so that the accumulated scientific evidence can be compared to their impressions and completed from new evidences obtained from experience. In our previous research, it was common that videos, especially those lasting more than ten minutes, presented several sections which had sensibly different purposes (such as recalling previous knowledge, working out examples, summarizing the previously presented knowledge…). It is logical to presume that different sections with different purposes should present different video viewing behaviours. At our stage, it is not viable to tell sections apart automatically —although Kim et al. (2014) seem to indicate that most sectioning would be directly related to relatively abrupt changes in the video frame. Thus, we offer in our tool an interface that allows users to mark as many spots in the video as needed. If a user defines n points, t1,..., tn, that results in n+1 segments ([0, t1), [t1, t2), … [tn, tf]). For each of these segments, we will present the corresponding summarized data (mainly, the number of pauses and their average length, and the number of backwards and forwards skipping actions, both inside the segment or moving outside of it, normalized by the number of video plays and segment length), and, where possible, the interpretations of said data available in the literature. At the current moment, this is generally limited to a correlation between high perceived difficulty and a high number of pauses and backwards skips, and a correlation of low perceived difficulty and high numbers of forward skips, as will be seen later in greater detail. Our previous research showed, though, that teachers will usually notice other relationships between those observed patterns and what is happening in the teaching and learning process which so not necessarily agree with what is currently present in the literature, or other distributions of behaviours that could also have an educational interpretation. These observed patterns and interpretations should be shared and observed, as this could lead to new evidence that could be of use in other situations.

The tool was developed using web technologies (HTML, CSS, and JavaScript) and employing the p5.js JavaScript library (P5JS)1 for visuals and interactivity. The collected data for each video is preprocessed (using R and Python) and a file in .csv format is generated to be then ingested by the web application. Each event takes a line in the file, with the form jump, t i, tf for jumps from a point t i to another point tf; pause, t, l for pauses of length l at a point t; or stop, t if a student stops the video at point t and does not resume playback. The data is then processed and displayed for the user, with the visual interface seen in Figure 2.

The central part in the image corresponds to the main timeline display. Above it, to the left, an information box is shown, displaying information related to the highlighted segment in the video. To the right of the information box, we show the video at the last selected point in the main timeline. Below, a minimap is provided to assist with navigation. As in Mubarak (2021), we use an arc visualization for jumps, with forward jumps being displayed above the timeline and backwards jumps below it. Stops and pauses are displayed as vertical lines below the timeline, of length proportional to the pause duration. As this could easily result in visual clutter, users can turn on and off the display of jumps and pausing and stopping behaviour.

To account for the possibility of videos with large amounts of overlapping jumps or pauses, these are drawn using transparency, so that overlap is visually translated as an increase in opacity.

As we are providing an exploratory interface, and we are assuming that different segments may show different student behaviours, we must include the ability to zoom in and out of the video. This leads to the need for a minimap for the video (displayed at the bottom in figure 1), which is then provided with a click and drag interface to move the main timeline. The user is also given the ability to add or remove hotspots at any point in the video timeline. As said before, these hotspots define segments in the video. When the user clicks on a segment, information is displayed about the segment itself and about student behaviour in that segment.

Initial validation

A first round of semistructured interviews with the two teachers was carried out when the tool was at an early stage of development, corresponding to the interface shown in Figure 1, so that early feedback could be incorporated into the project. Users were shown that early working prototype for the visualization tool showing the data for a particular video. One recurring comment was that it would be convenient to have a dashboard allowing easy navigation to the different videos for a course and presenting some basic information. As a result, a basic dashboard was created, showing an enumeration of the videos, showing the title, length, number of playbacks, and the of interactions per second and a thousand playbacks (so that numbers would be comparable for all videos presented), linked to the visualization for the video. This design is shown in Figure 3. Also, an additional, more visual, design was proposed (see Figure 4), adding an information bar showing the proportion of backwards skips, forwards skips and pauses, as high ratios of backwards skips are associated with demands on information processing, pauses are related to reflection by students and forwards skips to lower levels of processing. This design will be tested at a future date on a future iteration of our methodology.

Basic information that can be extracted from the data according to the literature As aforementioned, it is one of our objectives to present teachers with the evidence in the literature regarding the navigation patterns in the recorded clickstream. We now enumerate the connections between student behaviour while watching videobased learning resources and student perceptions about the content that can be found in the literature, match the collected data structure and are actionable.

Sinha et al. (2014a) define a number of behaviour patterns: “rewatching”, characterised by scrolling back; “skipping”, characterised by “scrolling forwards”; “slow watching” and “fast watching”, both characterised by playback rate changes; “clear concept”, characterised by a combination of seeking and scrolling backwards, and indicating high tussle with the video lecture content”; “checkback reference”, characterised by a wave of seeking backwards actions; and “Playrate transition”, characterised by a wave of playback rate changes.

They refer to four different student perceptions of the video lecture segments, characterised by information processing needs:

1. “Difficult to understand”, that would match to rewatching, clear concept, and slow watching.

2. “Instruction pace too high or too low”, matching to playrate transitions (for which we cannot check currently, and will not take into account here).

3. “Explicit recalling of facts required”, matching to “checkback reference”.

4. “Easy/Boring/uninteresting or simple to understand”, matching to skipping and fast watching.

Brinton et al. (2016) define four behaviours derived from clickstream data:

– “reflecting”, related to repeated playing and pausing;

–“reviewing”, related to playing for a length of video and then reviewing;

–“skimming”, related to skipping forwards (that may be combined with some “skipping backwards” for “skipping forward with caution”);

–“speeding”, related to speed changes (that we cannot check for with precision).

Additional discussions can be found in Sinha et al. (2014b), Giannakos (2015), Mubarak (2020) and Yoon (2021), but they do not present clear connections between student behaviour and either student perception or interventions to be made to the learning resources, so we are focusing on the information presented above.

In a first, naive approach, based on the presented information, we will pay attention, firstly, to high pausing rates, which will be presented to the user as segments where students are reflecting, as per Brinton et al. (2016). Then we will pay attention to the segments with a predominance of backwards skipping, and will grade that according to a five-degree scale: we assume that the distribution of the different kinds of events follows a uniform distribution that should result in a distribution roughly approaching the normal distribution (more data is needed to validate this assumption, and the application can be modified to adjust to other distributions) for the number of events present on a segment for the video. Thus, we assume that events in the top quartile will be associated with the highest demands on information processing, while the lowest will be associated with a low demand on information processing and the rest will be associated with average demand, according to the classification by Sinha et al. (2014a).

For every action, we calculate the expected number of actions on the segment (considering just the length of the segment, on this first approach). For pausing, we will highlight it as high if the real number is 5% above the expected value. For backwards skipping, we will associate the top level to 15% or more above the expected value, the second level to 5% to 15%, the third to 5% to 10% above and the fourth to average to 5% above. For forwards skipping, we will highlight segments where the real value is at least 5% above the expected value. To visualize high and low levels, for both skipping actions and pauses, a colour scale is used. The resulting visualization is shown in Figures 5 and 6 below.

Second stage of design and development

Finally, we integrate the feedback collected in the first iteration of our process, and we provide an interface so that teachers, and other stakeholders, can make their annotations for their interpretation of what is happening at any given segment. That interface is shown in figure 2. All the segmenting and the user’s annotations are continually saved so that they can be retrieved at a later time.

As for the design and development of the annotation tool, we chose the simplest possible interface. We add a text field, whose contents will be associated with the highlighted segments in the video. The global interface is shown in Figure 5. In this second iteration, the different areas of the tool have been more clearly differentiated, as a result of evolutions in the design process and feedback obtained from users. The video, minimap for navigation and detailed view of navigation patterns are on the right hand of the image. Colour coded icons are provided so that the relative frequency of skips and pauses can be observed easily. On the left-hand side, metadata for the currently highlighted segment of the video is provided. If available, recommendations from the literature are also shown. Both the metadata and the provided feedback can be seen in Figure 6. Finally, an interface is presented so that the user can provide and store their annotations regarding what is happening in each segment

On this second iteration, all annotations by teachers are shared with the researchers, for testing and validation purposes, on one hand, and to analyse annotations to extract useful information to be added to the analysis tool.

Second validation

When the feedback from the first round of interviews had been incorporated into the tool and the development of the first prototype had been finalised, a second round of interviews was carried out. As in the first round, interviews started with an onboarding process in which the tool and its functionalities were described with a practical example. The feedback from that phase fits what was shown in previous research (Córcoles et al. 2021): interviewees instantly show interest in the presented data and proceed to segment the different videos according to their perception of their structure, and to compare their impressions on the difficulty and complexity of each segment with the limited interpretations the tool currently presents to its users. Also, the visualization was generally perceived as easy to understand after the proposed onboarding process in which the tool and its functionalities is presented, according to comments by users.

To validate the functionalities and the usability of the tool, a task was presented to interviewees, asking them to go through all the videos in the course, to systematically segment them according to their perceptions about the structure of the presented content, and to compare their impressions to the interpretations (or lack of interpretation) offered. In most cases, the offered interpretations match the instructors’ perceptions but, on a few occasions, there are differences. As an example, instructors offer that, in short segments presenting procedural information (in the present case, demonstrations of manipulation of electronic hardware), a high number of backwards jumps – that the literature interprets as associated to a high cognitive load – could simply mean that students are watching once to get a general impression and a second time to reproduce themselves the demonstrated procedure. This offers an example of how the feedback collected by the tool can be used to give a wider interpretation of the clickstream data than in currently available in the literature. Also, in some cases, interpretations are offered that have to do with the sequence of the learning resources: similar segments present different behaviours in different videos, and instructors offer that the first time information is presented show behaviours linked with higher cognitive load, and lower in subsequent videos. This leads us to believe there is a need to explore in better ways the sequences of learning resources for better analysis.

Also, a comment is made about the usefulness of some kind of tool to filter the clickstream data by date (the limited amount of collected data for the videos used for these tests led us to present all the available data without filters). Finally, regarding the distribution of events, it is commented that a higher distribution of forwards jumps at the beginning of videos, and of backwards jumps at the end (and conversely, low levels of forwards jumps at the end, and low levels of forwards jumps at the end) is to be expected and that the tool should somehow take that into account when presenting information and its interpretation.

Results

The tool is working correctly and produces useful and significant results, although more iterations and testing in a wider variety of situations are needed.

The tool is giving useful information to instructors about the differences in cognitive load and the need for reflection by students among the different segments of a video. It also gives quantitative information about the usage of the learning resources in video format assigned to a course. It is also collecting information from instructors regarding video segments in which the presented information does not match what instructors gather from the combination of the presented clickstream data with their knowledge of the domain and of aspects such as the learning situation, or the sequence of learning resources in the course. As an example, instructors mentioned that in segments in some videos, the relatively high presence of pauses, which the application interprets as needing a high level of reflection, had more to do with a presentation of a sequence of steps demonstrated in the video that students should then reproduce, and thus it wouldn’t be necessary linked to reflection. This information can be used to complement the information currently available in the literature. It was also observed that in some of the videos, especially those presenting less recorded interactions per minute, little or no feedback is presented. One first possible interpretation of this fact is that a minimum number of recorded interactions is needed to give usual feedback. On the other hand, this could be pointing to aspects that the literature has not covered yet, where a more profound analysis could lead to filling gaps in our current knowledge. The collected behavioural data can also result in the detection of problematic segments in a learning resource, such as those presenting an unnecessarily high level of cognitive load, which should leave to the improvement of the affected learning resources, or the creation of new ones especially designed to address the needs of some of the students. In some cases, the collected information can also help detect video resources that are not used, because they are redundant or unnecessary, which can lead to improvements in the efficiency of the production of learning resources.

Regarding the presented visualization of the data, it presently requires of an onboarding process so that instructors and other stakeholders can use it proficiently, but after the process the interviewed users affirm the visualization is easy to understand and useful, as is the textual information presented, and the provided annotation tool is also considered useful and easy to use. Despite this, it is necessary to provide better documentation so that the tool can be used independently without the need of a personalized onboarding process.

Discussion and future work

In this research process, we have created and tested successfully a learning analytics tool that is useful beyond large-scale learning scenarios where hundred of students watch a series of learning resources in video format and leave a large footprint in the form of clickstream data, such as those seen in Sinha et al. (2014a, 2014b), Li et al. (2015), Brinton et al. (2016), Hasan et al. (2020), Mubarak et al. (2021) or Yoon et al. (2021), but more oriented to smaller volumes of clickstream data. In order to do so, knowledge accrued in those previous works can be leveraged and integrated into our tools, and it has been done in the case of Brinton et al. (2016).

The collected information is presented in a visual way, as is done in Kim et al. (2014), Kleftodimos et al. (2016), Giannakos et al. (2018), Wang et al. (2016), Min et al. (2019), and Mubarak et al. (2021). Our visualization also builds on the experience acquired in our previous research (Córcoles et al., 2021). In some aspects, our approach is similar to the arc visualization seen in Mubarak et al. (2021). In their case, the arc visualization was only a part of the different aspects shown in the visualization, and it was not easy to interpret due to the large volume of data. Also, no tools for annotation, or to collect feedback from stakeholders were present in the different applications seen in the literature. We believe this collection of feedback is essential to build on the available knowledge, and that an infrastructure should be built to collect and analyse that feedback.

There are multiple lines that can be explored in the future. The main one is being able to provide better instruments that allow an easy deployment of the tool, both to collect the required clickstream data, and to visualize it and offer feedback to instructors and other stakeholders. This, for example, could come as the development of plug-ins and other solutions so that the application can be integrated in popular content management systems and learning management systems. Such a tool should also offer better ways to collect and summarize the collected information so that the resulting knowledge can be leveraged and result in improvements to the domain of knowledge and in the production of future learning resources.

Moreover, there is a need to widen the learning situations to more types of videos, specifically paying attention to longer ones, and courses, widening the range of subjects and the taxonomy of types of presentation used in them. This is a necessary step to improve on the improvement and validation of the provided feedback, and to analyse how some feedback is only valid for particular subjects, types of presentation or other factors. Furthermore, a better quantification of events could help provide better information. E.g., forwards jumps are to be expected at the beginning of a video, and very rare towards the end, and this could be accounted for. Also, experiments should be carried out to fine tune what are now considered as sufficient levels of interactions to give certain kinds of feedback. It is therefore our intention to keep iterating on the design, development and analysis of our tool, keeping in spirit with our chosen methodology combining action research with design and creation. In these future iterations, we will keep the stakeholders we have already worked with in the process, but also to include new courses with different videos and a new batch of instructors to interview.

NOTES

1. P5.JS, https://p5js.org/. [Viewed 2024-6-6].

2. BANERJEE, A.; JAIN, S. P.; & GHOSH, J., 2001. Clickstream Clustering using Weighted Longest Common Subsequences [online]. Available from: https://www. researchgate.net/publication/2365525. [Viewed 2024-6-6].

REFERENCES

BRINTON, C. G.; BUCCAPATNAM, S.; CHIANG, M.; VINCENT POOR, H., 2016. Mining MOOC Clickstreams: Video-Watching Behavior vs. In-Video Quiz Performance. IEEE Transactions on Signal Processing, vol. 64, no. 14, pp. 3677 – 3692. Available from: https://doi.org/10.1109/TSP.2016.2546228. [Viewed 2024-6-6]. ISSN 1053-587X.

CÓRCOLES, C.; COBO, G.; GUERRERO-ROLDÁN, A.E., 2021. The usefulness of video learning analytics in small scale e-learning scenarios. Applied Sciences vol. 11, no. 21, pp. 1 – 16. Available from: https://doi. org/10.3390/app112110366 [Viewed 2024-6-6]. ISSN 2076-3417.

DODSON, S.; ROLL, I.; FONG, M.; YOON, D.; HARANDI, N. M.; FELS, S., 2018. An active viewing framework for video-based learning. Proceedings of the 5th Annual ACM Conference on Learning at Scale, L at S 2018, pp. 1 – 4. New York: Association for Computing Machinery. ISBN 978-14-50358-86-6.

GIANNAKOS, M. N.; CHORIANOPOULOS, K.; CHRISOCHOIDES, N., 2015. Making sense of video analytics: Lessons learned from clickstream interactions, attitudes, and learning outcome in a video-assisted course. International Review of Research in Open and Distance Learning, pp. 260 – 283. Available from: https://doi.org/10.19173/irrodl.v16i1.1976. [Viewed 2024-6-6]. ISSN: 1492-3831.

GOMEZ AGUILAR, D.A; SUAREZ GUERRERO, C.; THERON SANCHEZ, R.; GARCIA PENALVO, F., 2010. Visual analytics to support e-learning. Advances in Learning Processes, pp. 207 – 228. Rijeka: IntechOpen. Available from: http://doi.org/10.5772/7932 [Viewed 2024-6-6]. ISBN: 978-953-761956-5.

HASAN, R.; PALANIAPPAN, S.; MAHMOOD, S.; ABBAS, A.; SARKER, K. U.; & SATTAR, M. U., 2020. Predicting student performance in higher educational institutions using video learning analytics and data mining techniques. Applied Sciences (Switzerland), vol. 10, no. 11, pp. 1 – 20. Available from: https://doi.org/10.3390/APP10113894 [Viewed 2024-6-6]. ISSN: 2076-3417.

HENRIE, C. R.; HALVERSON, L. R.; & GRAHAM, C. R., 2015. Measuring student engagement in technology-mediated learning: A review. Computers and Education, no. 90, pp. 36 – 53. Available from: https://doi.org/10.1016/j.compedu.2015.09.005 [Viewed 2024-6-6]. ISSN: 0360-1315.

HERNÁNDEZ-LEO, D.; MARTINEZ-MALDONADO, R.; PARDO, A.; MUƒOZ-CRISTÓBAL, J. A.; RODRÍGUEZ-TRIANA, M. J., 2018. Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, vol. 50, no. 1, pp. 139 – 152. Available from: https://doi.org/10.1111/bjet.12645 [Viewed 2024-6-6]. ISSN: 0007-1013.

HOOSHYAR, D.; PEDASTE, M.; SAKS, K.; LEIJEN, .; BARDONE, E.; WANG, M., 2020. Open learner models in supporting self-regulated learning in higher education: A systematic literature review. Computers & education, no. 154, pp. 1 – 19. Available from: https://doi.org/10.1016/j. compedu.2020.103878 [Viewed 2024-6-6]. ISSN: 0360-1315.

HUI, H.; ZHANG, G.; GAO, W., 2020. Big data analytics for MOOC video watching behavior based on Spark. Neural Computing and Applications, no. 32, pp. 6481 – 6489. Available from: https://doi.org/10.1007/s00521-01803983-z [Viewed 2024-6-6]. ISSN: 0941-0643.

JOVANOVIĆ, J.; GAŠEVIĆ, D.; DAWSON, S.; PARDO, A.; MIRRIAHI, N., 2017. Learning analytics to unveil learning strategies in a flipped classroom. Internet and Higher Education, no. 33, pp. 74 – 85. Available from: https:// doi.org/10.1016/j.iheduc.2017.02.001 [Viewed 2024-6-6]. ISSN: 1096-7516.

KIM, J.; GUO, P.; SEATON, D.; MITROS, P. 2014. Understanding in-video dropouts and interaction peaks in online lecture videos. New York: Association for Computing Machinery. ISBN: 978-1-4503-2669-8.

KLEFTODIMOS, A.; & EVANGELIDIS, G., 2016. Using open source technologies and open internet resources for building an interactive video based learning environment that supports learning analytics. Smart Learning Environments, vol. 3, no. 9, pp. 1 – 23. Available from: https:// doi.org/10.1186/s40561-016-0032-4 [Viewed 2024-6-6]. ISSN: 21967091.

KLERKX, J.; VERBERT, K.; DUVAL, E., 2017. Learning analytics dashboards. Alberta: Society for Learning Analytics Research (SoLAR).

LI, N.; KIDZINSKI, L.; JERMANN, P.; DILLENBOURG, P., 2015. MOOC video interaction patterns: What do they tell us? Heidelberg: Springer International Publishing. ISBN: 978-3-319-24257-6.

MAZZA, R.; DIMITROVA, V., 2007. CourseVis: A graphical student monitoring tool for supporting instructors in web-based distance courses. International Journal of Human-Computer Studies, nol. 65, no. 2, pp. 125 – 139. Available from: https://doi.org/10.1016/j.ijhcs.2006.08.008 [Viewed 2024-6-6]. ISBN: 1071-5819.

MIN, Q.; CHEN, Y.; XIA, D., 2019. A Video Learning Analytics System for Students’ Learning Behavior Collection and Visualization. J. Comput., vol. 14, no. 6, pp. 381 – 388. Available from: https://doi.org/10.17706/ JCP.14.6.381-388 [Viewed 2024-6-6].

ISSN: 1796-203X. MERKT, M.; WEIGANF, S.; HEIER, A.; SCHWAN, S., 2011. Learning with videos vs. learning with print: The role of interactive features. Learning and Instruction, vol. 21, no. 6, pp. 687 – 704. Available from: https://doi.org/10.1016/j.learninstruc.2011.03.004 [Viewed 2024-6-6]. ISSN: 0959-4752.

MUBARAK, A. A.; CAO, H.; ZHANG, W.; ZHANG, W., 2021. Visual analytics of video‐clickstream data and prediction of learners’ performance using deep learning models in MOOCs’ courses. Computer Applications in Engineering Education, vol. 29, no. 4, pp. 710 – 732. Available from: https://doi.org/10.1002/cae.22328 [Viewed 2024-6-6]. ISSN:1061-3773.

PEREZ-NAVARRO, A.; GARCIA, V.; & CONESA, J., 2021. Students’ Behavior and Perceptions Regarding Complementary Videos for Introductory Physics Courses in an Online Environment. Applied Sciences, vol. 11, no. 2, p. 523, pp. 1 – 22. Available from: https://doi.org/10.3390/ app11020523 [Viewed 2024-6-6]. ISSN: 2076-3417.

SEIDEL, N., 2017. Analytics on video-based learning. A literature review. DeLFI/GMW workshops [online].

SIEMENS, G., 2013. Learning analytics: The emergence of a discipline. American Behavioral Scientist, vol. 57, no. 10, pp. 1380 – 1400. Available from: https://doi.org/10.1177/0002764213498851[Viewed 2024-6-6]. ISSN: 0002-7642.

SINHA, T.; JERMANN, P.; LI, N.; DILLENBOURG, P., 2014. Your click decides your fate: Inferring information processing and attrition behavior from MOOC video clickstream interactions. Available from: https://doi. org/10.48550/arXiv.1407.7131. [Viewed 2024-6-6].

SINHA, T.; LI, N.; JERMANN, P.; DILLENBOURG, P., 2014. Capturing “attrition intensifyingˮ structural traits from didactic interaction sequences of MOOC learners. Available from: https://doi.org/10.48550/arXiv.1409.5887. [Viewed 2024-6-6].

OATES, B., 2005. Researching information systems and computing. London: Sage Publications ISBN: 978-1-4129-0224-3

VIEIRA, C.; PARSONS, P.; & BYRD, V., 2018. Visual learning analytics of educational data: A systematic literature review and research agenda. Computers and Education, vol. 122, pp. 119 – 135. Available from: https://doi.org/10.1016/j.compedu.2018.03.018 [Viewed 2024-6-6]. ISSN: 0360-1315.

WANG, Y.; CHEN, Z.; LI, Q.; MA, X.; LUO, Q.; QU, H., 2016. Animated narrative visualization for video clickstream data. New York: Association for Computing Machinery. ISBN: 978-1-4503-4547-7.

YOON, M.; LEE, J.; & JO, I.-H., 2021. Video learning analytics: Investigating behavioral patterns and learner clusters in video-based online learning. The Internet and Higher Education, vol. 50, no. 100806, pp. 1 – 10. Available from: https://doi.org/10.1016/j.iheduc.2021.100806 [Viewed 2024-6-6]. ISSN: 1096-7516.