https://doi.org/10.53656/ped2024-5.01

Резюме. This research, conducted in June 2023 at Sofia University “St. Kliment Ohridski”, aimed at gaining in-depth insights of the extent to which Bulgarian Internet users in tertiary education had developed comprehension of generative artificial intelligence (AI) models and algorithm literacy. For the purposes of this study, a scale measuring the knowledge of generative AI models was devised and implemented, and an existing algorithm literacy scale was tested. Altogether, 125 university students across various majors in the field of education took part in the research. Findings revealed that the newly developed scale on generative AI models displayed good reliability and correlated positively with the measure for algorithm knowledge and students’ self-reported language skills. Group differences in relation to students’ university major were found to be significant for knowledge and use of generative AI models, language skills, and coding skills. As hypothesized, students in media education displayed high scores on most scales.

Ключови думи: generative artificial intelligence; algorithm literacy; locus of control; English language skills; coding skills

Given the speedy evolvement of generative artificial intelligence (AI), as well as the ever-broadening range of its applications, the implications of new technologies gain still greater prominence. AI has an impact on education, science, media, and other spheres of both societal and individual daily lives and concerns. Therefore, the development of algorithm literacy (AL) as well as the comprehension of AI technologies will play a key role in successfully managing the challenges, as well as maximizing the opportunities brought forth by rapid technological progress.

The present research aims at gaining in-depth insights of the extent to which undergraduate and postgraduate students in the field of education who use Internet resources, digital platforms, networks and services have developed comprehension of AI technologies and algorithm literacy. In particular, the survey sought to establish whether there would be significant differences between undergraduate students in media education and art communication, given their more intensive and creative engagement with digital media, and the students from other academic majors in education. For the specific objectives of this study a novel scale has been elaborated, seeking to establish the level of young Internet users’ comprehension and engagement with generative AI models. Respondents’ algorithm literacy was assessed with the two-dimensional AL Scale for Internet Users (Dogruel, Masur & Joeckel 2021). The conducted research additionally investigates respondents’ selfassessment of their foreign language and coding skills, and measures their locus of control, using Locus of Control Scale (Rotter 1966).

1. Context of the research

To briefly outline the national context within which the present survey has been conducted, we can present statistical data on digital skills and literacy in Bulgaria. The Annex for Bulgaria to ‘2023 Report on the state of the Digital Decade’1, published by the European Commission (EC), indicates that only around one third of the country’s population aged between 16 and 74 has at least basic digital skills and only 8% of Bulgaria’s population has above basic digital skills.

This finding is further corroborated by data, provided by the Bulgarian National Statistical Institute in 2021 for ICT usage in households and in particular e-commerce activities of individuals aged 16 – 742. Among the reasons why respondents choose not to buy goods or services over the Internet, insufficient digital skills rank much lower (7.6%) than the preference for shopping in person (31.4 %). Additionally, Bulgarian users are almost equally concerned about payment security and data privacy (3.5 %), on the one hand, and reliability or speed of product or service delivery (3.4 %), on the other.

In line with the EC recommendation for the development, implementation, and evaluation of digital education policy1 as well as the increasing applicability of digital tools in the sphere of education (Peytcheva-Forsyth, Aleksieva & Yovkova 2018), we believe that research on and promotion of algorithm literacy in Bulgaria can help alleviate digital divide and ensure autonomous, confident and satisfying online navigation for Bulgarian users.

2. Key aspects of research topic

2.1. Definitions of the constructs AI and AL

The robust recent research in AI technologies and their underlying algorithmic basis highlights their various aspects and aims at rendering precision to the existing terminology.

Thus, one of the definitions of the construct AI focuses on developing ways “for computers to simulate human-like intelligent behavior, able to interpret and absorb new information for improved problem-solving, and recognize patterns” (Head, Fister & MacMillan 2020, p. 49).

Another valid aspect of AI technologies is the analysis of users’ personal data they perform “in order to customize online experience” (Kozyreva et al. 2021, p. 3). Preserving user privacy is a major concern within human interactions with online systems and related devices as well (Ogunniye & Kökciyan 2023).

Interactions between human intelligence and AI are at the centre of research on explainable AI, defined by T. Miller as referring to “an explanatory agent revealing underlying causes to its or another agent’s decision making” (Miller 2019, p. 2). This is further corroborated by J. Binder when considering generative AI models like GPT-3 as “granting mathematics and language coequal roles” (Binder 2022, p. 211).

In the present study, algorithms are broadly viewed as socio-technological systems where user-to-algorithm interactions involve mutual impact and shaping. Further to that, we chose to follow Banh and Strobel (2023) definition of AI as “an umbrella term spanning over different computational algorithms capable of performing tasks that typically require human intelligence” (p. 2). As regards the definition of generative AI models, we adhered to the understanding of such models as comprising of “image generative AI models such as […] Dall-E 2 or large language models such as GPT-4” (Herrera 2023, p. 7).

In its turn, AL has increasingly been viewed as a multidimensional construct (Oeldorf-Hirsch & Neubaum 2023, Dogruel, Masur & Joeckel 2021, Swart 2021) which features cognitive, behavioural and affective aspects.

Currently, most studies regard the nature of algorithms as non-transparent and opaque (Oeldorf-Hirsch & Neubaum 2023, Binder 2022, Kozyreva et al. 2021). Specific focus is placed on concerns, ensuing from users’ distrust of algorithms (Shin, Rasul & Fotiadis 2022, Miller 2019). Further to that, ethical concerns about algorithm use are also considered relevant to AL (Head, Fister & MacMillan 2020) and can be mitigated by making algorithmic systems more explainable (Dogruel, Masur & Joeckel 2021).

Awareness and knowledge of how algorithms affect online content which users interact with enables them to avoid a wide range of potential risks. Among the latter are the spread of misinformation (Zarouali, Boerman & de Vreese 2021) or forming biased attitudes and decisions (Head, Fister & MacMillan 2020). Additionally, deficits in comprehending the functioning of algorithms in online navigation may further deepen the digital divide which is the most recent form of social inequality (Baidoo-anu & Owusu Ansah 2023, Micheli 2021, Hargittai & Micheli 2019, Lutz 2019)

For the needs of this research, we view AL in the terms, suggested by Dogruel, Masur and Joeckel (2021) – namely, as “users’ awareness about the implementation of algorithms in different areas of internet use and their knowledge regarding the operation of algorithms […]. This mirrors literacy approaches in neighboring domains (e.g., data literacy)” (p. 119).

The projection of the above considerations onto the field of education and educational sciences gives even greater prominence to their various aspects. It should be noted that students in education are currently active users and creators of online content, related to practical, as well as theoretical, academic assignments in the course of their study. Current studies also indicate the need to put a specific focus on algorithm literacy within existing media literacy curricula (Brodsky et al. 2020). In line with this vision, it is our belief that the present research in some way contributes to highlighting the relevance of algorithm literacy and AI comprehension for broadening of the content scope of media education.

2.2. Artificial Intelligence, Algorithms, User Autonomy and Locus of Control User behaviour and data input play a key role in algorithmic decision-making, performed by the digital tools these users operate with. Most studies agree that the greater sense of autonomy or control over algorithmically generated suggestions the users’ experience, the more prone they would be to engage with them (Swart 2021; Kozyreva et al. 2021). Additionally, user autonomy and control over online interactions, on the one hand, and algorithmic awareness as part of AL, on the other, are related (Dogruel, Facciorusso & Stark 2020).

In their study of German and U.S. social media users’ algorithmic awareness Oeldorf-Hirsch and Neubaum (2023) point out that the existing differences in data protection regulations are assumed to impact the survey participants’ perceptions of the extent to which they exert control over algorithmically generated suggestions.

On the other hand, locus of control is a construct which refers to people’s beliefs about the degree of control they can exercise over incidents in their own lives. People with internal locus of control believe that their own behaviour and actions influence life’s occurrences whereas people with external locus of control tend to perceive they have little control or influence over life’s events.

Findings of another study (Sharan & Romano 2020) are also relevant to research on the possible impact of locus of control on interactions with AI-algorithms. In their survey on whether personality traits and locus of control affect trust in human or AI-generated suggestions the authors conclude that interactions with either human or artificial intelligence do not impact survey participants with high internal locus of control.

It can be concluded that Internet users who have developed algorithm literacy are knowledgeable of the way in which algorithmically generated suggestions modify or filter online content. In practical terms, this implies that such users may exert greater control when navigating search engines or social media and protect their personal data. Yet, user agency in managing algorithmic systems as well as in impacting how algorithms operate requires further research.

In the present research we attempt to view these concerns about autonomy and agency in digital interactions from the perspective of the learner-centered educational process, which seeks to support students’ awareness of their own strengths and possible areas for improvement.

2.3. Measuring Algorithm Literacy

To the best of our knowledge, at present there are two validated comprehensive scales which measure this construct in terminologically and conceptually different ways. These are the Algorithmic Media Content Awareness Scale or AMCA-scale (Zarouali, Boerman & de Vreese 2021) and the Algorithm Literacy Scale for Internet users (Dogruel, Masur & Joeckel 2021). Both instruments are focused on the cognitive dimension of the construct AL.

The AMCA-scale “encompasses four components: 1) users’ awareness of content filtering, 2) users’ awareness of automated decision-making, 3) users’ awareness of human-algorithm interplay, and 4) users’ awareness of ethical considerations” ((Zarouali, Boerman & de Vreese 2021, p. 9) and has been tested for three media platforms. The authors consider this instrument to be also helpful in the identification of people with low literacy level that may need educational interventions (Zarouali, Boerman & de Vreese 2021).

In its turn, the Algorithm Literacy Scale for Internet users seeks to establish “internet users’ knowledge about algorithms and awareness of algorithm use” (Dogruel, Masur & Joeckel 2021, p. 118). The items in these two dimensions of the scale refer to “different use contexts instead of specific media platforms or applications” (Dogruel, Masur & Joeckel 2021, p. 130). This instrument is deemed to also be useful for future research which seeks “to assess intervention measures” (Dogruel, Masur & Joeckel 2021, p. 130), aimed at developing users’ algorithm literacy.

2.4. Algorithms, AI, language skills and language acquisition

Studies draw parallels between the structure, evolvement and acquisition of natural languages and algorithmic rule-based logic.

Constructivist perspectives view language as a computational system the structure of which restricts certain outputs (Chomsky 2006) and, therefore, meaning can be represented only through “computations, performed on the discrete symbols” (Evans 2009, p. 130) that constitute language. Further to that, grammar is viewed as a system that executes “computable functions, or rather algorithms” (Mondal 2017, p. 162). Nevertheless, meaning, conveyed with natural language, is regarded as less dependent on systemic restrictions than grammar, allowing for the expression of “varieties of feeling and poetic insight” (Binder 2022, p. 140) and different types of ambiguity (Kumar & Renuka, 2023). A fruitful insight into the dynamical interaction between computational and linguistic logic is brought forward in the proposition that language-based AI systems like GPT-3 “push the distinction between computation and communication to its utmost limits, and they thus provide an occasion to reconsider fundamental assumptions about how computational process relate to language” (Binder 2022, pp. 6 – 7).

Language learning, on the other hand, can also be thought of as an algorithmbased process. The combinations of learning algorithms have had an impact over how languages evolve (Niyogi 2006). Of particular interest are the findings of studies on the usability of AI in the field of second language acquisition. Intelligent software can be conductive to efficient tutoring and error correction - to the extent that it may “contribute to the eradication and the de-fossilization of some of the most common linguistic errors” (Dodigovic 2005, p. 62). Additionally, gains in reading skills can be achieved equally by either algorithm-led or teacher-led interventions or combinations of the two (Serra & Gilabert 2021). Researchers agree that further exploration and analysis is necessary into the role of AI technologies for the personalization of second language acquisition (Lopes, Furtado & Baierle 2023, Serra & Gilabert 2021) and learning in general (Kashive, Powale & Kashive 2021). Benefits for language learners by using a generative AI model, such as ChatGPT, have also been established (Baidoo-Anu & Owusu Ansah 2023).

To conclude, the present study seeks also to give insight into the associations between foreign language skills, and levels of comprehension of generative AI models and algorithm literacy among university students in the field of education.

3. Research method

3.1. Research hypotheses

The Generative Artificial Intelligence Models (GAIM) scale, presented in Appendix A below, has been devised and tested in this research for the first time. Moreover, the two-dimensional Algorithm Literacy Scale for Internet Users (Dogruel, Masur & Joeckel 2021), has been first trialed in Bulgarian settings. Thus, our first goal was to test the reliability of the measures when applied in tertiary educational environment in Bulgaria. Secondly, the following specific alternative hypotheses were statistically tested, namely:

H1: Students in education will exhibit a comparatively low level of familiarity and usage of generative AI models.

H2: Students will exhibit a relatively high level of algorithm literacy.

H3: There will be meaningful association between the students’ scores exhibited on the GAIM and the AL scale.

H4: There will be meaningful associations between the students’ scores on GAIM, AL, LC scales, and their self-reported language proficiency level in English, their coding skills and year of university studies.

H5: There will be meaningful differences on the scales in relation to students’ year of studies and major. In particular, students in media education and art communication will score higher on the GAIM and AL scales compared to their peers on the other university programmes.

3.2. Research design

3.2.1. Participants

The research project was carried out among students at the Faculty of Educational Studies and the Arts of Sofia University “St. Kliment Ohridski” in Bulgaria. Altogether 125 students on the BA programmes of Preschool Education and Foreign Language Teaching (PEFLT), Media Education and Art Communication (MEAC), Preschool and Primary School Education (PPSE), as well as students on the postgraduate programme for English language teachers participated in the survey. All programmes, except for PPSE, train students to become English language teachers. The mean age of the students was 22.27 (S.D. = 5.34; range 19 – 46). The profile of the students who took part in the survey is given in Table 1 and figures 1 – 2 below.

3.2.2. Procedures

The study was conducted in the month of June, 2023 at the end of the summer term at the university. All students on the three BA programmes and the postgraduate programme for English teachers (n = 251) were invited to participate in the survey on a voluntary basis. The overall return rate was 51%. Altogether, 127 survey forms were collected, of which 2 were not filled in correctly, so 125 responses were entered into the final analysis.

The survey was uploaded on Sofia University E-learning Platform Moodle3. Students were invited to enter their e-space there, fill in the survey and upload the filled in forms on the platform. Data from students’ responses in the survey were then checked for outliers, missing or invalid data before they were entered into the final analysis.

3.2.3. Instruments

The survey questionnaire comprised three scales, namely: Generative Artificial Intelligence Models Scale (Appendix A), Algorithm Literacy Scale for Internet Users (Dogruel, Masur & Joeckel 2021), and Locus of Control Scale (Rotter 1966). Additionally, students were asked to self-assess their foreign (i.e. English) language proficiency level and their coding skills (i.e. skills for writing computer programs) on a 5-point Likert scale from 1 (negligible) to 5 (excellent). All instruments are self-reported measures.

Generative Artificial Intelligence Models Scale is a new instrument designed to be used and tested in this research for the first time. It has been developed as a self-reported measure to test respondents’ knowledge and use of generative AI models. It is a 13-item dichotomous scale on which participants receive 0 or 1 point depending on their choice of statement on each item. All items are formulated as a multiple choice of three possible types of answers, i.e. ‘a true statement’; ‘a false statement’ and ‘I am not sure’. All responses which fall into the ‘I am not sure’ category receive 0 points. The instrument was translated into English language and the English version is presented in Appendix A below.

Algorithm Literacy Scale for Internet Users (Dogruel, Masur & Joeckel 2021) is a new instrument which comprises 22 items grouped evenly on two related subscales: The Algorithm Knowledge Scale (n=11) and The Algorithm Awareness Scale (n=11). It is a dichotomous scale on which participants receive 0 or 1 point depending on their choice of statement on each item. All items are formulated as a multiple choice of three possible types of answers, i.e. ‘a true statement’; ‘a false statement’ and ‘I do not know’. Five items on the Algorithm Knowledge Scale are reversed. All responses which fall into the ‘I do not know’ category receive 0 points. Consent for translation and application of the original instrument in Bulgarian context was received by the scale authors.

Locus of Control Scale (Rotter 1966) is believed to be to this date the most popular scale to measure the construct. A Bulgarian translation of the original version has been used in this research. It comprises 29 items in total. All items are in the form of paired-statements, of which 23 measure the participants’ locus of control and 6 are filler-items which do not refer to the construct but serve as a measure to divert the participants’ intense focus on the subject. Participants are invited to choose the statement in each pair which they agree with the most. Respondents receive 0 or 1 point on a 23-item scale depending on their choice of a statement. A high score on the scale will denote external locus of control and a low score - internal locus of control.

4. Research results

Both descriptive and inferential statistics were used in the analysis of data. The Statistical Package for the Social Sciences (SPSS) 23.0 was used in the analysis.

4.1. Reliability and validity

First, we estimated the internal reliability of the scales by computing the Cronbach’s coefficient alpha (α) (Cronbach & Meehl, 1995). Accordingly, it showed to be for: The Generative Artificial Intelligence Models Scale (n = 13, α = 0.78), The Algorithm Literacy Scale for Internet Users (Algorithm Knowledge Subscale (n = 11, α = 0.48) and The Algorithm Awareness Subscale (n = 11, α = 0.61); and Locus of Control Scale (n = 23, α = 0.68). Perceptibly, Cronbach’s coefficient alpha (α) for Algorithm Knowledge subscale showed a lower level than 0.70 which is considered satisfactory. Therefore, reliability of each dimension was also tested by computing “Cronbach’s Alpha if Item Deleted”, i.e. the estimated value of alpha if the given item were removed from the scale. Our results for the AK subscale showed that if three items were dropped alpha would insignificantly increase but not exceed 0.50 altogether. Given the insignificant level of increase in the coefficient alpha, we proceeded with the analysis of data collected on the original scale and all its items as designed and validated by its authors.

Results for the AA subscale showed that if any of the items were excluded from the scale, Cronbach’s alpha would stay lower than or same as Cronbach’s alpha of the scale. This was indicative that no item needed to be dropped off the scale.

Results for the LC scale showed again insignificant improvement of Cronbach’s coefficient alpha if three items were deleted. LC original scale (n = 23) showed acceptable for our analysis reliability so we proceeded with the data collected on the scale without dropping any of its items.

Finally, we tested the reliability of each item on the newly devised Generative Artificial Intelligence Models Scale. All in all, the new measure instrument showed satisfactory reliability with Cronbach’s coefficient alpha above 0.70. Therefore, data from all 13 items were entered into our final analysis.

4.2. Group differences

Secondly, we tested if there were differences found on the different scales in relation to the subject variables of gender, university major and year of studies. We calculated mean and standard deviation for all variables and conducted univariate analysis (one-way ANOVA) for independent sample tests to check if differences found in the groups were statistically significant. The means and standard deviations are shown in Table 2 and the correlation matrix is provided in Table 3.

4.2.1. Gender

It should be noted that there were only 6 male participants altogether, so this was not a sufficient base to draw comparisons or conclusions. Nevertheless, we performed ANOVA analysis to check for possible differences on any of the scales. As expected, all differences were found statistically insignificant: AK (F = 0.272), AA (F = 0.402), LC (F = 0.873), GAIM (F = 1.307), coding skills (F = 0.255) and language skills (F = 0.469) at p > 0.05.

4.2.2. University students’ major

The computed means and standard deviations of scores for the four groups of students according to their studied subject at university are presented below.

As is seen in the table above, congruent to our first two research hypotheses (H1 and H2) focusing on defining students’ algorithm literacy and knowledge and use of AI models, students on all programmes in education displayed relatively high level of AK (with mean total of correct responses: 6.92 out of 11 items) and AA (with mean total of correct responses: 7.80 out of 11 items). Their scores on the GAIM scale were visibly lower (with mean total of correct responses: 4.06 out of 13 items). The low total mean score of 1.86 (out of a maximum mean score of 5) for students self-reported coding skills did not come as a surprise since respondents are studying humanities, and not computer sciences.

The analysis of variance (one-way ANOVA) results confirmed that group differences were found statistically significant for GAIM (F = 12.531, p = 0.00), coding skills (F = 3.905, p = 0.01) and English language skills (F = 4.817, p = 0.00). There were no statistically significant differences found among the four groups in relation to their mean scores on the other scales. Results revealed insignificant group differences for AK (F = 2.172), AA (F = 2.057) and LC (F =2.173) at p > 0.05. As hypothesized (H5), students in media education and art communication scored the highest on the GAIM scale and AK subscale (AL).

4.3. Correlations

Next, the correlations of all variables were calculated to measure the direction and strength of the relationship between the different constructs. Pearson correlation coefficient (r) which indicates the existence or non-existence of such a relationship was computed. The results are presented below.

As is seen in the table above, regarding our next research hypotheses, statistically significant correlations were found between the two subscales of the Algorithm Literacy Scale for Internet Users (AL) as well as the GAIM scale and the AK subscale (AL)(H3). Also, as hypothesized (H4) students’ proficiency level in English was positively related to GAIM and AK subscale (AL). Students’ knowledge and use of generative AI models displayed a negative association with students’ year of studies at university which was quite expected. Students with higher level of proficiency in English exhibited a more external locus of control, whereas students in their last years of studies at university displayed a more internal locus of control compared to those in the first years at university.

5. Discussion

The present research, conducted at the Faculty of Educational Studies and the Arts of Sofia University “St. Kliment Ohridski”, aimed at exploring students in education’s level of algorithm literacy skills, skills for generative AI model comprehension and proficiency in English language. The rationale for this choice of construct exploration was the fact that respondents engage with and create online digital content on a regular basis for their academic assignments. As future teachers, both undergraduate and postgraduate students in education in our faculty are expected to interact with digital platforms and media in a way which is ethically and professionally straightforward and conductive to the implementation of good teaching practices. Comprehension of the uses and limitations of generative AI models as well as the understanding of how algorithms function fall within the wide scope of digital competences that educators need to cultivate in order to meet their learners’ needs. It is our belief that knowledge of generative AI and skills in AL will enhance the quality of materials and activities, designed by teaching professionals for pedagogical interaction with preschool and primary school children, as well as for creation of online media content for young audiences.

Special interest was paid to a group of students in media education and art communication (MEAC). This is the only BA programme of its kind in Bulgaria, offered to students at Sofia University. It is open to a fairly small amount of applicants and at the time of the study there were altogether 48 students enrolled on it (Year 1 – Year 4). As hypothesized, MEAC students exhibited high scores on most variables (AK, AA, GAIM, LC, language skills, etc.) and in most instances they scored the highest among all groups, distinctively on AK, GAIM, and language skills. These findings are in line with an observation, made by J. Swart in her exploratory study, conducted among Dutch young people aged 16 – 26, to the effect that media literacy curricula need to address how algorithms modify online content in much greater depth (Swart 2021). With a view to currently inconclusive data in different studies, we would support the recommendation that “future research might consider to investigate how algorithmic awareness relates to digital literacy” (Zarouali, Boerman & de Vreese 2021, p. 10), as well as to media literacy.

As anticipated, the newly developed and trialed Generative Artificial Intelligence Models Scale (GAIM) correlated significantly with the AK subscale (AL). However, it showed no meaningful correlation to the AA subscale (AL). GAIM scale also correlated positively with the students’ self-reported language skills and negatively with the students’ year of studies. It was compatible with our belief that younger students in their first years of studies at university would exhibit more interest in and knowledge of generative AI models compared to their older peers on the BA and postgraduate programmes. These findings are consistent with the assumption that “algorithm literacy should […] be negatively related to age” (Dogruel, Masur & Joeckel 2021, p. 120). Additionally, in the study conducted by Oeldorf-Hirsch and Neubaum (2023) which measures respondents’ algorithm awareness with the AMCA-scale (Zarouali, Boerman & de Vreese 2021) age of respondents has the same significant negative relation to algorithm awareness. Furthermore, the positive correlation between respondents’ self-reported language skills and GAIM may be interpreted along the lines of suggested recommendations for inclusion of generative AI systems like ChatGPT in language teaching as an additional interactive tool (Baidoo-anu & Owusu Ansah 2023).

Students’ coding skills did not show significant correlations with any of the other constructs measured in this study. Students’ English language skills, though, displayed a meaningful relationship with AK but not the AA dimension of the Algorithm Literacy Scale for Internet Users (Dogruel, Masur & Joeckel 2021). Furthermore, students’ language skills displayed a positive statistically significant but low correlation to ‘external locus of control’ type on the Locus of Control (LC) scale (Rotter 1966). Findings from previous studies at Sofia University revealed meaningful associations between students’ capacity to empathize and internal locus of control (Sofronieva, 2020). Another research on locus of control and English language skills among Bulgarian students (Shopov & Sofronieva 2018) showed relationship between the two constructs but, unlike the present findings, the relationship was found to be statistically significant in the other direction, i.e. language skills were found to be related to internal locus of control. Obviously, further research on how locus of control relates to other constructs will be of value.

Year of students’ studies at university also displayed a low negative correlation to the construct, measured on the LC scale which means that younger students display a more external locus of control, whereas older students in their last years of studies display a more internal locus of control, i.e. they feel they are more in control of their own studies and the learning processes. These findings can be viewed in the light of Sharan and Romano’s survey (2020) on locus of control and trust in human-generated and AI-generated suggestions. The study measured locus of control with a scale other than LC scale applied in the present research, yet the conclusion that AI generated suggestions are received with greater trust than human made ones (Sharan & Romano 2020) may benefit from additional research within the context of algorithm literacy.

6. Conclusions and summary

The research conducted among students in education at Sofia University revealed a constellation of associations between different constructs.

In terms of future work, further research is needed to test the reliability and validity of the GAIM scale. Also, a larger study on students’ language proficiency level, gender, university major, etc. in relation to these scales and other constructs may provide valuable educational insights. It could enhance understanding of young people’s traits, skills, inclinations and preferences so that educators can accommodate their students’ needs most effectively.

Finally, as suggested by the findings in this study, we remain in support of further research on and promotion of algorithm literacy and comprehension of AI technologies in order to ensure users’ autonomy, knowledge and confidence. The present survey attempts to initiate a novel perspective to the understanding of how the educational process evolves within the new digital reality. The implications of the established associations between the various constructs, explored in the study, merit further careful verification and consideration. They are posing different questions and perspectives which, we believe, are of immediate interest to teachers, educators and other pertinent stakeholders.

Appendix A

Generative Artificial Intelligence Models: Knowing and Using Them (GAIM scale items)

1. Are you familiar with the AI generated chatbot ChatGPT?

2. Do you use the chatbot ChatGPT?

3. Do you use the paid version of ChatGPT?

4. Are you knowledgeable about the technical principles underlying the algorithm of ChatGPT, and how it operates?

5. Do you know how to elicit as precise responses from ChatGPT as possible by using differently formulated prompts?

6. Do you use the generative AI model for image generation DALL-E/DALL-E 2?

7. Have you got any experience with other generative AI models?

8. Can you ascertain any measurable benefits of using AI so far?

9. Do you think that ChatGPT can have an effect on the sphere of education over the next three years?

10. Have you used ChatGPT or any other generative AI model for study purposes, related to acquiring an educational degree?

11. Would you support the statement that the current AI potential and benefits outweigh any threats and harms?

12. Would you pay the same monthly subscription fee for the use of the paid version of a generative AI model as the one you are used to paying to a mobile services telecom operator (monthly phone/data transfer plan)?

13. Do you know any users of ChatGPT, DALL-E or another generative AI model?

English language version: S. Markov, C. Beleva, E. Sofronieva, G. Georgieva, Sofia University “St. Kliment Ohridski”, 2023

NOTES

1. EUROPEAN COMMISSION. Shaping Europe’s digital future. 2023 Report on the state of the Digital Decade, Annex: Bulgaria. Available from: https://digitalstrategy.ec.europa.eu/en/library/2023-report-state-digital-decade [Viewed 2023-10-20).

2. REPUBLIC OF BULGARIA. NATIONAL STATISTICAL INSTITUTE. Reasons for not buying over the Internet. Available from: https://www.nsi.bg/en/ content/2839/reasons-not-buying-over-internet [Viewed 2023-10-20].

3. SOFIA UNIVERSITY E-LEARNING. Home page. Available from: https://elearn.uni-sofia.bg/?lang=en [Viewed 2023-10-30].

Acknowledgments & Funding

This study is financed by the European Union - NextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria, project SUMMIT № BG-RRP-2.004-0008-C01

REFERENCES

BAIDOO-ANU, D.; OWUSU ANSAH, L., 2023. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Journal of AI, vol. 7, no. 1, pp. 52 – 62 [viewed 27 October 2023].

BANH, L.; STROBEL, G., 2023. Generative artificial intelligence. Electronic Markets, vol. 33, no. 63, pp. 1 – 17 [viewed 12 December 2023]. Available from: https://doi.org/10.1007/s12525-023-00680-1.

BINDER, J., 2022. Language and the Rise of the Algorithm. Chicago: University of Chicago Press. ISBN 978-0-226-82253-2.

BRODSKY, J., ET AL., 2020. Assessing and fostering college students’ algorithm awareness across online contexts. Journal of Media Literacy Education, vol. 12, no. 3, pp. 43 – 57 [viewed 26 October 2023]. Available from: https://doi.org/10.23860/JMLE-2020-12-3-5.

CHOMSKY, N., 2006. Biolinguistics and the human capacity. New York: Cambridge University Press. ISBN 978-0-521-67493-5.

CRONBACH, L.; MEEHL, P., 1955. Construct validity in psychological tests. Psychological Bulletin, vol. 52, no. 4, pp. 281 – 302. ISSN: 00332909.

DODIGOVIC, M., 2005. Artificial Intelligence in Second Language Learning: Raising Error Awareness, Clevedon: Multilingual Matters Ltd. ISBN 978-1853598302.

DOGRUEL, L.; MASUR, P.; JOECKEL, S., 2021. Development and Validation of an Algorithm Literacy Scale for Internet Users. Communications Methods and Measures, vol. 16, no. 2, pp. 115 – 133 [viewed 23 May 2023]. Available from: https://doi.org/10.1080/19312458.2021.19 68361.

DOGRUEL, L.; FACCIORUSSO, D.; STARK, B., 2020. ‘I’m still the master of the machine.’ Internet users’ awareness of algorithmic decision-making and their perception of its effect on their autonomy. Information, Communication & Society, vol. 25, no. 9, рр. 1311 – 1332 [viewed 9 July 2023]. Available from: https://doi.org/10.1080/136911 8X.2020.1863999.

EVANS, J., 2009. The Emergence of Language. In: E. HOFF & M. SHATZ (Eds.). Blackwell Handbook of Language Development, pp. 128 – 147. Chichester: Blackwell Publishing Ltd. ISBN 978-1-4051-9459-4.

HARGITTAI, E.; MICHELI, M., 2019. Internet skills and why they matter. In: M. GRAHAM & W. H. DUTTON (Eds.). Society and the Internet: how networks of information and communication are changing our lives (2nd edn), pp. 109 – 124. Oxford: Oxford University Press. ISBN 9780-1988-4349-8.

HEAD, A.; FISTER, B.; MACMILLAN, M., 2020. Information literacy in the age of algorithms: Student experiences with news and information, and the need for change. Project Information Research Institute.

HERRERA, F., 2023. Towards Responsible Artificial Intelligence Systems: Safety and Trustworthiness. In: J. KOFROŇ, T. MARGARIA & C. SECELEANU (Eds.). Engineering of Computer-Based Systems. 8th International Conference, pp. 7 – 1 [viewed 12 December 2023]. Available from: https://doi.org/10.1007/978-3-031-49252-5.

KASHIVE, N.; POWALE, L.; KASHIVE, K., 2021. Understanding user perception toward artificial intelligence (AI) enabled e-learning. International Journal of Information and Learning Technology, vol. 38, no. 1, pp. 1 – 19 [viewed 7 November 2023]. Available from: https://doi. org/10.1108/IJILT-05-2020-0090.

KOZYREVA, A., et al., 2021. Public attitudes towards algorithmic personalization and use of personal data online: evidence from Germany, Great Britain, and the United States. Humanities and Social Sciences Communications, vol. 8, no. 117, pp. 1 – 11 [viewed 2 October 2023]. Available from: https://doi.org/10.1057/s41599-021-00787-w.

KUMAR, L.; RENUKA, D., 2023. Deep Learning Approach for Natural Language Processing, Speech and Computer Vision: Techniques and Use Cases. Boca Raton, FL: CRC Press. ISBN 978-1-003-34868-9.

LOPES, G.; FURTADO, J.; BAIERLE, I., 2023. How E-learning Is Correlated with Competitiveness and Innovation and Critical Success Factors. Education Sciences, vol. 13, no. 6, p. 619 [viewed 30 September 2023]. Available from: https://doi.org/10.3390/educsci13060619.

LUTZ, C., 2019. Digital inequalities in the age of artificial intelligence and big data. Human Behavior and Emerging Technologies, vol. 1, no. 2, pp. 141 – 148 [viewed 3 November 2023]. Available from: https://doi. org/10.1002/hbe2.140.

MICHELI, M., 2021. Avoiding Facebook: low-income youths’ (negative) discourses about Facebook. In: E. HARGITTAI (Ed.) Handbook of Digital Inequality, pp. 148 – 164. Northampton, MA: Edward Elgar Publishing Inc. ISBN 978-1-78811-656-5.

MILLER, T., 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, vol. 267, pp. 1 – 38 [viewed 5 October 2023]. Available from: https://doi.org/10.1016/j.artint.2018.07.007.

MONDAL, P., 2017. Natural Language and Possible Minds: How Language Uncovers the Cognitive Landscape. Leiden: Brill Rodopi: Leiden. ISBN 978-90-04-34420-4.

NIYOGI, P., 2006. The Computational Nature of Language Learning and Evolution. Cambridge, MA: MIT Press. ISBN 978-0-262-28068-6

OELDORF-HIRSCH, A.; NEUBAUM, G., 2023. Attitudinal and behavioral correlates of algorithmic awareness among German and U.S. social media users. Journal of Computer-Mediated Communication, vol. 28, no. 5, pp. 1 – 12 [viewed 4 June 2023]. Available from: https:// doi.org/10.1093/jcmc/zmad035.

OGUNNIYE, G.; KÖKCIYAN, N., 2023. A Survey on Understanding and Representing Privacy in the Internet-of-Things. Journal of Artificial Intelligence Research, vol. 76, pp. 163 – 192 [viewed 2 November 2023]. Available from: https://doi.org/10.1613/jair.1.14000.

PEYTCHEVA-FORSYTH, R.; ALEKSIEVA, L.; YOVKOVA, B., 2018. The impact of prior experience of e-learning and e-assessment on students’ and teachers’ approaches to the use of a student authentication and authorship checking system. In: L. CHOVA, A. MARTINEZ & I. TORRES (Eds.). EDULEARN18 Proceedings. 10th International Conference on Education and New Learning Technologies, pp. 2311 – 2321. Valencia: IATED Academy. ISBN 978-84-09-02709-5.

ROTTER, J. B., 1996. Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs: General and Applied, vol. 80, no. 1, pp. 1 – 28 [viewed 3 June 2023]. Available from: https://doi.org/10.1037/h0092976.

SERRA, J.; GILABERT, R., 2021. Algorithmic versus teacher-led sequencing in a digital serious game and the development of second language reading fluency and accuracy. British Journal of Educational Technology, vol. 52, no. 5, pp. 1898 – 1916 [viewed 26 September 2023]. Available from: https://doi.org/10.1111/bjet.13131.

SHARAN, N.; ROMANO, D., 2020. The effects of personality and locus of control on trust in humans versus artificial intelligence. Heliyon, vol. 6, no. 8, pp. 1 – 12 [viewed 11 October 2023]. Available from: https://doi. org/10.1016/j.heliyon.2020.e04572.

SHIN, D.; RASUL, A.; FOTIADIS, A., 2022. Why am I seeing this? Deconstructing algorithm literacy through the lens of users. Internet Research, vol. 32, no. 4, pp. 1214 – 1234 [viewed 19 September 2023]. Available from: https://doi.org/10.1108/INTR-02-2021-0087.

SHOPOV, T.; SOFRONIEVA, E. 2018. Vsichko da stava za pouka. Prouchvane na faktorite, vliyaeshti na uspeshnoto uchene na vtori ezik. Sofia: Sv. Kliment Ohridski. ISBN 978-954-07-4366-0.

SOFRONIEVA, E., 2020. The Empathy Concept: Measures and Application in Education. Sofia: Sofia University Press. ISBN 978-954-075026-2.

SWART, J., 2021. Experiencing Algorithms: How Young People Understand, Feel About, and Engage with Algorithmic News Selection on Social Media. Social Media + Society, vol. 7, no. 2, pp. 1 – 11 [viewed 4 October 2023]. Available from: https://doi.org/10.1177/20563051211008828.

ZAROUALI, B.; BOERMAN, S.; DE VREESE, C., 2021. Is this recommended by an algorithm? The development and validation of the algorithmic media content awareness scale (AMCA-scale). Telematics and Informatics, vol. 62, pp. 1 – 12 [viewed 27 May 2023]. Available from: https://doi.org/10.1016/j.tele.2021.101607.