Образователни технологии
MATHEMATICAL MODELLING IN LEARNING OUTCOMES ASSESSMENT (BINARY MODEL FOR THE ASSESSMMENT OF STUDENT’S COMPETENCES FORMATION)
Резюме. In modern educational institutions there is a necessity for effective assessment methods of competence formation. Both achievement and summative assessment of students’ progress needs to be produced with due expertise. One of the effective ways of improving students’ qualification assessment is through the development and use of a competence formation assessment model. The article presents mathematical models of the subject area as well as competence formation assessment models. These models are determined by subsets of operators, i.e. evaluation operations themselves and necessary materials and equipment. In general, the obtained models describe the structure of the competence formation assessment system corresponding to the disciplines of the particular study field curriculum. The models take into account the specifics of the subject area. They are expandable and customizable.
Ключови думи: mathematical modeling; theory of polychromatic sets; competence formation assessment; indicator
Introduction
Currently, there is an urge to develop students’ competence formation assessment models for the universities. This demand has evolved due to the use of a competence-based approach to assess learning outcomes in higher education. The application of a competence formation assessment model will automate the process of forming a competence for a particular area of training. It will also make the final assessment marked by a definite grademuch easier (Demenkova & Demenkov, 2017), (Azhmukhamedov, Knyazeva, Davidyuk & Gurskaya, 2017), (Algazin & Chudova, 2009), (Sibikina, Kvyatkovskaya & Kosmacheva, 2014), (Kvyatkovskaya, 2012), (Demenkov & Demenkova, 2016).
Based on the theory of polychromatic sets, we are going to consider a mathematical model for students’ competence formation. We are going to study:
– the system of operators describing the disciplines of a particular area of training in the curriculum, – the system of operators that determine the competences formed through training, – the system of indicators assessing the degree of professional competence formation measured according to the World Skills standards of professional excellence for the relevant training areas. Here we are going to examine the applied labor functions and assessment criteria.
The mathematical model of the subject area
The content of the particular direction curriculum A can be represented by a synergistic polychromatic set of the form
(1)\[ S(A)=\left(A, F(A),\left[A^{\prime} F(A)\right]\right), \]
where A is the set of disciplines of the curriculum, \(\mathrm{F}(\mathrm{A})\) is the set of properties of the curriculum, which are presented by the formed competences and indicators modeled by the set A of the curriculum, as well as the correspondence of the properties to the elements of the set A -Boolean matrix \(\left[A^{\prime} F(A)\right]\).
In the process of the research, we are classifying the properties. We build on the approach, where a set of properties is defined as a set consisting of three subsets. That is, \(\mathrm{F}(A)=\left(F(A)^{I}, F(A)^{I I}, F(A)^{I I I}\right.\) ) ), where \(F(A)^{I}\) are the competences; \(F(A)^{I I}\) are the indicators, which help to determine the degree of competence formation; \(F(A)\) \({ }^{I I I}\) is the level of required formation.
Moreover, each of the considered subsets can also be divided into subsets. For example, a group of competences can be divided into general cultural, general professional and other competences; a group of competence formation levels may imply the gradation of competence formation levels (first, second, etc.), i. e.
\[ \begin{gathered} F(A)^{I}=\left(F_{l}^{I}, F_{l}^{I}, F_{3}^{I} \ldots F^{I}\right)=F(A)^{C} \\ F(A)^{I I}=F(A)^{I N D} \\ F(A)^{I I I}=\left({F_{l}^{I I I}}^{I I},{F_{2}}_{I I I}, F_{3}^{I I I}, F_{4}^{I I I}\right) F(A)^{L E V} \end{gathered} \]
Then, the structure of the curriculum A can be modeled using the theory of polychromatic sets and graphs. In this case, the elements of the set describe the disciplines of the curriculum, i.e. the composition of plan elements in the PS-set is represented by the set (Pavlov, 2002), (Pavlov, 2006).
(2)\[ A=\left(a_{1}, \ldots, a_{i}, \ldots, a_{n}\right) \]
Each element of the curriculum has its own qualitative and quantitative properties. Here we are speaking about the competences, the formation of which is evaluated on the basis of the constructed model. The properties of competences will be the indicators, with the help of which competence formation is assessed.
The degree of correspondence of the properties of the simulated curriculum (object) to predetermined values is of great importance. Therefore, the main factor determining the properties of the PS-set is the conditions for the existence of properties and their combinations.
According to the conditions for the existence of all properties of the curriculum \(F\left(a_{i}\right)\), the relationship between them is represented as a subset of the Cartesian product \(F\left(a_{i}\right)^{\prime} F\left(a_{i}\right)\) i)´F(ai) in the form of a Boolean matrix (Pavlov, 2002), (Harari, 1973), (Steuer, 1992).
(3)\[ \left\|c_{k(j)}\right\|=\left[F\left(a_{i}\right)^{\prime} F\left(a_{i}\right)\right], \]
where
\[ C_{k(j)}=\left\{\begin{array}{c} 1, \text { if relations between } F_{k}\left(a_{i}\right) \text { and } F_{j}\left(a_{i}\right) \text { exist } \\ 0, \text { in other cases } \end{array}\right. \]
The relationship of the unitary properties of the model as a whole is modeled by a Boolean matrix:
(4)\[ \left\|c_{i(j)}\right\|=\left[F(A)^{\prime} F(A)\right] . \]
The matrix (4) presents the links between competences and indicators. To create a set of indicators when checking the formation of a competence, it is enough to look at the line corresponding to the competence and write out the indicators that have non-zero values. Thus, the vector \(L_{i}(F)\) of the test for each competency forms.
The existence of any unitary property \(F_{j}(A)\) depends on the existence of certain individual properties of the curriculum elements. This feature of unitary properties in F (A) is modeled using a Boolean matrix
(5)\[ \left\|c_{i(j)}\right\|=\left[A^{\prime} F(A)\right], \]
where \[ C_{k(j)}=\left\{\begin{array}{c} 1, \text { if the existence of } a_{i} \in A \text { impacts the existence of } F_{k}\left(a_{i}\right) \\ 0, \text { in other cases } \end{array}\right. \]
The sets of disciplines, their individual and unitary properties, Boolean matrices (3), (4) define the structural mathematical model \(\mathrm{S}(\mathrm{A})\) of curriculum A :
(6)\[ S(A)=\left(A, F(a), F(A),\left[A^{\prime} A\right],\left[F(A)^{\prime} F(A)\right],\left[A^{\prime} F(A)\right]\right) . \]
This is the way to describe the structural mathematical model \(\mathrm{S}(\mathrm{A})\) by the PS-set.
Table 1. presents the matrix of competencies formed in the study of curriculum disciplines as a Boolean matrix \(\left[A^{\prime} F^{C}(A)\right]\).
Table 1. Competence Matrix
If a competence is developed during the study of several disciplines, then a set of indicators will determine the level of competence formation specified by the curriculum.
Table 2. shows an example of the definition of interrelation between the indicators and competences as a Boolean matrix \(\left[F^{C}(A)^{\prime} F^{I N D}(A)\right]\) ).
Table 2. The Matrix of the Relationship of Competencies and Indicators
Thus, the content part of the initial data is determined by the sets A and F (A), and \(\mathrm{F}(\mathrm{A})\), and the relationships are described by Boolean matrices \(\left[A^{\prime} F^{C}(A)\right]\) и \(\left.\left[F^{C}(A)^{\prime} F^{I N D}(A)\right]\right)\).
For competence formation assessment the connections between model elements and their properties are particularly important. The curriculum under study can be presented in the form of a graph, for the formalization of which the Boolean matrix \(\left[A^{\prime} A\right]\) can be used. Structurally, the graph allows to understand the sequence of studying disciplines in the years of study and their relationship in the formation of a certain competence.
Mathematical model forcompetence formation assessment
We can propose the following method for competence formation assessment based on the presented model:
1. determine the competencethe formation of which will be assessed;
2. determine the disciplines which form the competence based on the matrix \(\left[A^{\prime} F^{C}(A)\right] ;\)
3. determine the list of indicators to be checkedaccording to the matrix \(\left[F^{C}(A)^{\prime} F^{I N D}(A)\right] ;\)
4. conduct a direct assessment to determine the values of indicators.
We recommend using the labor functions and evaluation criteria as defined by WorldSkills standards. In this case, it is possible to use Boolean matrices to obtain objective estimates;
5. producethe assessment.
In order to conclude that the competence is formed at the level of the matrix \(\left[A^{\prime} F^{L E V}(A)\right]\), the equality of 1 conjunction of indicators is necessary (after the assessment).
Then, the result of the assessment will be:
– assessment is completed, the competence is formed;
– assessment is completed, the competence is not formed.
For example, for \(F^{C}\) the list of indicators is (\(F^{I N D}, F^{I N D}\) ), so the competenceis formed if \(\left(F^{I N D}{ }_{1}{ }^{\prime} F^{I N D}{ }_{2}\right)\) is 1 , and not formed if \(\left(F^{I N D}{ }_{1}^{\prime} F^{\prime I N D}{ }_{2}\right)\) is 0.
It should be noted that testing can be used in the proposed model. For this, one of the indicators will be presented by a private model and expanded to the size of the tasks numberin the test. The answer to each question of the test should not be quantitatively measurable (“true” / “false”).
Mathematical model for competence formation assessment system in the curriculum disciplines
Modeling of the assessment system can be performed for each selected indicator using Boolean matrices.Indicators identification will be based on the analysis of professional standards and criteria defined by the WorldSkills standards. The process will be focused on the “be able” and “possess” components, which allow for the determination of:
– the ability to apply the acquired knowledge for the future profession, which means the opportunity to assess the preparedness of students as specialists;
– students’ readiness for demonstration exams and independent examinations.
All this will result in an increased trust of the employer to the educational organization.
We will consider a system for competence formation assessment at the set-theoretical level as a sets of operators union characterizing the processes of assessment (T) and the material objects necessary for the assessment (P)
\[ P=T \cup \Pi \] where
(8)\[ T=\left(t_{1}, \ldots, t_{i}, \ldots, t_{v}\right), \]
where \(t_{l}, \ldots, t_{l}, \ldots, t_{v}\) are the tasks, solving specific problems, practical skills demonstration, assessed as “completed / not completed” and so on.
The composition of operators and their properties at the set-theoretical level are represented by the sets of these elements and their individual properties. The set of unitary properties represent the evaluation system features as a whole. The process structure is modeled by thesets of operators and their individual and unitary properties.
\[ \Pi=\left(\pi_{1}, \ldots, \pi_{p}, \ldots, \pi_{w}\right) \] where \(\pi_{1}, \ldots, \pi_{j}, \ldots, \pi_{w}\)- equipment, consumables, laboratory equipment, software used, etc.
The proposed model allows for the definition of different options for indicators’ combination to verify the goals achievement at different levels of competence formation without a reference to specific disciplines. This may entail requirements for the formation of various options for the elements of an assessment system.
Thus, the model will allow for checking the competence formation. The inspector will have several assessment options. This will in some way optimize the assessment process, as well as expand the range of evaluated indicators.
The composition of the combinationsof evaluation operations (operators) properties and material objects are determined by the property matrices
(11)\[ \begin{aligned} \left\|c_{i(j)}\right\|_{T, F(T)} & =\left[T^{\prime} F(T)\right], \\ \left\|c_{i(j)}\right\|_{\Pi, F(I)} & =\left[\Pi^{\prime} F(\Pi)\right], \end{aligned} \]
Then the composition of the properties combinations of the evaluation system elements can be determined by the Boolean matrix of properties
(13)\[ \left\|c_{i(j)}\right\|_{P, F(P)}=[P \times F(P)] . \]
Modeling the process of competence formation assessment is implemented in the form of the \(T_{i}\) process as an ordered sequence of assessment operations (operators)
(14)\[ T_{i}=\left(t_{i 1}, t_{i 2}, \ldots, t_{i k-1}, t_{i k}, \ldots, t_{i n}\right), \]
which impact themodel properties of the assessed discipline competence \(F\left(A_{i}\right)\) of the curriculum \(A_{i}\). The properties of the model transform it from the previous \(F\left(A_{i}\right)_{k-1}\) to the next \(F\left(A_{i}\right)_{k}\) state. Herewith, \(\mathrm{F}(\mathrm{T})=\mathrm{F}(\mathrm{P})[10]\).
The relationship between the properties in the combinations of the assessment properties, is conjunctive during each of the checks of each competence.All the properties (competences and indicators inside them) must be evaluated.
If all the competences in a particular curriculum discipline are formed, this indicates that the discipline has been studied successfully. An effective study of all the curriculum disciplines indicates a successful acquisition of the curriculum.
In addition, the presented model can be used, for example, in assessing the residual knowledge of the competence. This happens in the situation when a competence is assessed without any reference to a specific discipline but as a separate unit.
The use of the matrix [ \(F^{C} \times F^{I N D}\) ] allows for the definition of various options for a specific competence formation assessment. Further on, the generated options can also be evaluated by an expert (using the assessment system P) according to the criterion of its production cost and then the best option can be selected.
In this case, the operation algorithm may be as follows:
1. Determine the disciplines of the curriculum, chosen for the assessment of the quality \(a_{i \text {. }}\)
2. Use the matrix \(\left[A^{\prime} F^{C}(A)\right]\) and determine the vector \(L\left(a_{i}\right)\) of theevaluated competences.
3. Use the matrix \(\left[F^{C}(A)^{\prime} F^{I N D}(A)\right]\) (vector \(L_{i, j}\) ), for competence \(F^{C}\left(a_{i}\right)\) and determine the set of indicators to be checked.
4. Define indicator values.
5. Calculate the values of the vector \(L_{i^{\prime}}\). If \(L_{i}=1\). Then we can talk about the formation of the competence. Otherwise, we cannot do it.
6. Determine theresulting valuefor the discipline acquisition.
Conclusion Thus, the proposed models allow for:
– the description of the structure of the competence formation assessment system corresponding to the disciplines of the curriculum;
– the description of the relationship between a competence and competence indicators;
– the modification of the competence formation assessment system taking into account the specifics of the subject area;
– the evaluation of both the formation of a selected curriculum competence and acquisition of the whole curriculum (either by course or by an individual discipline).
REFERENCES
Pavlov, V. V. (2002). СALS-technologies in mechanical engineering (mathematical models). Ed. by Solomentsev. Moscow: STANKIN, 328 p.
Pavlov, V. V. (2006). Structural modeling in CALS-technologies. Ed. By Solomentsev. Institute of Design and Technological Informatics RAS. Moscow: Nauka, 307 p.
Steuer, P. (1992). Multi-criteria optimization. Theory, Computing, and Applications. Moscow: Radio and communications, 504 p.
Harari, F. (1973). Graph Theory. Moscow: Mir, 300 p.
Demenkova, E. A. & Demenkov, M.E. (2017). Modeling the process of qualification assessment. Teaching Information Technologies in the Russian Federation: Sat. Materials XV Rev. All-Russian. conf. Arkhangelsk: Publishing House of NArFU, pp. 126 – 128.
Azhmukhamedov, I. M., Knyazeva, O. M., Davidyuk, N. V. & Gurskaya, T. G. (2017). Assessment of university graduates’s competency in the direction of training “Information Security” on the basis of a fuzzy cognitive approach. Bulletin of the Astrakhan State Technical University. Series: Management, Computing and Informatics. No. 1, pp. 115 – 124.
Algazin, G. I. & Chudova, O. V. (2009). Information technology of a comprehensive competency assessment of the university. Bulletin of Novosibirsk state university. Information technologyseries. V. 7 ( 3), pp. 70 – 78.
Sibikina, I. V., Kvyatkovskaya, I. Yu. & Kosmacheva, I. M. (2014). Assessment of the competency level of the university student with the use of a graph model. Bulletin of Saratovstate technical university. No 2, pp. 179 – 185.
Kvyatkovskaya, I. Yu. (2012). Entropy approach to the task of determining the system characteristics of the competency model. Scientific and technical bulletin of SPbSPU. Computer science series. Telecommunications. Control. No. 3, pp. 89 – 93.
Demenkov, M. E. & Demenkova, E. A. (2016). Structural modeling of technological processes. Development of the Northern Arctic region: problems and solutions. Materials of the scientific conference of the faculty, researchers and graduate students of the Northern (Arctic) Federal University named after M.V. Lomonosov. pp. 921 – 925.