Резюме. The misconceptions and error patterns of students are crucial to assess in order to support effective physics learning. However, exploring students’ misconceptions and error patterns is not easy. A web-based diagnostic test is an alternative approach to help the teacher in obtaining such information and giving quick feedback to students. In this article, we report the development of a new model of the web-based diagnostic test. The model uses isomorphic multiple-choice items, in which each learning indicator consists of three multiple-choice items with distracters that are based on students’ misconceptions or error patterns. The model provides feedback to students and teachers based on the consistency of the students’ answers. According to experts’ reviews, the model is valid and worth using. The preliminary field test of the model of the web-based diagnostic test has been used with some physics students. The web-based diagnostic test could be a new alternative for physics teachers and researchers to identify students’ misconceptions and error patterns.

Ключови думи: isomorphic item; misconception and error pattern; web-based diagnostic system

Introduction

Physics teaching needs to pay attention to students’ prior conceptions to improve its effectiveness (Martín-Blas et al., 2010; Türkmen & Usta, 2007). Some students’ prior conceptions are different from experts’ understanding, and are known as misconceptions (Demirci, 2005; Ince & Yilmaz 2012; Treagust, 2006). Students’ misconceptions should be diagnosed, discussed, and cognitively conflicted with other physical phenomena in order to facilitate the students to achieve a better understanding (Chen et al., 2007; Chen et al., 2013; Ince & Yilmaz, 2012; Muller & Sharma, 2007). Once learning and instruction neglect students’ misconceptions, the students will have learning difficulties and it causes poor mastery of concepts by students (Martín-Blas et al., 2010). Furthermore, it can also result in students’ failure in solving physics problems (Pathare & Pradhan, 2004).

Therefore, an effective approach to diagnose students’ misconceptions and error patterns is strongly required (Kusairi, 2012). Accurate information about students’ misconceptions and error patterns is important for teachers to implement the appropriate learning strategy (Chen et al., 2013). Feedback on students’ misconceptions is also needed by students to improve their mastery of concepts (Black et al., 2003; Irons, 2008; Tridane et al. , 2015). The interview, essay test, and concept maps are known as precise approaches to reveal students’ misconceptions. However, the implementation and scoring are time-consuming (Gurel et al., 2015), especially as the average number of students in Indonesia is relatively large. Therefore, a diagnostic test system that reveals students’ misconceptions and gives specific and timely feedback for teachers and students is essential.

Various studies have been widely conducted in identifying students’ misconceptions with the aid of diagnostic tests using a computer (Ahmad et al., 2010; Kim et al., 2007; Lai, 2007; Russell et al., 2009; Susono & Shimomura, 2006; Thissen-Roe et al., 2004). The computer-based diagnostic test can provide quick feedback to the teachers and students. Some researchers have reported that they used multiple-choice problems with rational distracters (King et al., 2004; Lin et al., 2010). The use of multiple-choice problems has a disadvantage in estimating students’ misconceptions (Anderson et al., 2002). To overcome that problem, two-tier and three-tier based problems were introduced (Chandrasegaran et al., 2007; Kirbulut & Geban, 2014; Kaltakci & Eryilmaz, 2008; Chen et al., 2002). However, the two-tier and three-tier based problems also have some disadvantages regarding many possible alternatives of answers which leads to usage difficulty of these diagnostic tests. The three-tier problems also serve a speculative answer (Gurel et al., 2015). Unspecific feedback creates another difficulty for further learning (Kusairi, 2012). Other researchers have done some studies on developing diagnostic tests (Chang et al., 2007). However, the diagnostic tests did not use webbased technology so that they could not provide timely and effective feedback to teachers and students.

Related to the circumstances above, the Department of Physics from Universitas Negeri Malang (UM) has developed a web-based diagnostic test with real-time feedback for teachers and students. The web-based diagnostic test system adopted isomorphic multiple-choice problems (Singh, 2008) with distracters that represent students’ error patterns (Aderonmu & Nte, 2014; Ahmad et al., 2010; Larson & Kelleher, 2009) and students’ misconceptions (Lin et al., 2010). The diagnostic test was designed to facilitate physics teachers to assess students’ error patterns and misconceptions. In the beginning, the diagnostic system was used in the Department of Physics UM (Kusairi, 2012), but could also be applied to any major.

Experimental method

The aim of this study is to develop a model of a web-based diagnostic test. It involed three experts and 13 lecturrers. The experts are those who have a doctorate degree and are experienced in developing learning and assessment in higher education. These experts assess the feasibility of the model and provide input to the development of the model. Lecturers are teachers that administer the basic physics course. They were tasked to pilot the feasibility of the model, and develop diagnostic problems. The lecturers used the diagnostic test for some courses in Universitas Negeri Malang and Universitas Kanjuruhan. They also gave feedback related to the technique of the developed model. There were 202 first year undergraduates who were involved in the implementation of the web-based diagnostic test. In this present study, we presented architectural models that have been developed and initial test results of the students.

Design of web-based diagnostic test

A web-based diagnostic test is a web-based tool that allows the designing and editing of diagnostic items, thus creating and delivering a diagnostic test. It also can deliver feedback to the students and report individual students’ and students’ group misconceptions and errors patterns. The system is an improvement of computer-based analysis tools that have been developed by the researcher starting in 2011. Initially, it was developed specifically for basic physics lectures at universities. It can also be applied to physics learning among high school students.

The web-based diagnostic test has some excellent characteristics as follows: (1) It can serve formative purposes, which are meant to help student learning; (2) It can give graphical feedback to physics teachers concerning the students’ misconceptions and errors patterns; (3) The feedback is also provided for students to help them overcome their misconceptions and error patterns; (4) A student can access the diagnostic test on-line; (4) It has a database system to save the history of student misconceptions and error patterns that can be referred to in the learning process; and (5) It is also applicable to another subject.

The structure of the web-based diagnostic tests system is can be seen in Fig. 1.

Fig. 1 shows that there are two interfaces that can be accessed, namely the interface for teachers and students. On the part of teachers, it is useful to design, manage, and run the test. On the part of the student, it is useful to access diagnostic tests that have been provided. Immediately after the test is completed, students will receive feedback on misconceptions or error patterns experienced by them. Information on the test results of students, both individually and in a group, are also immediately obtainable by the teacher. Processing data and settings are automatically performed by the main program.

All users must have an account to be able to take advantage of the web-based diagnostic tests. Teachers and students can access the web-based diagnostic test after getting a username and password from the administrator. Subjects taught by teachers are also determined by the administrator. The teacher can fill topics in a diagnostic test, write test descriptions, and provide the date and duration of the test as illustrated in Fig. 2.

Once the topics management form has been filled, the next step that lecturers need to fill is the form of learning indicators and categories of every distracter as illustrated Fig. 3. Learning indicators and this description is required to provide information or feedback to students.

The last step that the teacher needs to do is to fill out test items as shown in Fig. 4. The writing of test items is the same as usual test items, but each option needs to be determined by the category.

Based on the answers given, the student can view and print the feedback given by the computer quickly. A report of the results of the test consists of individual student profiles and class (group) profiles. The class profile includes reporting diagnostic results for all students. The test report contains students’ profiles. The class profile and a summary of the test results of the students in the class can be accessed by the teacher by using the name of the class. Examples of group profiles for the learning indicator “Applying Newton’s 1st Law” can be seen in Fig. 5. From these graphs, it is clearly seen that there are different characteristics of every class even for the same indicator. Category 3 diagnostic tends to be dominating for the class report. This result, at least, provides important information for the teachers to evaluate the learning and instruction processes.

Question items

The web-based diagnostic test model requires well-prepared question items with isomorphic characteristics and rational distracters. There are three multiple-choice problems for every indicator that contain distracters for evaluating students’ misunderstandings, misconceptions, or error patterns. The following steps were followed in developing the items: developing an indicator, developing a description for the correct answer and distracters, and developing isomorphic problems. A sample of isomorphic problems is given as follows.

Fig. 6 shows an example of the development of learning indicators and one of the items of diagnostic test that has been developed.

In addition, Fig. 7 shows how the web-based diagnostic test reveals students’ misconceptions or error patterns based on their answers to the isomorphic multiple-choice items. In the implementation, the computer will show the questions in a random way to ensure that the students do not realize that they faced problems with the same indicator. For every three isomorphic questions, if the students are able to answer two or more questions correctly, then it is decided that they understand the concept well. Also, if the students choose category 1 answers twice or more, the students are classified to have a misconception at category 1, etc. If the students choose a different category of answers, they will be classified as “inconsistent”, suggesting that they do not have proper understanding and tend to answer the problems randomly.

Preliminary field test

A preliminary field test was conducted with 202 students. It aimed to test the feasibility of web-based diagnostic tests. Students were from public (165 students) and private universities (37 students). One college involved a prospective student science teacher, while another was a prospective physics teacher candidate. All students involved took a course in basic physics. Diagnostic tests were administered to students consisting of three indicators that were translated into nine multiple-choice items. Table 2 shows the example of a classical report on the learning indicator, First Newton’s Law. The results show that there is a difference between the students. Students of private universities, for example, had a level not higher than the others.

Category 1: Student has misconception that a body at rest is due to heavier weight of the “puller” than the weight of the object

Category 2: Student has the misconception that a body at rest is due to the presence of weaker external force than object’s weight.

Category 3: Student has the misconception that a body at rest is due to the presence of stronger external force than frictional force.

Category 4: Student able to apply Newton’s 1st law for a body at rest correctly.

To ensure the feasibility of web-based diagnostic tests that have been developed, expert validation process of the models was carried out. Validation of the model involved three experts comprising lecturers who had a doctorate in physics. In general, the expert stated that the web-based diagnostic test was suitable for use in physics teaching, but the diagnostic quality is determined by the quality of items used in the test. In addition, web-based diagnostic tests were also introduced to 18 physics professors. They were also asked to complete a questionnaire relating to web-based diagnostic tests. Some of the results of the questionnaire are presented in Table 3.

These results indicate the need for teachers to have access to the diagnostic test tools that can provide information about misconceptions and the difficulties experienced by the students, and the need to provide feedback to students; both cannot be done easily. This indicates a need for devices that can assist teachers in implementing learning assessment. Lecturers’ opinions on the benefits of diagnostic test models in helping the learning process were also positive. A large proportion of lecturers argued that the model of diagnostic tests helps lectures that are listed in the table. One of the problems is the difficulty of development multiple-choice isomorphic items. Instruction books are not enough to give an explanation to the lecturer about isomorphic item development. Training is necessary for teachers to develop diagnostic test items.

Discussion

The web-based diagnostic test is an innovation for using computer and web technology for diagnosing students’ misconceptions and error patterns. The web-based diagnostic test can be used by teachers to give students feedback and assess students’ misconceptions and error patterns. The diagnostic test uses multiple-choice isomorphic items with distracters that can examine students’ misconceptions and error patterns. It is similar with other researchers that have used multiple-choice items to diagnose students’ misconceptions (Huang et al., 2011; Dow, 2003; McDonald & Haffanin, 2003) and to analyze students’ error patterns (Ahmad et al., 2010; .Ketterlin-Geller & Yovanoff, 2009; Kim et al., 2007). They used multiple-choice problems due to the several advantages in terms of efficiency in administering, high validity and rapid scoring and ability to cover many topics.

In this study, the diagnostic misconceptions and error patterns employed multiple-choice items with isomorphic characteristics. The isomorphic items consisted of three multiple-choice items with different contexts but having the same learning indicators. The diagnostic method in this test is different from the results of other research that used n-tier multiple-choice items such as two-tiers, three-tiers, and four-tiers (Gurel et al., 2015; Lai & Chen, 2010). The use of isomorphic items has many advantages compared to the three-tier and four-tier items because the confidence level of the testee in answering items was taken into consideration. In connection with the determination of whether the students had misconceptions or lack of knowledge, the use of isomorphic items is much simpler. If the students’ responses to multiple choice isomorphic items are different, then the student is classified as having a lack of knowledge.

On the other hand, the web-based diagnostic test also has some disadvantages. Firstly, the distracters may not cover all of the students’ misconceptions or error patterns. Secondly, the students’ response to isomorphic multiple-choice items is context dependent (Kraus & Minstrell, 2002). Thirdly, there is a possibility for students to cheat during solving the problems because a web-based test can be accessed from out of the classroom. Fourthly, the developing of isomorphic multiple-choice items is relatively difficult. Finally, the number of items becomes three times as many as the common multiple-choice problems. Future studies are expected to review the validity of diagnostic tests and their effectiveness in teaching physics.

Conclusion

The model of web-based diagnostic tests involving misconceptions and error patterns using multiple-choice isomorphic items has been developed. This model can be an alternative for teachers to identify student misconceptions and error patterns. The web-based diagnostic test could be a new alternative for physics teachers and researchers to identify students’ misconceptions and error patterns. As a relatively new diagnostic model, the model needs to be improved and perfected.

REFERENCES

Aderonmu, T.S.B. & Nte, F.U. (2014). Diagnosing error patterns of physics students in solving problems using progressive wave equation (PWE) in senior secondary schools in Rivers state , Nigeria. ARPN J. Sci. & Tech. , 4, 277 – 281.

Ahmad, A., Al-Mashari, A. & Al-Lawati, A. (2010). On the development of a computer based diagnostic assessment tool to help in teaching and learning process. Int. J. Educ. & Devel. Using Inf. & Comm. Tech. 6, 76 – 87.

Anderson, D.L., Fisher, K.M. & Norman, G.J. (2002). Development and evaluation of the conceptual inventory of natural selection. J. Res. Sci. Teaching, 39, 952 – 978.

Black, P.J., Harrison, C., Lee, C., Marshall, B. & William, D. (2003). Assessment for learning : putting it into practice. Maidenhead: Open University Press.

Chandrasegaran, A.L., Treagust, D.F. & Mocerino, M. (2007). The development of a two-tier multiple-choice diagnostic instrument for evaluating secondary school students’ ability to describe and explain chemical reactions using multiple levels of representation. Chem. Educ. Res. Pract. , 8, 293 – 307.

Chang, S.-H., Lin, P.-C. & Lin, Z.-C. (2007). Measures of partial knowledge and unexpected responses in multiple-choice tests. Educ. Tech. & Soc. , 10(4), 95 – 109.

Chen, C.C., Lin, H.S. & Lin, M.L. (2002). Developing a two-tier diagnostic instrument to assess high school students’ understanding the formation of image by plane mirror. Proc. Natl. Sci. Counc. ROC(D), 12, 106 – 121.

Chen, C.-M., Hsieh, Y.-L. & Hsu, S.-H. (2007). Mining learner profile utilizing association rule for web-based learning diagnosis. Expert Systems & Applications, 33, 6 – 22.

Chen, Y-.L., Pan, P.-R., Sung, Y.-T. & Chang, K-.E. (2013). Correcting misconceptions on electronics: effects of a simulation-based learning environment backed by a conceptual change model. Educ. Techn. & Soc. , 16, 212 – 227.

Demirci, N. (2005). A study about students’ misconceptions in force and motion concepts by incorporating a web-Assisted physics program. Turkish Online J. Educ. Techn., 4(3), 40 – 48.

Dow, R.B. (2003). The introduction of web-based formative assessment in medical science: sn action research project. Int. J. Innov. Sci. & Math. Educ., 10, Number 1.

Gurel, D.K., Eryilmaz, A. & McDermott, L.C. (2015). A review and comparison of diagnostic instruments to identify students’misconceptions in science. Eurasia J. Math., Sci. & Tech. Educ. , 11, 989 – 1008.

Huang, J., He, L. & Davidson-Shivers, G.V. (2011). Educational assessment via a web-based intelligent system. US-China Educ. Rev. , 8, 666 – 674.

Ince, E. & Yilmaz, O. (2012). The usage of alternative assessment techniques in determination of misconceptions about electromagnetic field-magnetism contents and effects of video-based experiments on preservice’s achievement. Procedia, 55, 206 – 211.

Irons, A. (2008). Enhancing learning through formative assessment and feedback. London: Roudledge.

Ketterlin-Geller, L.R. & Yovanoff, P.(2009). Diagnostic assessments in mathematics to support instructional decision making. Pract. Assess. Res. & Eval. , 14(16), 1 – 11.

Kim, M., Choi, J. & Song, J. (2007). Developing a web-based system for testing students’ physics misconceptions (WEBSYSTEM) and its implementation. J. Korea Assoc. Res. Sci. Educ., 27, 105 – 119.

King, K.V, Gardner, D.A., Zucker, S. & Jorgensen, M.A. (2004). The distractor rationale taxonomy : enhancing multiple-choice items in reading and mathematics. London: Pearson Education.

Kirbulut, Z.D. & Geban, O. (2014). Using three-tier diagnostic test to assess students’ misconceptions of states of matter. Eurasia J. Math., Sci. & Tech. Educ. 10, 509 – 521.

Kusairi, S. (2012). Analisis asesmen formatif fisika sma berbantuan komputer. Penelitian Dan Evaluasi Pendidikan, Dies Natal, No. 3, 68 – 87.

Kraus, P.A. & Minstrell, J. (2002). Designing diagnostic assessments. Proceed. Physics Educ. Res. Conf., August, 2nd – 5th.

Lai, A.F. (2007). The development of computerized two-tier diagnostic test and remedial learning system for elementary science learning. Proceed. 7th IEEE Int. Conf. Adv. Learning Tech., pp. 735 – 736.

Lai, A.-F. & Chen, D.-J. (2010). Web-based two-tier diagnostic test and remedial learning experiment. Int. J. Distance Educ. Tech. , 8(1), 31 – 53.

Larson, A.F. & Kelleher, M.E. (2009). Error identification assessments: diagnostic instruments for instruction. Paper presented at 2009 AERA Convention, pp. 1 – 41.

Lin, J., Chu, K-.l. & Meng, Y. (2010). Distractor rationale taxonomy : diagnostic assessment of reading with ordered multiple-choice items. London: Person Educatiin.

Martín-Blas, T., Seidel, L. & Serrano-Fernández, A. (2010). Enhancing force concept inventory diagnostics to identify dominant misconceptions in first-year engineering physics. Eur. J. Eng. Educ. , 35, 597 – 606.

McDonald, K.K. & Hannafin, R.D. (2003). Using web-based computer games to meet the demands of today’s high-stakes testing : a mixed method inquiry. J. Res. Tech. Educ., 35, 459 – 472.

Muller, D.A. & Sharma, M.D. (2007). Tackling misconceptions in introductory physics using multimedia presentations. UniServe Science Teaching & Learning Research Proceedings, pp. 58 – 63.

Pathare, S. & Pradhan, H.C. (2004). Students’ alternative conceptions in pressure, heat and temperature. epiSTEME- 1 International Conference to Review Research on Science, Technology, and Mathematics Education, pp. 38 – 41.

Russell, M., O’Dwyer, L.M. & Miranda, H. (2009). Diagnosing students’ misconceptions in algebra: results from an experimental pilot study. Behavior Res. Methods, 41, 414 – 424.

Singh, C. (2008). Assessing student expertise in introductory physics with isomorphic problems. II: effect of some potential factors on problem solving and transfer. Phys. Rev. Special Topics - Physics Educ. Res., 4, art. no. 010105.

Susono, H. & Shimomura, T. (2006). Using mobile phones and QR codes for formative class assessment. Curr. Develop. Tech. - Assisted Educ. , 2, 1006 – 1010.

Thissen-Roe, A., Hunt, E. & Minstrell, J. (2004). The DIAGNOSER project : combining assessment and learning. Behavior Res. Methods, Instruments & Computers, 36, 234 – 240.

Treagust, D.F. (2006). Diagnostic assessment in science as a means to improving teaching, learning and retention. UniServe Science – Sypm. Proceed.: Assessment in Science Teaching and Learnint, pp. 1 – 9.

Tridane, M., Belaaouad, S., Benmokhtar, S., Gourja, B. & Radid, M. (2015). The impact of formative assessment on the learning process and the unreliability of the mark for the summative evaluation. Procedia, 197, 680 – 685.

Türkmen, H. & Usta, E. (2007). The role of learning cycle approach overcoming misconceptions in science. Ekim Kastamonu Educ. J., 15, 491 – 500.