In 2018 I introduced online quiz banks to Physics 1A and Physics 1B. The introduction of these banks corresponded with a 12% improvement in final exam marks for students enrolled in Physics 1A, PHYS1121. Getting these quizzes introduced was a long process.

While introducing prelab tests in Moodle for the laboratory component of the Physics 1A and 1B courses it became clear that there was only a very limited set of question types. It was not possible to ask multi-part randomized questions, include a randomized string in a question or use student's responses to earlier parts of a question to calculate the answer to later parts. In order to find out whether there was a suitable solution to this problem I presented at the 2014 moodlemoot in Brisbane presenting what we were doing with our prelab quizzes and asking if anyone new how to overcome these shortcomings [1]. We heard about STACK questions as a possible solution to our problem.

At the end of 2014 I was the academic lead on a grant we submitted to get STACK questions introduced to Moodle at UNSW. This project got funded through alternate means and during 2015 along with John Paul Possada, Julian Cox and Russel Waldron we assessed the viability of STACK questions. The testing was successful and in 2016 the STACK question type was integrated into UNSW Moodle. STACK questions are now used across the university with engineering, business and science all using them extensively.

With the possibility of setting STACK questions it became clear that the assessment of Physics 1A and Physics 1B could be improved. Students perform better on exams when they practice answering questions over the semester and when they practice answering questions under test conditions [2]. In our first year physics courses we were giving students very little opportunity to practice answering exam style questions. Due to large student numbers, there was no time during the semester when all the students had a class at the same time, the mid-semester test was removed shortly before I took over as first year director. Building on the success of my implementation of other online quizzes for labs, I wished to develop a large bank of multi-part questions closely aligned with lecture learning outcomes. I began by clearly defining learning outcomes for each lecture, these are now shared with the team of lecturers teaching into Physics 1A and 1B to ensure lectures remain aligned with assessment.

In 2017 and 2018 I wrote a question bank with 180 three part STACK questions, five based on each lecture, for Physics 1A and 135 (Prof. Joe Wolfe wrote the other 45) three part questions for Physics 1B. These were programmed in STACK as part of a digital uplift of these two courses. The quizzes were implemented in semester 1 2018 for Physics 1A and semester 2 2018 for Physics 1B. Students have six fortnightly at home quizzes that draw randomly from each lecture bank. Twice during the semester the students sit an invigilated in class test where they are assigned four questions drawn randomly from the question bank. The introduction of this strategy correlated with a 12% improvement of the Physics 1A students on the exam from the two previous years. There is excellent correlation between how student perform on these quizzes and their performance on the final exam (see graph below).

During the invigilated quizzes students use Safe Exam Browser (SEB) to access Moodle. The School of Physics IT staff, David Jonas and Kristien Clayton came up with a way we could hit a switch which locked down all the browsers on the computers in the first year lab, greatly reducing the chances of students cheating during these tests. I wrote a series of protocols for staff invigilating the tests. The tests have run very smoothly with very few issues. The risks are very manageable as the questions are pulled from a bank so if anything goes badly wrong the students can resit the test. This has only been required so far for one student out of a total of approximately 5000 who have sat these tests.

In 2018 I presented at the engineering showcase and the Australian Institute of Physics (AIP) congress on the introduction of these quizzes [2,3].

While the students were happy to have the opportunity to practice answering lots of questions they had two main complaints about the new quiz system:

• Different students received different questions in the invigilated quiz which they felt was unfair.
• The quizzes were marked by computer so their working was not marked, just their final answer.

In order to reduce the impact of these limitations as much as possible I made a few changes in semester 2 2018. From student performance on questions in the quiz bank I could guage the level of difficulty of a question. I used this level of difficulty to categorize questions into easy, medium, medium-hard and hard and students get one from each category when they take the invigilated tests. Tom Dixon, Christine Lindstrøm and myself have carefully analysed the quiz data and found that there is a much stronger correlation between student performance and student ability than between student performance and the selection of questions. Tom presented his analysis at a couple of conferences in 2019 [4,5].

Students have the opportunity to check their answers as they go. There is a 33% penalty per question part for each incorrect answer. This gives students the opportunity to fix their answer if they have just made a small mistake in the working. Because the questions are programmed in STACK they use carry on marking, if a student calculates the first part incorrectly and then uses this to calculate the second part they are not penalized, STACK calculates the answer they should give based on their response to the first part.

[1] E. Angstmann, R. Waldron, J. Cox (July 2014) Moodle Quizzes to Improve Learning in Large First Year Physics Classes, moodlemoot au, Cairns

[2] Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58.

[3] E. Angstmann (December 2018) Using online testing effectively in large first year physics courses, AIP congress, Perth

[4] T. Dixon, C. Lindstrøm, E. Angstmann (December  2019) Datamining of online quizzes: student study patterns and outcomes. AIP summer meeting, RMIT

[5] T. Dixon, C. Lindstrøm, E. Angstmann (October 2019) Quantitative assessments: encouraging surface learning in introductory physics? ACSME, Sydney