AN INTERNET SITE FOR FRESHMAN CHEMISTRY . By Chung Chieh (e-mail), Department of Chemistry, University of Waterloo, Waterloo, Ontario, Canada N2L3G1	1999 CONFCHEM

6. CACT Quizzes

Quizzes administered at regular intervals demand that students study according to a schedule that we have set for them. Thus, we traditionally have small group tutorials and quizzes. When the DOS CACT was in use, it offered students an opportunity to write quizzes administered by computers in a DOS environment. After CACT was made accessible on the Internet, students had the opportunity to write quizzes over the Internet.

In this section, I will review the traditional Small Group Tutorials and Quizzes, the DOS CACT Quizzes, and the Internet CACT quizzes. I will share my Experience with CACT Quizzes to the extend of addressing issues in computerized quizzes in general. Furthermore, I will give reasons for our Strategies for Quizzes.

6.1 Small Group Tutorial Quizzes

Freshman chemistry classes have increased over the years, and now the classes have approximately 200 students per lecture section. There are several lecture sections in the Science Faculty, totalling more than 1000 freshman chemistry students in the Fall term, and more than 700 in the Winter term. The Department used to give biweekly small-group tutorials to about 30 students, and that class size would have increased considerably had CACT quizzes not been available. At the end of each small-group tutorial, we administer a 15-minute written quiz. Students solve one problem per quiz. The quiz papers are collected and marked by tutors, and they are returned two weeks later. These tutorials are offered so that students can ask questions in a less formal atmosphere. Most students, however, attend them biweekly to write quizzes.

Writting quizzes has always been a part of our Freshman Chemistry courses, the DOS CACT and the Internet CACT administered quizzes are substitutes for written quizzes in the tutorials.

For the past many years, students have written five (5) quizzes by attending 5 small group tutorials or nine to ten computerized quizzes. During 1998-1999, more than 70% of the students opted to write the quizzes over the Internet.

6.2 DOS CACT Quizzes

During the 1980s, our local area network (LAN) of computers was only lightly used. Following several years of running the DOS CACT on the LAN, I implemented the computerized quizzes, which followed the Dialogue format. In the Dialogue, students could answer a question many times, and the marks were not recorded. In quizzes, marks were recorded.

Each CACT Quiz consists of 5 to 10 questions randomly chosen from a pool of questions. Marks were the same for all questions. Partial marks were awarded. Three question types have been used:

multiple choice questions - The student enters a letter to indicate his or her choice,
questions requiring short-answers - The student enters a keyword, symbol, or term,
questions requiring numerical answers - The student must enter a numerical value.

The computer displays one question at a time, and there is no time limit within which to give an answer. As soon as an answer is entered, the computer reveals the mark earned for the question. For a correct answer, the computer returns with a thought provoking comment or question. For an incorrect answer, a hint is given.

Many years ago during a lecture, a student made a suggestion:

"Sir, we know the purpose of CACT Quizzes is to encourage us to learn, why don't you give us a second opportunity to give a correct answer for partial marks after we have seen the Hint?"

The class applauded and the strategy of quizzes was changed thereafter, partly also because I wanted students to read the Hints carefully. The Hints suggested a formula to use or a theory to apply, but the answers were not given. A student could answer a question n times, but the earned mark would be divided by n. We have not carried out research to see if such a strategy indeed led to more learning, but the number of requests to raise their quiz marks was drastically reduced.

At the end of a quiz, the mark earned for the quiz was displayed together with previous quiz marks for the students to review.

6.3 Internet CACT Quizzes

The conversion of the Instruction and Dialogue in the DOS CACT into Internet documents was completed early 1998. At this point, I wanted to begin quizzes over the Internet in the Fall term of 1998. Planning, exploring possible software, modifying, testing, and making up the Internet CACT Quizzes was my project for 1998.

After having explored several possibilities, I decided to modify a locally developed program called WEBTEST.pl for the Internet CACT Quizzes.

During the winter term of 1995, Nevil Bromley and Paul Snyder of the Faculty Projects Group at the University of Waterloo developed a preliminary version of the program WEBTEST.pl. During 1997, there had been some limited testing and applications of the program, with very limited support provided by the Information Systems and Technology (IST).

Together with Wai Chun Li, a former Chemistry graduate working on his Computer Science degree at that time, we got a copy of WEBTEST.pl, and tested it. The question formats used for WEBTEST.pl are similar to those of the DOS CACT Quizzes, but now students can point and click their choices when WEBTEST.pl is used to ademinister the quizzes. Questions requiring keywords and numerical values as answers are handled the same way as the DOS CACT Quizzes.

The program WEBTEST.pl sends quiz marks to a quiz master in the form of an e-mail, and the quiz master has to extract the marks from these e-mails. Thus, students can write the quizzes any number of times. On the other hand, the DOS CACT Quizzes checked the student mark every time he or she started a quiz, and the mark earned during (not after) the quiz is immediately recorded. A student has only one chance to write a quiz. Once they start the quizzes, they have to finish them, because if they stop or turn the computer off, the marks they have earned at that time become their marks for those quizzes.

I decided to modify WEBTEST.pl so that I could incorporate the strategies and designs used for DOS CACT Quizzes, which have worked well. We have consulted many Perl experts about file handling students quiz marks. We eventually used a technique used in the DOS CACT Quizzes.

We have added an interesting and useful feature to the selection of questions from a pool of questions in our modified WEBTEST.pl. We divided the pool of questions into groups, and a definite number of questions can be chosen from each group to make up a quiz. This feature enables the questions to be grouped according to difficulty, topics, style, or type (multiple choice, numeric answers, key-word answers). Thus, several skills can and will be tested in a quiz.

Together, Wai Chun Li and I tested how the system behaved under various conditions. For example, we tested the program as if three students were writing a quiz at the same time. During the past year, we modified the program to accommodate the problems experienced by students.

During the Fall term (September-December, 1998), 650 of the 995 students registered in Freshman Chemistry I (CHEM120) wrote more than 7 quizzes each over the Internet CACT. During the Winter term (January-April), 350 of the 606 students registered in Freshman Chemistry II (CHEM123) have written 8 or more quizzes each. Seven and eight best quiz marks were used to derive an average for the two terms respectively.

April 5, 1999, was the last day of lectures for the Winter term. April 4 (Sunday) was set as the deadlines for two CACT Quizzes. As a result, more than 150 students tried to write their quizzes, and unfortunately, the Internet server went out of service. The system manager thought that heavy usage by students writing Internet CACT Quizzes that day might have contributed to the problem. In the future, deadlines shall be set during office hours on week days to avoid this problem recurring.

A special CACT Quiz has been set up for CONFCHEM participants, whose identity and marks are not checked or recorded.

6.4 Problems of Computerized Quizzes

Computerized quizzes can be given either under the supervision of a proctor in a computer room where the students write the quizzes or unsupervised, in which case, no proctor is present when a student writes a quiz. Students writing unsupervised quizzes over the Internet can do so at any time from anywhere. Problems or issues for the two quiz types are different. Since the CACT Quizzes are unsupervised, the problems addressed here are for the unsupervised quizzes. These problems can be divided into computer related and people related categories:

6.4.1 Computer related problems

Computers freeze (hung) during quizzes.
Power failures (due to thunderstorm for example) during quizzes, or during a period students want to write quizzes.
Failure of the Internet server, especially during the nights and on weekends.
Internet connections interrupted during quizzes.
Workstations in a computer network system may have security measures. For example, all the workstations in the University library restart themselves when no action from the keyboard or mouse is detected within three minutes. After restarted, Internet communication between the workstation and the Internet server is lost.
After a set period, the Windows operating system activates a screen saver. Some students are unable to get back to the Quiz window after the screen saver had been activated during the quiz.

6.4.2 People related problems

Students have claimed experiencing computer related problems, such as frozen computers, power failures, accidental computer reset (or turned off by other students), Internet connection interruption, etc.

Students have claimed that they were incorrectly marked by the computers.

Students reported seeing several of their fellow students working together and helping each other during quizzes.

Students claimed that they had written the quiz, but the computers failed to record their marks. Other students claimed that they were about to write the quiz, and found their quizzes written by others.

Abilities to use computers vary from student to student. Some sophisticated computer users tried to beat the system, whereas some inexperienced students were simply frustrated by computers.

Some years ago, I was sympathetic when a student claimed a computer froze-up during a quiz, and often allowed them to rewrite the quiz. This upset other students, who wrote to tell me about the cheating practices of some of their fellow students by simply turning off their computers and then complaining that the computer froze-up. No, policy is more important than a student's plea.

Correspondence regarding CACT Quizzes have been collected on a web page called Questions and Answers on Quizzes. Students were encouraged to read this page to help them avoid the common mishaps.

6.5 Strategies for CACT Quizzes

On the one hand, the quiz marks must count for something to encourage students to write the quizzes. On the other hand, the weight should be light enough not to make students greedy for (quiz) marks. The goal of CACT Quizzes is to encourage students to learn chemistry, not computer literacy. Only 10% of the final grade is allocated for Quiz marks. Despite the low weight, many students have argued for higher marks.

So far, students must complete the CACT Quizzes once they have started them. This policy prevents them from not submitting for marking unless they have acquired a good mark. On the other hand, the goal of CACT Quizzes is to tempt them to learn, and the WEBTEST.pl has just been changed so that when students get less then 50% in a quiz, they will be allowed to repeat the quizzes in the Fall of 1999. Larger question pools will be made to insure that different sets of questions will be given in all quizzes.

Because of computer related problems, one or two of the worst quiz marks have been ignored. If students are allowed to rewrite failed quizzes, allowing one worst mark to be ignored will be adequate.

Questions in a pool are divided into groups. Most of the time, they are grouped by topics or concepts in a learning unit. For each topic, we prepare a group of questions. One question from each group is chosen randomly to make up a quiz. Thus, all the topics are tested in a quiz, and no quiz will get too many questions from one topic.

In a separate file from the mark file, the Internet CACT Quizzes record the question numbers in the pool and students answers. Students' claims of incorrect marking by computer can be checked by instructors, who can change the record in the mark file.

6.6 Benefits of Internet CACT Quizzes

The educational value of Internet CACT Quizzes is hard to assess, but there are some benefits associated with them. Thus, developing tools for computerized quizzes is a worthwhile venture. For example, Question Mark: is such a business venture to provide tools for computerizing quizzes, tests, assessments and surveys. Furthermore, the San Jose State University has set up the Testing and Evaluation Department to provide consulting services for test construction, survey construction, and data analysis to its own faculties.

Some well known institutions have adopted computerized examinations. In 1998, the Educational Testing Service (ETS) introduced computerized testing for TOEFL examinees in the United States, Canada, Latin America, Europe, the Middle East, Africa, Australia, and selected countries in Asia. Written tests are no longer available. In 1999, the Dental Aptitude Test (DAT) will also replace the written examination by Computerized DAT. The Computer-Based Graduate Record Examinations (GRE) has already replaced its written tests.

The TOEFL, DAT and GRE tests are aimed at assessing a student's ability, rather than for educational purposes. These tests are supervised under controlled conditions. The unsupervised Internet CACT Quizzes are mainly aimed at enticing students to learn or study.

During the past two terms, about 70% of students taking freshman chemistry chose to write the Internet CACT quizzes. In our opinion, these quizzes offer many advantages:

CACT Quizzes provide immediate score reporting and feedback to students to let them know their results. Poor marks serve as a warning of inadequate preparation.
CACT Quizzes give hints or methods for problem solving to help them improve in areas where they need help.
Once set up, the cost of increasing the number of computerized quizzes is very low compared to the cost of providing man power to set, supervise, and mark written quizzes.
Computerized quizzes offers the possibility of using graphics, sound, video etc.
Content and questions in Internet CACT Quizzes can be revised and changed easily on the fly in response to students needs.
Internet CACT Quizzes are at least partially individualized, and all topics are tested.

A freshman chemistry student has a chance to practice writing a CACT Quiz on measurements and units over the Internet. This quiz mark is not counted. The same quiz is made available for the CONFCHEM participants. Participant's identity and mark are not kept. Feel free to give it a try.

E-mail: cchieh@uwaterloo.ca