Scantron performance assessment, so far

We began the online Scantron Performance Assessment at our school last week. Here are some initial observations about the testing so far.

As a plus, the web-based software has been responsive. I assume there is some significant load on the test servers, since I think most CPS schools are operating within the same test window as us. (I'm not sure of this -- in any case I haven't seen a problem with test server response.) When tests have been interrupted, the Scantron site has been doing a good job starting back up where the students were. (We have had a particular situation where students have accidentally unplugged computers mid-test.) The administration functions are easy to use, and the reports are nice (all caveats regarding the type of test that it is notwithstanding!).

On the negative side, I see a number of issues with the assessment:

• Younger students (3rd graders in particular) are unfamiliar with the testing medium. Some have limited experience with mouse buttons, especially avoiding the right mouse button. Accidentally pressing the right-mouse button throws up a context menu, and clicking the wrong item seems to exit the test. This confuses them.
• Then there is the general anxiety about "taking a test". If the regular classroom teacher is not present to help ease student anxieties, I suspect that students will not do as well.
• There is of course the general stress of test-taking, but this is true of paper-based tests as well (or does the computer have a special authority?) Still, for some students at least, the stress probably has a negative effect on performance.
• Student focus, behavior, and alertness varies during the school day. Prime testing time seems to be at the very beginning of the day. However, it is not possible to test all students at the beginning of the day (unlike ISAT), given equipment constraints, and the fact that we are administering 3 tests X 15 classrooms = 45 test sessions in a 12-day test window.
• There is an opportunity cost to the assessment -- diverting teachers from teaching, expropriating technology prep time or lab availability to do assessments, plus the administrative time to organize sessions.
  

The adaptive aspect of the test means some students may take longer to complete the test. This gives rise to a few issues:

• The reading assessment contains lengthy passages. After an hour of reading text on-screen, test fatigue starts to set in, which must affect performance. The test actually seemed cruel for some 4th graders who kept seeing additional reading passages to read and answer questions on.
• Students who complete the test early need to be kept occupied. It is hard enough to keep students still for extended periods, especially if their regular classroom teacher isn't present. If they are in a lab, they want to do fun things with the computer. Splitting the class up (sending students who are done back to their regular classroom) presents additional challenges for both the classroom teacher and the test proctor.
• Keeping students focused on the test for 45 minutes or an hour can be a challenge.
• Students who take longer to finish see their peers finish. This looks like it becomes another source of anxiety.

Test integrity is another issue:

• When students take a paper-based test, desks are re-arranged in rows. In a lab setting, students are sitting next to their fellow test-takers. Students have to be told not to talk to the students next to them, not to be distracted by what is on their screen, not to volunteer answers. Students have to be trained how to take the assessment.
• Some classes, when inclusion students are present, exceed the number of computers in the lab. Special arrangements have been made (commandeer some laptops from the Parent Resource Center, use an overflow room), but test administration and adequate proctoringbecomes an issue.
• At least one student had the wits to call up an online dictionary to help with some vocabulary passages. Oops. It is difficult to monitor 35 computer screens at once.
• I see this one as a big flaw: Testing is active for students outside of school hours. We have one student who completed the test on Sunday from an ATT IP address. I am able to log on as a student and begin and resume testing.

From at least some of the points above, it is clear that the assessment medium is in-between the student and their skills and their understanding. It is not a transparent, neutral thing. I repeat a sentence from above: Students have to be trained how to take the assessment. Not only do they need to know the content being assessed -- they have to master the testing medium ("testing as a genre"). This includes not just the format of text passages, the deliberate deception built into multiple choice answers, the often ambiguous test questions and answers. It also includes things like pacing and focus (testing as an athletic event). With the computer thrown in the mix, they need to master the mouse and keyboard skills, understand how buttons, program flow and the browser works, and deal with physical factors like screen glare, reading text onscreen, and sitting in a chair at a keyboard for extended periods of time.

If this was just part of the overall assessment toolbox for teachers, the negatives above might not be such a big issue -- a good teacher would just factor in the limitations of the testing medium. But this test is different I think -- teacher careers will hang on the results. (See previous posts for the rapid advance of evaluation teachers on the basis of student test performance.)

jd