Sunday, February 7, 2010

Scantron performance assessment, so far

We began the online Scantron Performance Assessment at our school last week. Here are some initial observations about the testing so far.

As a plus, the web-based software has been responsive. I assume there is some significant load on the test servers, since I think most CPS schools are operating within the same test window as us. (I'm not sure of this -- in any case I haven't seen a problem with test server response.) When tests have been interrupted, the Scantron site has been doing a good job starting back up where the students were. (We have had a particular situation where students have accidentally unplugged computers mid-test.) The administration functions are easy to use, and the reports are nice (all caveats regarding the type of test that it is notwithstanding!).

On the negative side, I see a number of issues with the assessment:
  • Younger students (3rd graders in particular) are unfamiliar with the testing medium. Some have limited experience with mouse buttons, especially avoiding the right mouse button. Accidentally pressing the right-mouse button throws up a context menu, and clicking the wrong item seems to exit the test. This confuses them.
  • Then there is the general anxiety about "taking a test". If the regular classroom teacher is not present to help ease student anxieties, I suspect that students will not do as well.
  • There is of course the general stress of test-taking, but this is true of paper-based tests as well (or does the computer have a special authority?) Still, for some students at least, the stress probably has a negative effect on performance.
  • Student focus, behavior, and alertness varies during the school day. Prime testing time seems to be at the very beginning of the day. However, it is not possible to test all students at the beginning of the day (unlike ISAT), given equipment constraints, and the fact that we are administering 3 tests X 15 classrooms = 45 test sessions in a 12-day test window.
  • There is an opportunity cost to the assessment -- diverting teachers from teaching, expropriating technology prep time or lab availability to do assessments, plus the administrative time to organize sessions.
The adaptive aspect of the test means some students may take longer to complete the test. This gives rise to a few issues:
  • The reading assessment contains lengthy passages. After an hour of reading text on-screen, test fatigue starts to set in, which must affect performance. The test actually seemed cruel for some 4th graders who kept seeing additional reading passages to read and answer questions on.
  • Students who complete the test early need to be kept occupied. It is hard enough to keep students still for extended periods, especially if their regular classroom teacher isn't present. If they are in a lab, they want to do fun things with the computer. Splitting the class up (sending students who are done back to their regular classroom) presents additional challenges for both the classroom teacher and the test proctor.
  • Keeping students focused on the test for 45 minutes or an hour can be a challenge.
  • Students who take longer to finish see their peers finish. This looks like it becomes another source of anxiety.
Test integrity is another issue:
  • When students take a paper-based test, desks are re-arranged in rows. In a lab setting, students are sitting next to their fellow test-takers. Students have to be told not to talk to the students next to them, not to be distracted by what is on their screen, not to volunteer answers. Students have to be trained how to take the assessment.
  • Some classes, when inclusion students are present, exceed the number of computers in the lab. Special arrangements have been made (commandeer some laptops from the Parent Resource Center, use an overflow room), but test administration and adequate proctoringbecomes an issue.
  • At least one student had the wits to call up an online dictionary to help with some vocabulary passages. Oops. It is difficult to monitor 35 computer screens at once.
  • I see this one as a big flaw: Testing is active for students outside of school hours. We have one student who completed the test on Sunday from an ATT IP address. I am able to log on as a student and begin and resume testing.
From at least some of the points above, it is clear that the assessment medium is in-between the student and their skills and their understanding. It is not a transparent, neutral thing. I repeat a sentence from above: Students have to be trained how to take the assessment. Not only do they need to know the content being assessed -- they have to master the testing medium ("testing as a genre"). This includes not just the format of text passages, the deliberate deception built into multiple choice answers, the often ambiguous test questions and answers. It also includes things like pacing and focus (testing as an athletic event). With the computer thrown in the mix, they need to master the mouse and keyboard skills, understand how buttons, program flow and the browser works, and deal with physical factors like screen glare, reading text onscreen, and sitting in a chair at a keyboard for extended periods of time.

If this was just part of the overall assessment toolbox for teachers, the negatives above might not be such a big issue -- a good teacher would just factor in the limitations of the testing medium. But this test is different I think -- teacher careers will hang on the results. (See previous posts for the rapid advance of evaluation teachers on the basis of student test performance.)



NMZumpano said...

Hey JD. I liked your take on the Scantron Assessment Testing. Like you, we are within the two week testing window and in the midst of squeezing everyone in. We have the advantage of having given this assessment once before in the fall. Based on that, we've modified this testing block. Here are my thoughts:
1. I agree 100% that the reading passages are far too long. Students don't have the interest or attention span to sit through all of the text only to click "next" and have another full page of text appear. Worse over, as students are allowed to use hightlighters on the ISAT some are used to being able to write in their test booklet. Not possible with the Scantron Assessment.
2. The students know its an adaptive test. They take this to mean "the smarter I answer the longer the test will continue". So....are some students intentionally marking answers wrong to finish quickly?
3. Scantron Assessments are a full time job during the two week window. Everything else gets pushed to the backburner. We chose to have our teachers test their own students. In the fall, here I happily sat when the testing was complete only to see an email from CPS saying "you have 192 students that didn't finish the test. You have 2 days to finish." I wasn't even aware that everyone didn't finish. So....even though we told the teachers it was important to have the students complete each test, I never checked up on them. This time I'm not looking to receive a late night Sunday email. I log on twice daily to see the # of incomplete tests. This creates additional work for me because now I am charged with finding all of these students and pulling them out of their classrooms to finish. Just when I think I'm a leg up on the situation the next day comes. And I'm back to 40+ students not finished. There have even been three ENTIRE classrooms that just...forgot to test.
4. Pick a test....any test. We are testing the kids to death. ISAT practice, weekly assessments, CPS benchmarks, 5 week assessments, and now Scantron Assessments. We spend so much time testing them to see what they know that I don't know if we have any time time teach anymore.

jd said...

Great comments. Thanks for adding to the collective Scantron experience...