Thanks to the awesome Grace Chen, I saw the following article yesterday:

The unscientific thinking that forever lingers in the minds of physics professors

Here are the two key paragraphs from the post (emphasis mine):

Deborah Kelemen and her colleagues presented 80 scientists (including physicists, chemists and geographers) with 100 one-sentence statements and their task was to say if each one was true or false. Among the items were teleological statements about nature, such as “Trees produce oxygen so that animals can breathe”. Crucially, half the scientists had to answer under time pressure – just over 3 seconds for each statement – while the others had as long as they liked. There were also control groups of college students and the general public.

Overall, the scientists endorsed fewer of the teleological statements than the control groups (22 per cent vs. 50 per cent approx). No surprise there, given that mainstream science rejects the idea that inanimate objects have purpose, or that there is purposeful design in the natural world. But look at what happened under time pressure. When they were rushed, the scientists endorsed 29 per cent of teleological statements compared with 15 per cent endorsed by the un-rushed scientists. This is consistent with the idea that a tendency to endorse teleological beliefs lingers in the scientists’ minds. This unscientific thinking is usually suppressed, but time pressure undermines that conscious suppression.

So let me get this straight—when professional scientists, who have years of training in a discipline, are forced to assess whether or not scientific statements are true under time pressure, their performance decreases by half and they blow almost one third of the questions, essentially earning a 71%, were this a test in school? That is, professional scientists become C- scientists when exposed to pressured multiple choice testing.

And we think that having novice students with almost no training in a particular science discipline take multiple choice tests that are filled with distractor responses designed to elicit misconceptions is an accurate, fair or useful way to judge students’ scientific understanding?

This also ties in beautifully to the some of the points Michael Pershan was making in his masterful discussion last night of the “Failure(?) of the Math Mistakes Blog”, which you should make a point to listen to. In his talk, Michael talks about why students make mistakes on ideas that they already know, such as why students will say $100^{\frac{1}{2}}=50$, and yet he knows that students know this isn’t true, as they can explain why when confronted with this mistake.

Michael points to a possible explanation for this discrepancy from Daniel Kahneman, author of Thinking Fast and Slow. Khaneman writes:

intuitive capacity] will find a related question that is easier and will answer it. I call the operation of answering one question in place of another substitution.

So the quick, rapidly responding part of our brain that often succumbs to misconceptions and incomplete heuristics will too often leap at answering a question intuitively and incorrectly when faced with time pressure, before the slower, reasoning brain has a chance to process and come to a thoughtful and often correct response to the question when given time to think. My recollection of Kahneman also is that he said that this leaping to the wrong answer can never be eliminated—it can only be reduced by giving ourselves more time to process, often by deliberately introducing mental speed bumps (like drawing a diagram) designed to slow us down and allow our reasoning brain time to process.

So what’s the point of timed tests again? Is it to simply remind our students of how error prone their brains are when operating under high stress and time pressure?

November 29, 2012 6:52 am

I think the point of timed tests is so that students can get to their next class. I’d love to give students an “untimed” test, but ultimately they have to get on with their other studies, and I have to get my next group of students in the door. I don’t think anyone is really in favor of timed tests in an ideal world — but they are one practical solution in a world where time in the school day is scarce.

• November 29, 2012 7:18 am

The point of school is for people to become educated. If a system is not well designed for its purpose then modify the system so that it does its job better. There are a lot of solutions that could be applied so that timed tests actually take into account the reality of human abilities. There could be longer class periods and less classes. Student could be allowed to use free periods, such as study hall, to complete unfinished tests/works. After school and before school hours could be utilized. Tests could be broken up into smaller segments and administered over more than one day. I’m sure there are many other options… even starting with the impulse in our society to shove as many facts into a brain as can possibly (or, rather, impossibly) be achieved and at the expense of exercising critical thinking and problem solving skills.

• November 29, 2012 7:53 am

Absolutely.

• November 29, 2012 7:28 am

You’re right. I’m not really saying that we should have tests that are truly unlimited in terms of time, but instead we should work to reduce the importance of time pressure on assessment in our classes. I think this study is further evidence that multiple choice tests that rely on some element of time pressure (like the SAT II, where you must answer 75 questions in an hour) are likely poor overall measures of a student’s ability to think scientifically and overcome misconceptions.

There are lots of ways to reduce the importance of time pressure in assessment, starting with just asking fewer questions on assessments. Take home tests and lab practica in science are are a couple of others. I’ve personally had good success cutting the content length of my assessments in half—reducing them to one page, and having students take assessments weekly. Students also have more time then to assess their own work on after the test, using Frank Noschese’s method of having students give themselves feedback right after the assessment.

2. November 29, 2012 7:47 am

I think Mister J. hit the nail on the head… the “next thing” is waiting, so let’s get this test over with. On Saturday, some of my swim team is taking the SAT’s. We’ve moved the meet to 3 pm to accomodate this testing. But that’s just to make sure they can get to warm-ups after working on the timed test until 12:30. So, test is over at 12:30 because, goodness knows, we need to have a sports event.

So why don’t we just ban sports events on testing days? That doesn’t work because nearly every other Saturday seems like a testing day! We’ve replaced Saturday’s “go outside and have fun playing a sport or working on the musical” with “sit inside and take another shot at the SAT or ACT.” Most of our kids will take the PSAT, the SAT three times and the ACT. Then throw in a couple of weekends for the SAT subject tests. Whew.

One way to look at this is to compare the time spent taking standardized test to the time I spent trying to relate a student’s character and intellectual strengths in a recommendation letter. If I spent as much time writing each kid’s recommendation as they spent sitting for standardized tests, it would take me more than two months working full time to write recommendations.

3. November 29, 2012 8:02 am

Time pressure MATTERS.

4. November 29, 2012 10:14 am

Timed tests also reinforce the misconceptions so there’s even more that has to be unlearned, and reinforce the students’ habits of never using the slower, reasoning brain on tests (and believing they don’t have one anyway).

5. November 29, 2012 5:12 pm

I could be wrong, but it seems to me that the point of the Kelemen research was to show a difference between believing teleological statements vs fact, which isn’t necessary a case of “right vs wrong”. For example, a scientist who is religious would most likely answer “true” to all or many of the teleological statements, which is what probably accounts for the average un-rushed scientist being at 15% (I don’t think our professional scientists are “B students” in science…).

Of course, having said that, I believe that rushing students increases the likelihood of making mistakes, I’m there’s probably tons of research out there supporting that. I think one of the reasons a test like the ACT is super-timed (1 min per problem on the math, I believe) is to separate students who just “know how to answer it” and students who “know how to answer it quickly” which sometimes, but doesn’t always translate into “students who know how to answer a question in more than one way because they have a more complete understanding of the problem”.

Having said *that*, I also believe in giving students as much time as possible on tests and quizzes. I usually let them come back and finish tests and quizzes, although I discourage it (or at least make it look like I don’t want to do this) because I don’t want students to take advantage of the system–doing something like going out and finding out all the answers from friends and then coming back and finishing the quiz or test.

I think that a bigger problem than “students underperforming when time-pressured” is that “not all students work/learn at the same rate” and the related question: is *rate* of work/learning the same as *quality* of work/learning? (I guess the ACT thinks so, to some degree) It is very possible that a more capable student learns slower than the average student, which usually gets him/her put into a lower level class, which is a tragedy.

• November 29, 2012 6:33 pm

I think you’re right about the research being focused on distinguished teleological statements vs fact, and I guess I’m stretching to call a teleological statement like “trees produce oxygen so animals can breathe” a misconception. I’ve tracking down the complete paper and look forward to reading it in detail.

But I agree with your larger point—putting students into situations, like timed multiple choice tests, that trigger our often easily mislead and wrong intuitive capacities, and therefore do a very poor job of measuring what we usually claim to be interested in, namely thoughtful reasoning.

• November 29, 2012 8:34 pm

But we aren’t *only* interested in ‘thoughtful reasoning’. We’re also interested (as a society and me personally as an educator) in ‘working under pressure’, ‘meeting deadlines’, ‘overcoming adversity’ and a bunch of other stuff that timed testing DOES ‘measure’ and help to promote.

If you are telling me that timed testing isn’t great at *necessarily* identifying the deepest thinkers, then I would agree (although very often it actually DOES), but that isn’t what schools are exclusively about.

We’re in danger of producing a generation of deep thinkers that will be useless in the real world, since all of the competitiveness, combativeness and chutzpah will have been coddled out of them.

• December 3, 2012 10:39 am

I think the greater danger is the trend towards “the fast answer — and sell it well enough so people think it’s right!” That, IMHO, is not deep thinking; it’s marketing, which has its place, but not for selling oversimplifications.

6. November 29, 2012 8:02 pm

Of course, people can do better with more time. Since there are suggestions, IIRC correctly a study on first-person shooters and quickness of response, that the brain can be trained to be able to respond more quickly, what kinds of classroom practices can improve this skill? If timed tests don’t do it, what about timed practices but untimed tests? If that doesn’t work, what could help students become faster, better, stronger?