Proof of concept: could students learn more if Khan made mistakes?
So after a thought provoking comment by Mylene, and Frank’s great post on how you need to confront misconceptions in science videos for students to really learn, I got to wondering how effective it would be to create a series of screencasts featuring me solving problems incorrectly, and then challenging students to find the mistake in my work. It took all of about 5 minutes to put this together, and I’m thinking it would be great to build up a library of these.
You can see I’ve got a way to go before my video skills can match Khan’s, but I think there’s something too this. It seems intuitively obvious that if kids have to watch a video to find a mistake, rather than just watching it to get the “yeah, this makes sense feeling” they’ll pay much more attention to it. And when they find the mistake, might this make them more cognizant of the same mistake in their own thinking?
Quantum Progress 
I recently came across some research (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0004957) that apparently demonstrated the ‘switching off’ of some important decision-making portions of the brain when confronted with advice from experts. At the risk of reading too much into one study, I do worry that many of the kids who need the most help trust us too much, and are willing to cede all responsibility to the experts. I could definitely see how the videos that you’re talking about could re-engage the critical-thinking, decision-making portions of the brain that I would imagine shut down when watching Kahn do his thing.
That paper looks interesting, I really wish I had the funding/resources/expertise to do some fmri studies of my students while solving physics problems. I do think one downside that Joss mentions is that it’s possible that students don’t find the mistake at all, so I think I would need to be very explicit about informing students that these videos contain mistakes and their job is to find them.
I remember reading something about one professor who said that to keep everyone on his toes, he would intentionally tell one lie every day, and if I recall, there was some sort of forum and reward for discussing and identifying the lies. Then one day, no one could find the lie, and it turned out the “I will tell one lie each class” was the lie.
Hi John,
I’ve been looking into the ISLE physics curriculum developed by the Rutger’s PAER group and the “find what is wrong with the solution” problem is one of their common evaluation tasks, along with jeopardy questions which I think Frank posted or tweeted about recently, as well as lots of multiple representation tasks. Anywho you can find some examples in here: http://paer.rutgers.edu/ScientificAbilities/Kits/Mech/MechEval/default.aspx
I like your take on this. Using these videos to give them a learning task that is embedded in a larger mini-lesson / example.
I use pre-lecture reading assignments (in the style similar to Just-in-Time Teaching), but am planning to slowly transition to vide0-based delivery of the pre-lecture material instead of asking them to read. Your embedded mistakes videos for simple exercises strike me as a great additional pre-lecture task to add to my arsenal.
What seems interesting is if a student has a deeply held misconception and just can’t see the mistake that you have made. Of course this would be very question-dependent, but do they end up finding a different “mistake” based on their misconception, or do they just chase their tail around for a while trying to find the mistake and then eventually throw their hand up in the air in defeat?
Joss,
I do think you’d need to be pretty intentional about how you design these problems, at least at first, so that students can’t go too far off the rails and find mistakes in things that are correct. This is why it would be good, probably to over-explain the solution, showing each step, so that students could essentially test each statement for the truthfulness. This seems to be somewhat tied to the Tree of Reasoning problems that Josh writes about over at Newton’s Minions.
Really nice first-go at this! What programs are you using to record, and separately to draw?
What I wonder if this approach works differently for “mistakes” vs. “misconceptions”. My hunch is that there’s a big difference. The normal force “mistake” shown here seems like, to me, an over-generalization error caused by seeing to many examples where N & W are equal and opposite. . However, there is a “misconception” that is often talked about is where students think objects like “ramps” and “tables” don’t exert forces at all–they just get in the way.
I also wonder if these kinds of video may be useful for teacher prep– to not only have teachers learn about “student misconceptions” or “difficulties”, but to develop the skill for noticing those misconceptions and difficulties.
I used the smart notebook software to write (I give it a solid meh) and an old wacaom wireless graphire tablet (meh as well), I also recorded everything using Camtasia (pure awesomeness). I think Andy could tell you that the newer bamboo tablets (or the intous wireless, if you got money to burn) are far superior.
I think you’re right on the mistakes vs misconceptions front. I think you could probably do something similar for both, but I’m not sure if it would be as easy for students to identify/explain misconceptions. This is something worth testing out.
And I like the idea of using these for teacher prep. It’d be fun to create a Khan equivalent filled with incorrectly solved problems. I would just pity the naive student who stumbles upon it.
Math teacher checking in here…
First, let me say I have really been enjoying reading this blog for the past month or so and I’ve been sharing it with some science colleagues. Three points regarding the ‘identify the mistake’ idea
1) I had a professor who argued, unconvincingly in my mind, that we should not make mistakes like these because some students will only remember that they saw us do something – not that we pointed out that what we did was wrong. I disagree but I do keep that in mind when this comes up in class.
2) I used to write exercises where I presented two solution ideas to the same problem and the students’ job was to identify which of the two was the correct solution path.
3) I had a colleague who wrote exercises with a recurring character called ‘Careless Carl’ who was always making mistakes. She would present his work on tests or quizzes and the students’ job was to identify his mistake AND correct the solution themselves. I always liked these exercises although I think it would be more effective if she opened up the possibility that Carl might be right sometimes.
I distinctly remember my 8th grade history teacher writing in some grad school paper she let us read about how she liked having students find mistakes in newspapers, but not textbooks, since it might make them think the textbooks are flawed. That seemed strange to me then, and even stranger now, when I really want my students to learn from mistakes, shouldn’t I show them that we all make mistakes.
I like the two competing solutions idea, and as Mark said, we’ve done something very similar to Careless Carl, and the kids really love it.
Nice idea! Donald Knuth famously offers a cash reward if you find a mistake in The Art of Computer Programming. It’s become such a high honor to find a mistake (and get your name listed as someone who found one) that I bet there are people who go over every algorithm with a fine-toothed comb, just in hope of catching something. Unfortunately, most of the time all they wind up doing is learning a lot.
Yes, I forgot about Knuth’s famous rewards, and didn’t even think of the connection as to how that might help students to learn. If/when I write a textbook, I will definitely do this.
I like the “two solutions” model that Jim Doherty mentions. I could see alternating between good solutions, “find the mistake” and “find the right solution” type of videos.
“Careless Carl” makes appearances in my class all the time, as John knows. He went by the name Bobo when John and I taught together, but he’s now going by Throckmorton. I like this device, and I started to write about it here, but it got too long. I’ll put something on my blog about it instead. The upshot is I think I have a way of avoiding what Joss mentions about “throwing their hands up in the air in defeat.”
I think it might be fun to use voicethread to let the student add their commentary to what they think is wrong. I’ve been trying to find a way to use voicethread and this seems like a great way to start.
great idea Andy. I too had been looking for a way to make VT useful in physics, but never thought of this.
That sounds like a great idea. At the same time, my students and I have had much frustration with the video feature on voicethread. It works great for pictures and voice, though:)
I like the “find my mistakes” idea. We’ve used that. We also use the “make a challenging test” up for the teacher. My kids will very happily design tests in the hopes of tripping us up & then they enjoy grading it. My younger son deducted 5% for procrastination from one of the tests that my husband put off for a few days.
Alright folks. I am teaching two sections of Calc-based Mechanics in the fall and am seeing if I can piece together a study that would look at the learning benefit of these “mistakes in the video” or “find the mistake” tasks in general. In terms of an ethical study, both sections would have to spend equal time being control and experimental groups so the learning effect of these “effects in the video” would have to be evaluated somewhere around the topic-scale level.
So the first question is what does the control look like? If one group is asked to “find the mistake” and perhaps even correct it, what sort of equal-time-on-task activity does the control group do? Two examples instead?
I could also do some controls with something that falls in the “interactive engagement” category. As an in-class activity, I could do a “mistake on purpose” example with the experimental group and what I call a clicker-based example in the control group. The quicker version of the clicker-based example is one where I take an example that would be worked at the board in a traditional course and identify the 2-4 critical steps that students would have the most trouble with in the example and develop clicker questions for those steps. I then will work the example, but stop at these critical points and run a peer-instruction sequence on the relevant clicker question. I think these two different approaches to examples offer similar levels of interactive engagement.
I have enough individual assessments built into my courses (homework, weekly quizzes, late-in-the-term term test, final exam) that I should be able to test them on their learning of the concept multiple times for the purpose of comparing control and exerimental groups.
Thoughts?
Joss,
I like how quickly your one-man PER group works. How about having your experimental group have to find the error in the video, while your control group watches a video either where you find the error and explain it, or where you simply solve the problem correctly?
Yup, my research group meetings involve very few battle of wills.
I’m think it would be very compelling if one could show that one interactive example leads to more learning than two traditionally worked examples. I keep on wanting to set that up for my clicker-based examples, but this upcoming double section is the first time I will have the opportunity to do this. In terms of time-on-task, asking them to (1) find the error, explain it and then finish off the solution properly, and (2) watch idly as two examples are worked by me should be about the same time-on-task if the ones from (2) are comparably difficult to the find the error example. I can also compare both (1) and (2) to clicker-based examples.
This concept of having students watch the video looking for an error is brilliant! Suddenly this method of delivery makes sense to me!!!
Thanks agnes, I would think this would be particularly powerful in a language classroom. You could narrate while typing out a response, and make and make some sort of grammatical error in your writing and have students find it. I also think the process of hearing you vocalize your thinking (in the target language) while writing might be very helpful for students to begin to grasp more deeply how to write in a foregin language.