iTunes University: better than the real thing?

Reviewing some draft course materials about the evaluation of e-learning innovations, I was reminded of a rather nice piece of evaluation research which caused quite a stir when it was published in 2009. The nice thing about it is that it's very simple, yet perfectly illustrates the importance of not taking evaluation data at face value and looking at what's going on beneath the surface.

The experiment was very simple. A university psychology class was divided into two; half went to a regular lecture, and half were given an audio podcast of the same lecture. A week later they took a test on the lecture content - and guess what happened? The students who'd had the podcast scored significantly higher.

Cue headlines in the educational press and blogosphere, along the lines of "iTunesU better than the real thing!", and delight for the proponents of new technology (such as here, here, and here). But there was something puzzling about the result. As any lecturer who has tried producing audio teaching materials knows all too well, the average lecture  makes a pretty poor podcast: stripped of the motivation and engagement factors which come from physical presence at a live lecture, the soundtrack on its own tends to be dull and slow - unless the lecturer is a truly stellar performer, which most are not. So how was it that the students who had the lecture podcast did better in the test?

The answer of course lies in the use which students made of the podcast (which was only reported by some of the commentators, such as here and here). Most of these students took notes, and took really good notes, some of them listening to the podcast several times to make sure they'd understood everything. That they scored highly is not surprising! Those students who DIDN'T take notes from the podcast  - one listened to it while working out at the gym - got rubbish scores. So the real finding turns out to be, not that a podcast is better than a live lecture, but that taking notes is better than not taking notes - and a podcast can be one way of helping students take good notes.

There are two lessons I see I this in story. The first is that evaluation of learning technology needs to get down to the level of student activity, because it's there, in what students do or don't do, that changes need to occur for there to be benefit. The second is that when we introduce new learning technologies, we need also to work on students' understanding of how to use them. There's a cultural norm that you take notes in university lectures, but there's no such common understanding in the case of podcasts. In fact, the cultural norm is rather then other way, a podcast being something you listen to while doing something else, such as working out at the gym. The associations of the technology encourage students to assume that they will learn the information simply by listening to it.

The malignant way in which information technology supports a view of education as the transmission of information is something on which I'm intending to blog in a future post.

References
Dan McKinney, Jennifer L. Dyck and Elise S. Luber, "iTunes University and the classroom: Can podcasts replace Professors?", Computers and Education, 52 (2009), 617-623
Hannah Fearn, "Coming to a screen near you", Times Higher Education Supplement, 11 June 2009, p 38