iTunes University: better than the real thing?

Reviewing some draft course materials about the evaluation of e-learning innovations, I was reminded of a rather nice piece of evaluation research which caused quite a stir when it was published in 2009. The nice thing about it is that it's very simple, yet perfectly illustrates the importance of not taking evaluation data at face value and looking at what's going on beneath the surface.

The experiment was very simple. A university psychology class was divided into two; half went to a regular lecture, and half were given an audio podcast of the same lecture. A week later they took a test on the lecture content - and guess what happened? The students who'd had the podcast scored significantly higher.

Cue headlines in the educational press and blogosphere, along the lines of "iTunesU better than the real thing!", and delight for the proponents of new technology (such as here, here, and here). But there was something puzzling about the result. As any lecturer who has tried producing audio teaching materials knows all too well, the average lecture  makes a pretty poor podcast: stripped of the motivation and engagement factors which come from physical presence at a live lecture, the soundtrack on its own tends to be dull and slow - unless the lecturer is a truly stellar performer, which most are not. So how was it that the students who had the lecture podcast did better in the test?

The answer of course lies in the use which students made of the podcast (which was only reported by some of the commentators, such as here and here). Most of these students took notes, and took really good notes, some of them listening to the podcast several times to make sure they'd understood everything. That they scored highly is not surprising! Those students who DIDN'T take notes from the podcast  - one listened to it while working out at the gym - got rubbish scores. So the real finding turns out to be, not that a podcast is better than a live lecture, but that taking notes is better than not taking notes - and a podcast can be one way of helping students take good notes.

There are two lessons I see I this in story. The first is that evaluation of learning technology needs to get down to the level of student activity, because it's there, in what students do or don't do, that changes need to occur for there to be benefit. The second is that when we introduce new learning technologies, we need also to work on students' understanding of how to use them. There's a cultural norm that you take notes in university lectures, but there's no such common understanding in the case of podcasts. In fact, the cultural norm is rather then other way, a podcast being something you listen to while doing something else, such as working out at the gym. The associations of the technology encourage students to assume that they will learn the information simply by listening to it.

The malignant way in which information technology supports a view of education as the transmission of information is something on which I'm intending to blog in a future post.

References
Dan McKinney, Jennifer L. Dyck and Elise S. Luber, "iTunes University and the classroom: Can podcasts replace Professors?", Computers and Education, 52 (2009), 617-623
Hannah Fearn, "Coming to a screen near you", Times Higher Education Supplement, 11 June 2009, p 38

Seen and heard: June 2012

Rowan Williams (Archbishop of Canterbury) sermon for the Queen's Diamond Jubilee celebrations - on dedication and public service. It was attacked by the Daily Mail, so he must be getting something right.
Julia Fisher, Bach Violin Concertos - with the Academy of St Martin in the Fields. Ordered this CD after hearing the first movement of the double violin concerto played on ClassicFM: great sound, great rhythm!
Inca - band specialising in South American music, after the manner of Incantation, one of whose members is in the new group. Nice to hear them perform live at The Stables, but afterwards it was an Incantation CD that I bought.
The Stone Book Quartet, by Alan Garner. For those who think of Alan Garner's books as being about magic, there's no magic here - except for the magic of landscape and language, of stone and iron and wood, of the craft of the hand and the bond of the family.
The Identity of the History of Science and Medicine, by Andrew Cunningham - a collection of articles by my friend and former colleague, including our jointly-authored and already-anthologised "De-Centring the 'Big Picture': The Origins of Modern Science and the Modern Origins of Science" (History of Science, 26 [1993], 407-432).
Antigone - cracking production by the National Theatre, using a straight-to-the-point modern English translation (everyday language, without being colloquial), and a top-notch performance by Christopher Ecclestone. Bang up to date, with modern army uniforms and an opening scene echoing the assassination of Bin Laden, and its themes of authority and morality, power and defiance, terror and fate; a play about Now, not about then. Truly, as the Chorus says: "Today it has happened here."
Amerzone - 1999 adventure game by Benoît Sokal, which led to his later greater Syberia. Lovely artwork, but this being a first-person game (in the mould of Myst) I missed the presence of a protagonist; I think there's a good reason why more recent narrative games have gone for the third-person approach.

Seen and heard: May 2012

Debussy's Danse Sacrée et Danse Profane, at a concert by the Milton Keynes City Orchestra - also on the programme Britten's Simple Symphony and Vaughan Williams' Dives and Lazarus, and three short pieces by pupils at The Purcell School, following a week-long project with the Orchestra.
The Night Shift, by Sarah Waters
West Wing spoof - starring several members of the original case, in a public service film to promote walking (though talking while you walk, as the West Wing people characteristically do, is strictly optional)
The King and the Playwright - BBC TV series presenting Shakespeare as a Jacobean rather than an Elizabethan author, showing how his later plays reflected the religio-politics and social concerns of the reign of James I / VI.
War and Peace - the Russian version, directed by and starring Sergey Bondarchuk, with the Red Army as extras in the battle scenes.
Bette and Joan, starring Anita Dobson as Joan Crawford, Gretta Scacci as Bette Davies, in a two-hander exploring the friction between the two stars during their filming of Whatever Happened to Baby Jane.
Episodes, series 2 - not as good as the first series (now that there's no obvious sense of direction) but still very funny.
Cloven tongues - Rowan Williams sermon on the theology of biblical translation, given last year during the commemoration of the King James Bible.

Being known, as a condition for learning

One of the few good things about being ill over a bank holiday weekend is that it gives you a chance to catch up with those videos you've been meaning to watch for ages - in my case, a documentary called "We Are the People We've Been Waiting For", distributed with The Guardian a couple of years ago. The film is a call for transformation in the educational system, to give the next generation the creative and human qualities (and not just the employability skills) they will need for the road ahead. The case is well-made and engagingly stated, though I don't think there's anything in it which would be new or surprising to anyone working in the field.

Except perhaps one thing. The film featured an American school called MetWest High where they have a zero per cent drop-out rate and a 96% graduation rate, which Eve Gordon the Principal explained by the three conditions for learning which they aimed to fulfil. When she started her list, the very first item surprised me completely and had me scrabbling for a notepad.

What would be your top three conditions for learning, those circumstances in which people learn best? My guess is that most learning theorists would include the second on Gordon's list:

"People learn best when they are intrinsically motivated, when they are learning about something that they chose, that they are excited about, that they have real questions about."

The third of her conditions for learning might not occur quite so readily to people in higher education:

"People learn best when they do mind and hand learning together."

Manual operations of course tend to be ignored or disparaged in universities, as though the only learning processes of relevance to their curriculum are purely intellectual.  We acknowledge the importance of "active learning", but we tend to conceive this only in terms of mental activity. Perhaps we should be more explicit about the importance of physical action even in academic learning: the physicality of writing actual notes and actual answers, the embodiment of knowledge and understanding in physical artefacts, the physical transfer of resources from one location or context to another. I found myself thinking that we still have much to learn from school education.

But it was Eve Gordon's very first condition for learning which was the surprise for me.

"People learn best when they are known well, in the context of a relationship."

Now the importance of relationship is  I believe seriously under-acknowledged by theorists. My experience is that if you try to talk about it, people think that you're talking about learning communities or social learning: about peer relationships between learners, not what Gordon is talking about here, which is the fundamental relationship between learner and teacher. Ken Robinson described it like this, elsewhere in the film: " The kids who are not doing well suddenly find themselves in a programme with a teacher, somebody who looks into their eyes and sees who they really are, gives them a different way of doing [things], and they come back to life."

Practitioners have long acknowledged the importance of knowing your learners (in the sense of connaître, acquaintance, rather than savoir, cognition) and of giving them a sense of being known. It's why at the Open University we have from our foundation put much more effort and investment into tutorial support than other distance learning providers; when students come to us short on self-confidence or familiarity with high-level study, the sense of there being someone else who believes in them may make all the difference between their being able to marshal the resources to continue and giving up the unequal struggle. In our learning materials, where the relationship with our students is mediated through text, we always address the student in the second person, as "You", adopting the voice of what Derek Rowntree called "a tutorial in print" - and when we get it right, as I have found in my research, students do indeed feel that there is a teacher who knows and understands them, and with whom they develop a relationship of trust.

I wish there was better theoretical understanding of how this works: what's happening in the learner when they have this sense of being known, of there being someone who believes in them, who will hold the promise, the future for them, and how it is that this can affect their learning so profoundly. What is it that the teacher reaches in and touches, that brings about this transformation? If somebody understands this, I wish that they would tell me.

Reference: Derek Rowntree, Preparing Materials for Open, Distance and Flexible Learning: An Action Guide for Teachers and Trainers (Kogan Page, 1994), p 14.

Short writing: when less is more

Some tips on writing very, very short stories have been posted by David Gaffney in a Guardian blog. (Some readers' stories, inspired by these tips, are here.)

Most of these tips are applicable also to writing of learning materials, when we need to write concisely to suit the online medium or simply need a punchier, more motivational style than the academic norm.

• "Start in the middle." Academic writing traditionally starts with the most general and abstract statement of the subject, which may make logical sense but is precisely the wrong thing to do for ease of understanding. Just as back-story is best introduced some way in, when you already care about the characters, rather than in chronological order, so generalities and abstractions make more sense once the particulars and concrete instances are established.
• "Don't use too many characters." The teaching equivalent would be: you can't say everything that you know, so you're going to have to be selective.
• "Make sure the ending isn't at the end." The rationale here is that once your reader reaches the end of the text, you've lost contact with them, so if you want to set them thinking about what your conclusion means and what its implications are, you need to do that before the end of the text.
• "Sweat your title - make it work for a living." In academic writing, a title's main function is to describe and summarise the contents. In other kinds of writing, especially online, a title is often the basis on which a reader decides to read your piece or not. A description or summary is one of the ways to bring a reader in, but a good title can do more: it can pique the interest and set the tone, like the titles of a TV series.
• "Make your last line ring like a bell." It's not only in fiction that we can aspire to writing final lines that "leave the reader with something which will continue to sound after the story has finished."
• "Write long, then go short." Sometimes writing longer than the word limit is the only way to begin, but after that you need to prune, prune, prune. I sometimes advise writers to pretend that they live in the days of telegrams, where you had to pay for every single word, and every word you could cut out of your message was money saved.

Recently an academic colleague thanked me for something I'd said to her three years ago when writing her first online course materials and facing a terrifyingly small word limit. What I'd advised her was to think in terms of writing haiku: a small number of words, getting the reader to do the maximum amount of work - active learning, active reading, at its most extreme.

There's an old example sometimes given to illustrate the nature of story. This:

The king died

is an event. This:

The kind died and the queen died

is two events in temporal sequence - still not a story. This:

The king died and the queen died of grief

is a story. Its power lies not in the words but in what the reader is inspired to bring to the text. If you can get the reader to do the bulk of the work, you don't need a lot of words to tell a story or provoke deep thought.

eBook readers and academic reading

Research on the use of eBooks in higher education suggests that reading devices such as Kindles and iPads aren't going to transform university study in the way some have hoped.

A literature review by Simon Cross in the OU's Institute of Educational Technology covers twelve recent publications and finds a remarkably consistent picture emerging of the advantages and disadvantages of eBooks. The three main advantages reported are those one would expect, and apply whether eBooks are read online with a conventional computer or on a portable reading device: ease of access (by comparison with a physical library), low or zero cost (by comparison with expensive print textbooks), and the searchability of texts. In addition, where students were issued with reading devices for the purposes of the trial, staff found advantage in being able to assume a common hardware and software platform. The only surprise here is that portability, when accessed through a mobile device, did not seem to figure strongly as an advantage.

The disadvantages, however, reveal an important distinction between leisure and academic use. While respondents were very happy with eBook readers for reading novels or magazines, they experienced significant difficulties in using them for the kind of high-level reading required for university work. As I have already observed, academic reading falls into the category which Jakob Nielsen has called "high-value use", for which mobile devices are less well suited than desktop or laptop machines because of their smaller screen size and inferior keyboard interfaces. With the help of the IET literature review,  it's possible to go further and identify some of the distinctive requirements of academic reading.

1  Non-linear navigation.  When reading for academic purposes, you need to do more than simply move forwards through the document. You may need to skip back to earlier parts to remind yourself of what was said previously or skip ahead to see where the text is leading, and then return to where you were. You may need to flick back and forth repeatedly between two or more parts of the text to compare and relate them. You may need to skim read the text to get a sense of its overall structure. All these things are possible on an eBook reader, but they're not simple, because the devices are optimised to show a single page at a time and to navigate linearly forwards and backwards.
2  Note-taking. Academic reading is active reading, to understand and assimilate and interpret, and that usually involves taking notes. Highlighting text and adding annotations, which are supported (if awkwardly) by many eBook readers, are helpful, but only provide part of what is needed. Deep reading involves incorporating the text into your own understanding, which means making your own synthesis: the notes  need to be part of your own text, not the text of the document you're reading. Mobile devices, because of the limitations of their interfaces, tend not to support such synthesis very well; it was said of the iPad on its launch that it was a consumption device rather than a creation device, and (notwithstanding the iPad paintings of David Hockney) that largely remains true. A more fundamental limitation with mobile devices is that they are designed to do only one thing at a time, so that task switching is poor, whereas for note-taking you need to be doing at least two things simultaneously: reading and writing. The difficulties are such that one lecturer concluded that eBook readers were encouraging passive reading in his students.

What seems to be emerging is a pattern of mixed use: combining screen reading for discovery and searching with print reading for note-taking and deep study; using a mobile device for reading and another device for note-taking; or using an eBook reader as a secondary device for when portability is important. eBook readers may have made on-screen reading more comfortable and convenient, through their high resolution imaging and re-sizeable text as well as their physical lightness; but they are unlikely to transform academic use unless and until they can better support these high value functions. Split screen operation would be a start.


Postscript. Since writing the above, I've seen this interesting post by Alex Golub, an American academic and iPad enthusiast, who makes very similar points about the limitations of an iPad, by comparison with a laptop, for academic work.

References The IET literature review is available only to Open University staff through http://kn.open.ac.uk/public/workspace.cfm?wpid=9673, but the reports surveyed include the following:

• Darden School of Business (2010) Darden Shares Results of Kindle Experiment,
  University of Virginia News and Media. 5 November 2010. Available at:
  http://www.darden.virginia.edu/web/Media/Darden-News-Articles/2010/Darden-Shares-
  Results-of-Kindle-Experiment/
• Marmarelli, T. and Ringle, M. (2010) The Reed College Kindle Study. 26 February
  2010. Available at http://www.reed.edu/cis/about/kindle_pilot/Reed_Kindle_report.pdf
• Li, C., Poe, F., Potter, M, Quigley, B. and Wilson, J. (2011) UC Libraries Academic e-
  Book Usage Survey: Springer e-Book Pilot Project, University of California Libraries.
• Foasberg, N. (2011) Adoption of E-Book Readers among College Students: A
  Survey, Information Technology and Libraries, September 2011, 108-128

Data processing vs. getting things done

I think the reason I like Jakob Nielsen's online posts so much, and always seem to get something from them, is the way he alternates concrete observational detail with top-level principles, so that you learn to see in a different way: you learn to look at websites and software interfaces as he does, through the lens of usability design.

His latest post, though starting off with detailed discussion of the winning applications in some recent awards for user interface design, rapidly moves into some refreshing ideas on what we should be striving for in systems design.

"Software's real goal should not be to simply process transactions in a system where users are nothing more than data operators who click required buttons to make things happen. Rather, software should work to augment human capabilities, helping us to overcome weaknesses and emphasize our strengths. "

It sounds obvious. Yet the temptation for software designers, who tend to be more interested in data processing that the intended users, is to design the kind of interfaces they would like themselves. As Alan Cooper observed (2004, pp 93-94), there are two kinds of people: those who on entering an aeroplane would turn left into the cockpit if they could and those who'd prefer to turn right into the cabin; in other words, those who are prepared to tolerate complexity if it gives them control and those who are prepared to tolerate absence of control if it gives them simplicity. Most IT people, including most interface designers, are of the first kind; most users are of the second. As Nielsen says:

"We want to empower users to be creative and accomplish advanced things with our software. But we should also recognize that users sometimes just want to get their tasks done without having to explore numerous options and new ideas. ...To speed users through infrequent or complicated tasks, it's often good to present a linear workflow with minimal disruptions or alternatives. Yes, the lack of flexibility can feel constraining, but it can be faster to just power through all the steps instead of having to ponder which steps are needed. Also, the cost of too much freedom is that users have to decide the order of the steps — something that they're often happy enough to delegate to the computer. "

The temptation for software designers to complexify the interface is compounded by the temptation for teachers to complexify the learning options for our learners. There are so many things we want them to do, or to be able to do, or which we think they might want to do, and we're so worried about whether we've provided enough that we pile resource upon resource upon resource, and we're so worried about whether they'll find them that we put them all at the top level of the interface, until the learner is faced with something as complicated and intimidating as an aeroplane cockpit.

What we need to remember is that most learners, most of the time, are not interested in the technical system or even (though we hate to admit it!) in the learning process. They just want to get on and do the next thing. The trick for us is to create an interface which will let them do that with minimum fuss, and yet provide them with the power to go deeper and do more when the occasion arises.

Reference: Alan Cooper (2004), The Inmates are Running the Asylum: Why High-Tech Products Drive us Crazy and How to Restore the Sanity, 2nd edn (Sams Publishing)