I’ve been reading Means, Toyama, Murphy, Bakia & Jones, 2010. Evaluation of Evidence-based practices in online learning: A Meta-analysis and review of online learning studies. US Department of Education – http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf

“The meta-analysis found that, on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes—measured as the difference between treatment and control means, divided by the pooled standard deviation—was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se.” (from the abstract)

The strongest finding seems to be that all activities should encourage students to reflection and to self-evaluation. I believe we do this well when we give timely formative feedback that routes students into discussion, if we monitor and stimulate peer support in discussions.

INSTRUCTOR DOMINANCE

“Distance learning outcomes were less positive when instructor involvement was low (as in “canned” applications)” (p53)

VIDEO

• – Does NOT itself lead to superior learning compared to other media
• – Should allow learners to control their viewing
• – Should require learners’ reflection/analysis

QUIZ

“Simple multiple choice quizzes did not appear to enhance online learning” (p48)

• – Does NOT itself lead to superior learning compared to homework assignments
• – Should trigger learner activity or reflection and self-monitoring of understanding

In more detail –

Key statistical findings

(quotes from the Executive summary)

• Few rigorous research studies of the effectiveness of online learning for K–12 students have been published
• Students in online conditions performed modestly  better, on average, than those learning the same material through traditional face-to-face instruction.
• Online learning is better than purely face-to-face instruction – average effect size of +0.20, p < .001 favoring online conditions
• Blended learning is better than face-to-face instruction – mean effect size was +0.35, p < .001 favoring blended conditions
• The observed advantage for blended learning conditions is not necessarily rooted in the media used per se and may reflect differences in content, pedagogy and learning time.
• Good instruction is better than independent learning. The mean effect sizes for collaborative instruction (+0.25) and for instructor-directed instruction (+0.39) were significantly positive whereas the mean effect size for independent learning (+0.05) was not.
• The use of a blended rather than a purely online approach and instructor-directed or collaborative rather than independent, self-directed instruction) were the only statistically significant influences on effectiveness of online learning, out of 13 variables.
• Online learning appeared to be an effective option for both undergraduates (mean effect of +0.30, p < .001) and for graduate students and professionals (+0.10, p < .05) in a wide range of academic and professional studies.

Key findings of narrative reviews

(quotes from the Executive summary)

• When a study contrasts blended and purely online conditions, student learning is usually comparable across the two conditions
• Elements such as video or online quizzes do not appear to influence the amount that students learn in online classes.
• Online learning can be enhanced by giving learners control of their interactions with media and prompting learner reflection. Studies indicate that manipulations that trigger learner activity or learner reflection and self-monitoring of understanding are effective when students pursue online learning as individuals.
• When groups of students are learning together online, support mechanisms such as guiding questions generally influence the way students interact, but not the amount they learn

Caveats

(quotes from the Executive summary)

In many of the studies showing an advantage for blended learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages. At the same time, one should note that online learning is much more conducive to the expansion of learning time than is face-to-face instruction.

Quizzes

(paraphrased from the section on Online Quizzes p.43 and Individualised instruction p.44 and Supports for Learner Reflection p.44-45)

Four quantitative studies on quizzes found little or no advantage to inclusion of quizzes. One study found that Quizzes are less effective than online discussion. One found that the software interface does affect test performance. However, these studies used quizzes in a manner similar to conventional homework assignments, not as interactive instruction.

Individualised instruction is justified.

“There were only two online learning studies of the effects of individualizing instruction, but both found a positive effect.”

Other studies showed positive effects from computer prompts, formative online assessment and feedback, in the context of encouraging and guiding self-reflection.