August 29, 2008
TECHNOS QUARTERLY Summer 1995 Vol. 4 No. 2
The Strange but True Story of Multimedia and the Type I Error
By Richard P. Lookatch
Is multimedia really all it's cracked up to be? Not according to this 20-year veteran educational psychologist. He says the new kids on the block—the techno wizards graduating from our universities with Ph.D.s in instructional systems—are selling educators a bill of goods. And it's all because of a basic error in research, one they should have learned about a long time ago.
There is no evidence that computers and multimedia improve learning. Once more? Research to date on the impact of technologies on learning has never established that using a computer or any other technology improves learning. I have closely monitored the literature on teaching technologies since the bloom of computers in the classroom in the early 1980s and have yet to see a study that was without a fundamental flaw.
It's called the “Type I Error,” and it means the researcher has found benefits that aren't really there. Strange but true.
This is Research?
Multimedia has become such a science that universities are now granting doctorates
in what is essentially multimedia design and programming. Typically these
degrees are called something along the lines of “Instructional Systems
Design.” Unfortunately, their graduates appear so narrowly educated that
they know little or nothing of instructional design outside of that which
is packaged in slick authoring systems. These same universities spend thousands
of research hours on background colors, text style, acceptable video compression
rates, etc.—a significantly greater “dosage” than that provided
for research on instructional strategies exclusive of media.
Experimental design is a critical component of scholarly research. It also appears to have taken a back seat to the latest fashions in media. When I hear research paper presentations, I am often puzzled by the failure of the new Ph.D.s to use such basics as: true control groups; pre-tests that enable the researcher to equalize individual differences among subjects when crunching the numbers; and failure to control for a whole myriad of confounding variables. Whatever happened to true experimental design?
Case in point. Throughout the first part of this century, some misled psychometricians revealed that, as indicated by a variety of instruments, members of lower socioeconomic status and members of certain racial groups were of inferior intelligence. The data was there; unfortunately, the design wasn't. Dangerous? Yes. These studies proved nothing and could have been avoided had they been properly designed. At best, they damaged the perception of those identified as “inferior”; at worst, they fostered racism and class envy. They were fundamentally flawed in that they failed to control for a host of environmental experiences and conditions that contribute to test performance of any kind, extraneous factors that could account for the observed differences. What does this have to do with the more recent glowing multimedia research conclusions?
The flaw in the research on multimedia's impact on learning—for it is flawed—is much like that in the “IQ” research of the 1950s. Just as those misled souls failed to control for environmental differences (perhaps an impossible task), many multimedia researchers to date have also failed to control for a host of conditions that may account for the observed impact on learning (other than the fact that a computer was used to mediate the instruction). And this flaw leads to their Type I Error.
A typical research design on the impact of multimedia pits a videotape version of an instructional lesson and a non-media classroom version against an interactive multimedia version. Comically, the traditional classroom version is often referred to as a “control group.” In reality, there typically is no control group—certainly not one that would help assure that differences are attributable to the treatment rather than to some extraneous variable. There is no pre-test to account for individual and resulting group differences at the onset, differences that may influence the statistics. Rather, the assumption is made that because there are 20 or so students in each group, any differences would “average out.”
A Matter of Dosage
But there is an even bigger problem. The typical study described in the multimedia
research parallels a pharmacological study in which a new drug is tested,
with the exception that the multimedia research study often excludes a
true control group while the pharmacology industry has the sense to include
one. A drug is tested at three significantly different dosages, though the
vehicles of delivery are identical (for example, the same-looking pill). Surprise:
The group receiving the highest dosage experiences the greatest effects! Unfortunately,
our multimedia researchers lack the insight to see that their dosages (of
content and instructional strategies) vary significantly from one experimental
group to another—in other words, from multimedia group to classroom group.
The pills even look different. Typically, the multimedia group receives either
more content or content for which greater care was taken in its preparation.
And with average development costs of CD-ROMs in the $700,000 range, rest
assured that the latter difference is common. The difference in preparation
of the teacher who is to be compared with the preparation invested in some
$700,000 CD-ROM, is glaring.
Poor media research design predates multimedia. For example, it was once commonplace to measure the effectiveness of televised instruction in a manner similar to what is now done with multimedia. Typically, the best teacher was selected to provide the televised instruction and given a full day to prepare the surrounding lesson. The progress of the televised instruction group was then compared with the progress of students in conventional settings with average teachers who taught the usual load of classes and had the usual (minimal) time to prepare. The results and subsequent conclusions of studies that hailed televised instruction as education's savior were, of course, flawed. The real finding was that better teachers with more preparation time and novel teaching tools resulted in greater student achievement. Had a group of better teachers substituted dramatic roleplays or field experiences for the televised instruction, their students' performance might have equaled or surpassed that of the televised instruction group. Experimental design flaws like this thrive in today's multimedia research and lead to the Type I Error.
To add to the confusion, the contexts in which most media studies are carried out are so complex that researchers cannot possibly control, and are likely unaware of, all the variables in the environment that might affect the results.
The Myth of Multimedia Benefits
There are no unique educational benefits from multimedia or its attributes.
Once the information content and instructional strategies are controlled for,
or equalized, the differences will disappear. Such controls are absent from
multimedia studies revealing benefits of the medium. The motivational effects
and preferences of multimedia often cited in studies are a charade as well.
What occurs here is blind belief in a new and novel machine, a machine that
students find more interesting and easier to accept than a talking head in
front of the class. It is not, however, the machine that motivates. Rather,
it is curiosity, content, and instructional strategies that motivate the learner.
Case in point. In my 20 years in the teaching business, having produced some 75 interactive laserdisc and multimedia programs, I've found the most motivating medium I ever used was a print-based lesson. That's right, print! It was called the “Wayne Powers Action Maze,” and it was published by Employers Mutual Liability Insurance Company in 1975 and sold for a mere $1.25. This inexpensive handbook was a highly interactive instructional tool for teaching managers and supervisors how and how not to handle employees with behavioral/medical problems such as alcoholism. When part of a dynamic lesson plan, the “Wayne Powers Action Maze” provided as much enthusiasm and student achievement as any multimedia version could have. It was an interactive tool that included a captivating case study with complex consequential branching the likes of which I have never seen again outside of multimedia. And it was a very effective training tool.
The Economics of Multimedia
There is no mediated program that cannot be mediated by an alternate medium.
Unfortunately, the battle between fashion and efficacy seems to have been
won by fashion. Today it is fashionable to “do a CD-ROM” with little
regard for other less expensive methods of delivery (such as print). What
is puzzling is that the question “Can it be done cheaper using other
forms of mediation?” seems to be routinely overlooked by developers.
Fashion seems to have won out even over economics.
Clearly, there are applications for which multimedia has the most efficacy. But even in these cases, the advantages are economic. For example, it may be a lot less expensive to use a flight simulator than hands-on flight training for pilots. But the advantages are economic—time, cost, liability—not learning-oriented. Conversely, I seriously question the use of multimedia to supplant hands-on classroom science experiments or simple dissections in the biology lab. A computer simulation is clearly neater, cleaner, and easier, but it is not an experience. The true experience occurs on the lab table.
Dangers of the Type I Error
The Type I Error—finding benefits that are not really there—apparent
in much of the multimedia research would seem to present education with a
panacea: a path (albeit an expensive one) to equity, efficacy, and high SAT
scores. The reality, in stark contrast, is that the Type I Error will lead
to inequity, either lower scores or lower standards, and wasted financial
resources.
Clearly, multimedia is not the great equalizer. The same groups indicted by the IQ research of the first half of this century will be impacted similarly by the multimedia panacea. For example, access to technology is clearly inequitable and will continue to be so for the foreseeable future. Poor urban and rural school systems do not now and may never approach the access to technology resources afforded well-financed systems. And even if this inequity were somehow magically remedied, performance in a multimedia environment will clearly favor those with off-campus access and preschool exposure, common in today's middle and upper-socioeconomic status households, rare in others.
A second consequence of the Type I Error is the resulting misdirection of resources toward applications that may be better served by cheaper, less fashionable instructional strategies such as field experience, role playing, and print-based works. A truckload of quality print-based interactive instruction can be produced for the $700,000 cost of producing a single CD-ROM program. And its impact will be equal to or more than that of the CD-ROM, if teachers have the time to prepare to use it and the motivation to deliver it. Furthermore, misappropriating resources to multimedia projects better or equally served by other instructional strategies leaves fewer funds for those projects best served by multimedia.
Yet another outgrowth of the Type I Error is the blind thirst for multimedia at the expense of instructional design. A recent instructional designer search I conducted at the Association for Educational Communications Technology national convention yielded a large crop of vitae. Each and every one of them highlighted multimedia experience and quasi-experience and little, if any, instructional design accomplishments. The ultimate consequence of hiring these multimedia whiz kids will be poorly designed yet flashy, fashionable, and entertaining instructional materials—great baby-sitters but poor teachers. And what will become of the teachers? Will we give up on them and employ technology specialists instead? With that, will we also give up on reality in favor of simulation? Why undertake a field activity, when we can simulate it with a CD-ROM?
A Matter of Semantics
Is this treatise merely a matter of semantics? Some may view it as such. However,
careful reflection can only lead to the conclusion that multimedia and a host
of other technologies such as the Internet; the WEB, a hyperlink system within
the Internet; and virtual reality are fast becoming the new educational panacea.
But we must avoid the Type I Error. We cannot lose sight of the fact that
it is instructional strategies that cause improvements in achievement, not
media. The richness of life's contexts, its faces. fragrances, and emotions,
will never be effective candidates for computer simulations. Neither will
the training of a surgeon nor the handiwork of a carpenter.
Some instructional strategies are well suited for multimedia and other technologies. Simulated outcomes of decisions impractical to experience live, literature review, and data collection are a few examples of technology applications that are consistent with sound instructional strategies. Though any of these could be accomplished using other media, multimedia and on-line technologies are clearly more efficient and economical with little if any experiential cost. However, for each well-guided application, there is a host of misapplications—misapplications with measurable costs in achievement and dollars. More important, they are misapplications that deprive students of those human interactions and hands-on experiences that yield feeling, thinking adults.
Media clearly enable us to design better instruction. For the past 30-plus years, film and video have brought the world into the classroom. Unfortunately, we relied on film and video to bring the community into the classroom rather than the class into the community. Multimedia provides the opportunity to interact with the images behind a glass screen. Let's hope we don't let it replace interaction with each other and the world around us. Multimedia and other technologies are simply tools that assist with instruction. They have no more influence on achievement and wholeness than a scalpel has on healing.
“Costly Toys” computer illustration by Frank Morris.
Click here to access Multimedia-Based Video Options/Questions to Ask Sidebar that accompanied this article.
Richard Lookatch is an educational psychologist with the Agency for Instructional Technology's instructional design unit in Bloomington, Indiana. He received his Ph.D. from the University of Wisconsin and has served the training and education field for 20 years, specializing in educational technologies.