The following was e-mailed to me by Elizabeth Smith
who follows a listserv where this discussion was held.
Her message is included below in its entirety (I have added
spacing and formatting, but only to clarify where
one posting ends and another begins)
Date: Tue, 03 Mar 1998 09:08:39 -0400
From: Elizabeth Smith
Organization: North Dakota State Univesity
MIME-Version: 1.0
To: slator@badlands.NoDak.edu
Subject: Researching learning & technology
Brian,
There has been an interesting discussion (in a listserv
I subscribe to)
concerning a recent experiment designed to test whether
technology increases
learning. Thought you might find it interesting.
The
original article
about the experiment is http://www.csun.edu/sociology/virexp.htm
Relevant postings about the study
>From Ed Neal
There must be a "Gresham's Law" of educational research--bad
research drives out good research--since that is the only
explanation for the continued attention being paid to the
Schutte study. I believe can learn very little from this
study because the research design is fundamentally flawed.
He reported that students in his Social Statistics course who
were taught via the WWW performed 20% better on mid-term and
final exams than students taught in the traditional way, *but
he didn't use the same teaching methods for the treatment
group and the control group.*
Schutte's traditional class met for lecture one day a week
and turned in weekly homework problems. Students in his
virtual class had four assignments each week: statistical
reports generated by three-member student groups via e-mail,
a "hypernews discussion" in which every student responded to
a discussion topic twice a week, a "moderated Internet relay
chat" ("in the virtual presence of the professor"), and
the same homework problems assigned in the traditional class.
Clearly, students in the "virtual class" had more
opportunities to be involved with each other and the teacher
and (very significantly) were intensively involved with the
course material over the entire week. It seems clear that
Schutte would have to provide similar small-group activities,
discussion opportunities, and other assignments for the
traditional class if he wants to show that the superior
performance of the "virtual class" was the result of the
technology and not completely different teaching methods.
Schutte simply demonstrated that cooperative learning methods
and intensive engagement with the material yield improved
performance, results that educational researchers discovered
years ago.
Some have pointed out that, without technology, Schutte might
not have experimented with these teaching methods. That may
be true, but it begs the question of whether we need the
technology at all if we can be more effective in the
traditional classroom by adopting these methods.
The Schutte study *does* illustrate the confusion (shared,
unfortunately, by many in higher education) between teaching
*methods* and various delivery systems. This muddy thinking
leads to absurdities such as transferring lectures wholesale
to PowerPoint (or some other presentation software) "to
increase their effectiveness." I often feel as though we
are caught up in a kind of "cargo cult" in higher education--
adopting the trappings of technology in hopes that the magic
will bring us trade goods. Studies that purport to show the
superiority of technology promote this mythology.
Ed Neal
Director of Faculty Development
Center for Teaching and Learning
Campus Box #3470
University of North Carolina at Chapel Hill
*************************************************
>From Dan Rehak:
First, it is vital that we continue to do formal evaluations
of our attempts to introduce technology in education. This
study is one of the few which goes beyond reporting anecdotal
evidence. However, there are serious flaws in what is
inferred from the data (and the underlying experimental
design), and those who cite the study focus on the hype and
not the process or even the caveats in the report.
The result that one group scored 20% higher is meaningless in
inferring anything about the experiment. The two groups are
shown to not differ significantly in their demographics. But
it is possible that the two groups started with some
difference (up to 20%) in their understanding of the material
and that this difference did not change significantly during
the experiment. Meaningful conclusions require a pre-
test/post-test of the subject matter to measure the change
(hopefully an increase) in learning of either group over the
course.
Given a measurable difference, we can then try to correlate
it with differences in the two classes. For this experiment,
if we assume there is a real learning difference, the
question then is what contributed to the difference. The two
classes were different in both the learning experience and in
the use of technology. In terms of pedagogy, course
structure and learning environment (and not technology), one
group had a text, lectures, exams, available faculty office
hours and independent weekly assignments. The other had the
same text, available faculty office hours, weekly
assignments, and exams, but added required discussions
(hypernews), required group problem solving and required
group discussion (chat). One course appears to be a very
traditional passive, individualized learning experience. The
second was a more active learning, group environment with
more time spent on learning activities. Valid pre/post-test
performance differences might be attributed to the different
learning model, and not the technology. Indeed, given the
anecdotal comments about the student's problems with
technology, if the experiment were simply traditional versus
active learning without introducing computer-mediated
collaboration and communications, the resultant differences
might be even greater. Unfortunately, even with measurable
pre/post-test learning differences, we also would need to
know the learning styles and preferences of the two groups to
infer that they did not have an initial bias towards the
style of course they were placed into.
There are still more unknown details which preclude a
thorough analysis of this study. In particular, in this
case, even if we knew that learning approach or technology
was the overall influencing factor, it would be useful to
have information, such as: if the grading were subjective,
was it done as blind to the group; the record of actual
time on task from the chat logs, etc., to measure actual time
spent versus perception, ...
If we want to show that technology or any other change
improves learning (versus saving costs, improving access, or
any of the other potential benefits), we must be careful to
produce studies where we can isolate the factors and
influences which show the differences we want to measure.
Unfortunately, conducting such controlled experiments is very
difficult and done too little.
Dan Rehak
Carnegie Mellon Univ.
-=-==-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-==-=-=-=
>From Steve Ehrmann:
There is a more interesting way to look at Schutte's findings
on learning on the Web
than that suggested by Ed Neal in AAHESGIT #18. First, some
background.
It's well-known in educational research (as Ed mentions) that
technology per se has "no" predictable effect on educational
outcomes. What "do" strongly influence educational outcomes
are the teaching and learning choices made by the players:
students, faculty, materials developers, and others. That's
where technology DOES play a role.
Most teaching and learning behavior makes use of one or more
old or new technologies: chalk, classrooms, computers, the
campus itself, the Web, and so on. A faculty member who
wants to use a particular approach, for example a
presentation, can choose among a number of (technology-
enabled) alternatives: a live lecture in a lecture hall, a
typed presentation, a text to be downloaded from the Web, a
satellite videoconference,... The mode of teaching
(presentation) dictates the outcome: a presentation by any
medium produces the same learning outcomes. But the
technologies affect the costs, the access, the ease of use of
that approach in ways that are in turn affected by context.
In one situation, an instructor may find the lecture hall to
be the best and cheapest choice, while in another situation
a satellite broadcast (or a one-to-one lecture by phone) may
be the best alternative.
Thus, as Bob Kozma said some years back, there are at least
two interesting but entirely distinct questions about
technology and learning:
1) which patterns of teaching and learning behavior (or
choices) tend to foster good learning outcomes? (for example,
the approach to teaching and learning sometimes called self-
paced, branching instruction has been demonstrated to be
exceptionally effective in most cases.)
2) for a given good pattern of teaching and learning and a
given context, which technologies provide the best means?
For example, computers provide a good platform for self-paced
instruction (SPI) for many purposes. Paper is OK for SPI but
awkward to manage if there's a lot of branching. Videotape is
quite awkward as a vehicle because it's so linear that
branching is quite difficult.
Ed Neal infers that Schutte is saying "The Web is the way to
create high test scores." That would certainly be a
fallacious claim. But I think Schutte is saying something
else.
Many studies demonstrate that more student-student
collaboration usually leads to better learning results. It's
enshrined as one of Chickering and Gamson's Seven Principles
of Good Practice in Undergraduate Education
.
Some faculty members have told me that the Web is
fundamentally a poor or impossible environment for supporting
collaborative learning and the superior learning outcomes
that such interaction fosters. Thus, they say, the Web should
not be used, even if students' schedules or locations might
make the Web a more desirable place to learn. They tell me
that on-campus instruction is guaranteed to be more
interactive than Web-based distance learning.
Schutte provides good evidence that, in his case, they're
wrong. The means he devised for teaching on the Web fostered
more collaboration (and the resulting superior performance)
than his customary way of teaching on campus. (Nor was
Schutte's on-campus teaching unusual; there is other evidence
that many on-campus courses, even upper division courses,
feature almost no talk by students, let alone interaction
with one another.) By assigning students randomly Schutte
avoids the criticism that the high test scores were because
Web students were more able or intrinsically more motivated.
Schutte's study is one of the few so far that suggest that
Web-based collaboration can be that good, and with learning
outcomes that good. That does NOT mean that the Web
automatically produces either better outcomes or even that
the Web automatically fosters more collaboration. The
success of the Web-based course is, as Ed points out, due to
how Schutte taught on-line and on-campus.
For people who want to use the Web to extend access for
students with constrained schedules, disabilities or distant
homes (which totals almost all students, these days),
Schutte's study provides evidence that it is "possible" to
foster both healthy collaboration and superior learning on
the Web. It's what the mathematicians call an existence
proof.
*************************************************
>From Paul K. Brandon:
Good points! Have you noticed the more fundamental flaws
(inexcusable for a teacher of statistics ;-) -- the use of
random assignment with a small sample size, and the use of
post-hoc significance level selection. Some of this could
have been mitigated by the use of a pre- post-test design for
assessing student performance. For some reason, they chose
not to do this and relied on self-report survey data.
PAUL K. BRANDON
Psychology Department
Mankato State University
Mankato, MN 56002-8400
507-389-6217
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=
>From Charlotte J. Wharton:
In response to and in support of the comments from Ed Neal of
UNC:
It seems that many are missing the point with technology in
teaching. Technology allows us to do some things in better
ways or more conveniently, such as asynchronous
communication, or immediate, non-threatening feedback from a
computer. Ultimately, good teaching and learning require
energy, involvement, and inspiration. Technology alone will
not improve learning. It is the way we choose to employ the
tool that will make the difference.
I'm not so sure we should be trying to prove that technology
increases learning, but rather that technology enhances
certain kinds of learning or allows us to bring the learning
environment to students in a more convenient way. If used
well, the computer makes the student the focal point rather
than the teacher. Technology promotes a different classroom
dynamic that encourages more student involvement. However,
if all we do with technology is produce glitzy PowerPoint
slides, we are missing the boat!
Charlotte J. Wharton, Director
Interactive Language Resource Center (ILRC)
Miami University
Oxford, OH 45056
(513) 529-2510 (voice)
(513) 529-1807 (fax)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=
>From James H. Stephens:
In reading the comments about Schutte's study, and especially
the distinctions between teaching methods and delivery
systems, I thought about one of the real benefits of
technology-based delivery systems that we sometimes overlook:
the benefits to the TEACHER, with the attendant benefits to
the student.
Technology can be used to streamline our work and make it
more efficient, thereby providing a release from the drudgery
and repetitive tasks that drag so many of us down. If I can
be more efficient--perhaps with automated grading for
objective tests, on-line interactive tutorials with
hyperlinks that address typical questions, or even with
(heaven forbid!) PowerPoint lecture notes that keep me from
producing handouts or writing so much on the board--then I
can devote needed time to individual students. I might
even have a bit more time to think about and produce new
course materials and to try some new ideas.
It's frustrating for teachers to know they could do a better
job if they just had the time to experiment and have some
fun. Unfortunately, most of us have trouble just fitting
everything into a normal schedule. Many of our duties could
be reduced or eliminated through technology--without
compromising pedagogy (I had to use the "p-word" somewhere in
this message).
So as we consider how technology, either as a delivery tool
or as a teaching method, can affect teaching and learning, we
shouldn't ignore the benefits to us--the teachers. I see
technology as a tool that makes me not only more efficient,
but more flexible and more enthusiastic--and it gives me the
freedom to explore methods that make me a better teacher.
James H. Stephens
Southern Polytechnic State University
1100 South Marietta Parkway
Marietta, GA 30060
770 528-7229
email:
home page:
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=
>From Colin D. Laing:
I have no notion whether presentation tools enhance student
absorption of my material - I have only anecdotal evidence
that it does meet some of my objectives.
I do not think at all that the delivery makes the course, the
lecture or the teacher. I think I can enhance my delivery
with presentation tools.
I now teach first and second year Computer courses completely
using PowerPoint; I like it. It seems to please the students
(yes, that is one of my objectives) and they come more often,
stay awake more often, and ask for my course more often than
they did when I used blackboard or overhead foils.
Of course, I may teach differently than I did before, or it
may be with the lights low it is easier to sleep through the
drone from the front of the room. I ask the students whether
they prefer the slide shows, and they nod. The usual lot
speak out, but its the small, unconscious nod from the
rest that I believe. Quite possibly the use of color, bright
moving lights, and occasionally some sound reminds them of
Sesame Street and I get credit thereby.
The other part I like is that I find it much easier to
create, enhance and revise PowerPoint material. I maintain my
lecture notes with the slides, as Author Notes, and so can
create slide point and lecture point together, in one medium.
I can re-order and hide slides to customize the show quickly.
I can use pictures to enhance a point very easily.
And now I use action buttons to bounce from PowerPoint
to the computer application to integrate demonstrations with
the lecture material.
But ...
I hate pushing a computer on a wheeled cart AND a projector
on a cart to the elevator, then through self-closing doors
down corridors to the classroom.
I hate hooking all this junk up before class while the
students try to get my attention. I hate jury-rigging
lighting schemes of my own purchase to try to provide enough
light for writing while presenting a visible image.
I really hate having a bulb go, a disk fail, or a picture
vanish; someday I'm going to teach Math - the guy next door
walks in with one small notebook; his backup is a piece of
chalk.
Still, I like the results as I subjectively view them, and
wouldn't like to be without my toys. They work for me, and
the stuff I teach. In all, this process makes my life
easier, not harder, and leaves me feeling more successful.
Colin D. Laing
Calgary, Alberta, Canada
-=-==-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-==-=-=-=-==-
>From John Driessner:
I believe that Ed Neal also incorporates a fundamental flaw
in his response to Schutte's research, in that he attempts to
view the realities of educational and organizational change
through a pure experimental research paradigm. Unfortunately,
what we are talking about are very complex changes that occur
within highly developed academic cultures embedded in dynamic
institutions.
To dissect out a small delivery mode change without
acknowledging the impact of these ideas and changes on
instructional design, instructional strategies, and student
motivation may be an inviting intellectual process, but miss
the significant potential that these fundamental changes in
how we define higher education have for engaging faculty in
reengineering the academy.
Schutte's research actually looked at the impact of total
instructional redesign associated with teaching a course in a
digitally mediated environment as compared to a "traditional,
unmodified" course. If the treatment was "successful," the
complexity of the treatment and the labels applied to the
elements of that complexity are secondary to the findings.
Let's get beyond research rhetoric and focus on improved
learning.
John Driessner, Ed.D.
Provost
Concordia University
Portland, OR
*******************************************************
-- Forwarded from
Elizabeth Smith
Instructional Designer
North Dakota State University
IACC 250A
email: elsmith@badlands.nodak.edu
voice: 701.231.6283