Listserv Assessment Discussion

The following was e-mailed to me by Elizabeth Smith who follows a listserv where this discussion was held. Her message is included below in its entirety (I have added spacing and formatting, but only to clarify where one posting ends and another begins)

Date: Tue, 03 Mar 1998 09:08:39 -0400
From: Elizabeth Smith 
Organization: North Dakota State Univesity
MIME-Version: 1.0
To: slator@badlands.NoDak.edu
Subject: Researching learning & technology

Brian,

There has been an interesting discussion (in a listserv I subscribe to) concerning a recent experiment designed to test whether technology increases learning. Thought you might find it interesting.

The original article about the experiment is http://www.csun.edu/sociology/virexp.htm

Relevant postings about the study

>From Ed Neal

There must be a "Gresham's Law" of educational research--bad research drives out good research--since that is the only explanation for the continued attention being paid to the Schutte study. I believe can learn very little from this study because the research design is fundamentally flawed.

He reported that students in his Social Statistics course who were taught via the WWW performed 20% better on mid-term and final exams than students taught in the traditional way, *but he didn't use the same teaching methods for the treatment group and the control group.*

Schutte's traditional class met for lecture one day a week and turned in weekly homework problems. Students in his virtual class had four assignments each week: statistical reports generated by three-member student groups via e-mail, a "hypernews discussion" in which every student responded to a discussion topic twice a week, a "moderated Internet relay chat" ("in the virtual presence of the professor"), and the same homework problems assigned in the traditional class. Clearly, students in the "virtual class" had more opportunities to be involved with each other and the teacher and (very significantly) were intensively involved with the course material over the entire week. It seems clear that Schutte would have to provide similar small-group activities, discussion opportunities, and other assignments for the traditional class if he wants to show that the superior performance of the "virtual class" was the result of the technology and not completely different teaching methods. Schutte simply demonstrated that cooperative learning methods and intensive engagement with the material yield improved performance, results that educational researchers discovered years ago.

Some have pointed out that, without technology, Schutte might not have experimented with these teaching methods. That may be true, but it begs the question of whether we need the technology at all if we can be more effective in the traditional classroom by adopting these methods.

The Schutte study *does* illustrate the confusion (shared, unfortunately, by many in higher education) between teaching *methods* and various delivery systems. This muddy thinking leads to absurdities such as transferring lectures wholesale to PowerPoint (or some other presentation software) "to increase their effectiveness." I often feel as though we are caught up in a kind of "cargo cult" in higher education-- adopting the trappings of technology in hopes that the magic will bring us trade goods. Studies that purport to show the superiority of technology promote this mythology.

Ed Neal

Director of Faculty Development
Center for Teaching and Learning
Campus Box #3470
University of North Carolina at Chapel Hill

*************************************************

>From Dan Rehak:

First, it is vital that we continue to do formal evaluations of our attempts to introduce technology in education. This study is one of the few which goes beyond reporting anecdotal evidence. However, there are serious flaws in what is inferred from the data (and the underlying experimental design), and those who cite the study focus on the hype and not the process or even the caveats in the report.

The result that one group scored 20% higher is meaningless in inferring anything about the experiment. The two groups are shown to not differ significantly in their demographics. But it is possible that the two groups started with some difference (up to 20%) in their understanding of the material and that this difference did not change significantly during the experiment. Meaningful conclusions require a pre- test/post-test of the subject matter to measure the change (hopefully an increase) in learning of either group over the course.

Given a measurable difference, we can then try to correlate it with differences in the two classes. For this experiment, if we assume there is a real learning difference, the question then is what contributed to the difference. The two classes were different in both the learning experience and in the use of technology. In terms of pedagogy, course structure and learning environment (and not technology), one group had a text, lectures, exams, available faculty office hours and independent weekly assignments. The other had the same text, available faculty office hours, weekly assignments, and exams, but added required discussions (hypernews), required group problem solving and required group discussion (chat). One course appears to be a very traditional passive, individualized learning experience. The second was a more active learning, group environment with more time spent on learning activities. Valid pre/post-test performance differences might be attributed to the different learning model, and not the technology. Indeed, given the anecdotal comments about the student's problems with technology, if the experiment were simply traditional versus active learning without introducing computer-mediated collaboration and communications, the resultant differences might be even greater. Unfortunately, even with measurable pre/post-test learning differences, we also would need to know the learning styles and preferences of the two groups to infer that they did not have an initial bias towards the style of course they were placed into.

There are still more unknown details which preclude a thorough analysis of this study. In particular, in this case, even if we knew that learning approach or technology was the overall influencing factor, it would be useful to have information, such as: if the grading were subjective, was it done as blind to the group; the record of actual time on task from the chat logs, etc., to measure actual time spent versus perception, ...

If we want to show that technology or any other change improves learning (versus saving costs, improving access, or any of the other potential benefits), we must be careful to produce studies where we can isolate the factors and influences which show the differences we want to measure. Unfortunately, conducting such controlled experiments is very difficult and done too little.

Dan Rehak
Carnegie Mellon Univ.

-=-==-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-==-=-=-=

>From Steve Ehrmann:

There is a more interesting way to look at Schutte's findings on learning on the Web than that suggested by Ed Neal in AAHESGIT #18. First, some background.

It's well-known in educational research (as Ed mentions) that technology per se has "no" predictable effect on educational outcomes. What "do" strongly influence educational outcomes are the teaching and learning choices made by the players: students, faculty, materials developers, and others. That's where technology DOES play a role.

Most teaching and learning behavior makes use of one or more old or new technologies: chalk, classrooms, computers, the campus itself, the Web, and so on. A faculty member who wants to use a particular approach, for example a presentation, can choose among a number of (technology- enabled) alternatives: a live lecture in a lecture hall, a typed presentation, a text to be downloaded from the Web, a satellite videoconference,... The mode of teaching (presentation) dictates the outcome: a presentation by any medium produces the same learning outcomes. But the technologies affect the costs, the access, the ease of use of that approach in ways that are in turn affected by context. In one situation, an instructor may find the lecture hall to be the best and cheapest choice, while in another situation a satellite broadcast (or a one-to-one lecture by phone) may be the best alternative.

Thus, as Bob Kozma said some years back, there are at least two interesting but entirely distinct questions about technology and learning:

1) which patterns of teaching and learning behavior (or choices) tend to foster good learning outcomes? (for example, the approach to teaching and learning sometimes called self- paced, branching instruction has been demonstrated to be exceptionally effective in most cases.)

2) for a given good pattern of teaching and learning and a given context, which technologies provide the best means? For example, computers provide a good platform for self-paced instruction (SPI) for many purposes. Paper is OK for SPI but awkward to manage if there's a lot of branching. Videotape is quite awkward as a vehicle because it's so linear that branching is quite difficult.

Ed Neal infers that Schutte is saying "The Web is the way to create high test scores." That would certainly be a fallacious claim. But I think Schutte is saying something else.

Many studies demonstrate that more student-student collaboration usually leads to better learning results. It's enshrined as one of Chickering and Gamson's Seven Principles of Good Practice in Undergraduate Education .

Some faculty members have told me that the Web is fundamentally a poor or impossible environment for supporting collaborative learning and the superior learning outcomes that such interaction fosters. Thus, they say, the Web should not be used, even if students' schedules or locations might make the Web a more desirable place to learn. They tell me that on-campus instruction is guaranteed to be more interactive than Web-based distance learning.

Schutte provides good evidence that, in his case, they're wrong. The means he devised for teaching on the Web fostered more collaboration (and the resulting superior performance) than his customary way of teaching on campus. (Nor was Schutte's on-campus teaching unusual; there is other evidence that many on-campus courses, even upper division courses, feature almost no talk by students, let alone interaction with one another.) By assigning students randomly Schutte avoids the criticism that the high test scores were because Web students were more able or intrinsically more motivated.

Schutte's study is one of the few so far that suggest that Web-based collaboration can be that good, and with learning outcomes that good. That does NOT mean that the Web automatically produces either better outcomes or even that the Web automatically fosters more collaboration. The success of the Web-based course is, as Ed points out, due to how Schutte taught on-line and on-campus.

For people who want to use the Web to extend access for students with constrained schedules, disabilities or distant homes (which totals almost all students, these days), Schutte's study provides evidence that it is "possible" to foster both healthy collaboration and superior learning on the Web. It's what the mathematicians call an existence proof.

*************************************************

>From Paul K. Brandon:

Good points! Have you noticed the more fundamental flaws (inexcusable for a teacher of statistics ;-) -- the use of random assignment with a small sample size, and the use of post-hoc significance level selection. Some of this could have been mitigated by the use of a pre- post-test design for assessing student performance. For some reason, they chose not to do this and relied on self-report survey data.

PAUL K. BRANDON
Psychology Department
Mankato State University
Mankato, MN 56002-8400
507-389-6217


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=

>From Charlotte J. Wharton:

In response to and in support of the comments from Ed Neal of UNC:

It seems that many are missing the point with technology in teaching. Technology allows us to do some things in better ways or more conveniently, such as asynchronous communication, or immediate, non-threatening feedback from a computer. Ultimately, good teaching and learning require energy, involvement, and inspiration. Technology alone will not improve learning. It is the way we choose to employ the tool that will make the difference.

I'm not so sure we should be trying to prove that technology increases learning, but rather that technology enhances certain kinds of learning or allows us to bring the learning environment to students in a more convenient way. If used well, the computer makes the student the focal point rather than the teacher. Technology promotes a different classroom dynamic that encourages more student involvement. However, if all we do with technology is produce glitzy PowerPoint slides, we are missing the boat!

Charlotte J. Wharton, Director
Interactive Language Resource Center (ILRC)
Miami University
Oxford, OH 45056

(513) 529-2510 (voice)
(513) 529-1807 (fax)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=

>From James H. Stephens:

In reading the comments about Schutte's study, and especially the distinctions between teaching methods and delivery systems, I thought about one of the real benefits of technology-based delivery systems that we sometimes overlook: the benefits to the TEACHER, with the attendant benefits to the student.

Technology can be used to streamline our work and make it more efficient, thereby providing a release from the drudgery and repetitive tasks that drag so many of us down. If I can be more efficient--perhaps with automated grading for objective tests, on-line interactive tutorials with hyperlinks that address typical questions, or even with (heaven forbid!) PowerPoint lecture notes that keep me from producing handouts or writing so much on the board--then I can devote needed time to individual students. I might even have a bit more time to think about and produce new course materials and to try some new ideas.

It's frustrating for teachers to know they could do a better job if they just had the time to experiment and have some fun. Unfortunately, most of us have trouble just fitting everything into a normal schedule. Many of our duties could be reduced or eliminated through technology--without compromising pedagogy (I had to use the "p-word" somewhere in this message).

So as we consider how technology, either as a delivery tool or as a teaching method, can affect teaching and learning, we shouldn't ignore the benefits to us--the teachers. I see technology as a tool that makes me not only more efficient, but more flexible and more enthusiastic--and it gives me the freedom to explore methods that make me a better teacher.

James H. Stephens
Southern Polytechnic State University
1100 South Marietta Parkway
Marietta, GA 30060
770 528-7229
email:

home page:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=

>From Colin D. Laing:

I have no notion whether presentation tools enhance student absorption of my material - I have only anecdotal evidence that it does meet some of my objectives.

I do not think at all that the delivery makes the course, the lecture or the teacher. I think I can enhance my delivery with presentation tools.

I now teach first and second year Computer courses completely using PowerPoint; I like it. It seems to please the students (yes, that is one of my objectives) and they come more often, stay awake more often, and ask for my course more often than they did when I used blackboard or overhead foils.

Of course, I may teach differently than I did before, or it may be with the lights low it is easier to sleep through the drone from the front of the room. I ask the students whether they prefer the slide shows, and they nod. The usual lot speak out, but its the small, unconscious nod from the rest that I believe. Quite possibly the use of color, bright moving lights, and occasionally some sound reminds them of Sesame Street and I get credit thereby.

The other part I like is that I find it much easier to create, enhance and revise PowerPoint material. I maintain my lecture notes with the slides, as Author Notes, and so can create slide point and lecture point together, in one medium. I can re-order and hide slides to customize the show quickly. I can use pictures to enhance a point very easily.

And now I use action buttons to bounce from PowerPoint to the computer application to integrate demonstrations with the lecture material.

But ...

I hate pushing a computer on a wheeled cart AND a projector on a cart to the elevator, then through self-closing doors down corridors to the classroom.

I hate hooking all this junk up before class while the students try to get my attention. I hate jury-rigging lighting schemes of my own purchase to try to provide enough light for writing while presenting a visible image.

I really hate having a bulb go, a disk fail, or a picture vanish; someday I'm going to teach Math - the guy next door walks in with one small notebook; his backup is a piece of chalk.

Still, I like the results as I subjectively view them, and wouldn't like to be without my toys. They work for me, and the stuff I teach. In all, this process makes my life easier, not harder, and leaves me feeling more successful.

Colin D. Laing
Calgary, Alberta, Canada

-=-==-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-==-=-=-=-==-

>From John Driessner:

I believe that Ed Neal also incorporates a fundamental flaw in his response to Schutte's research, in that he attempts to view the realities of educational and organizational change through a pure experimental research paradigm. Unfortunately, what we are talking about are very complex changes that occur within highly developed academic cultures embedded in dynamic institutions.

To dissect out a small delivery mode change without acknowledging the impact of these ideas and changes on instructional design, instructional strategies, and student motivation may be an inviting intellectual process, but miss the significant potential that these fundamental changes in how we define higher education have for engaging faculty in reengineering the academy.

Schutte's research actually looked at the impact of total instructional redesign associated with teaching a course in a digitally mediated environment as compared to a "traditional, unmodified" course. If the treatment was "successful," the complexity of the treatment and the labels applied to the elements of that complexity are secondary to the findings.

Let's get beyond research rhetoric and focus on improved learning.

John Driessner, Ed.D.
Provost
Concordia University
Portland, OR

*******************************************************

-- Forwarded from
Elizabeth Smith
Instructional Designer
North Dakota State University
IACC 250A
email: elsmith@badlands.nodak.edu 
voice: 701.231.6283