Several days ago, the U.S. Department of Education published to Congress, the results of a recent study on the effects of computer software on reading and math test scores. The report said…
Test scores were not significantly higher in classrooms using the reading and mathematics software products than those in control classrooms. In each of the four groups of products-reading in first grade and in fourth grade, mathematics in sixth grade, and high school algebra-the evaluation found no significant differences in student achievement between the classrooms that used the technology products and classrooms that did not.
Many respected educators have already commented about this study, suggesting flaws in its design (read eSchoolNews for a fairly in depth report). The designers of the study maintain that it, “..was a very well-done study, there are no flaws in it..”
I want to broaden the conversation a bit and post a survey question about scientific research in general. Think back to those teachers who truly influenced your life, who’s teaching is actually a part of who you are and what you do now. Think about what happened in those classrooms that influenced you so deeply and share with us the percentage of it that might have been measured with research, and what percentage could not be effectively measured.
100% measurable – 0% not measurable
75% measurable – 25% not measurable
50% measurable – 50% not measurable
25% measurable – 75% not measurable
0% measurable – 100% not measurable
Now these surveys are not meant to be scientific. They are, instead, meant to generate some conversation about teaching and learning in this time of rapid change. So please do comment.
What do you think?
Image Citation:
Wyn. “Rock Star Scientist.” W_yvr’s Photostream. 29 Mar 2006. 9 Apr 2007 <http://flickr.com/photos/w_yvr/120043078/>.
Hi Dave
To me the question centers on the learning, not what the teacher ‘did’ to get me to learn. For example, a high social studies teacher of mine would describe historical events by acting them out. Very theatrical! The fact that he did it that way of course cannot be scientifically measured. However, the fact that I retained the information, and was able to use it as the basis for drawing conclusions, etc., is.
Jeff Branzburg
I studied marketing research analysis and flirted with the industry a little before I realized my passion for teaching.
In marketing research, the sciences of statistical analysis, psychology, sociology… have been carefully developed to maximize the ability to read/predict various consumer attitudes and behaviors. This process is (usually) rigorously defined and monitored to minimize statistical error. All very important when considering the profits that can be won or lost when dealing with R&D, product launches, advertising, pricing, placement… you get the picture.
I really don’t know much about how this DOE testing was handled. But please, “a very well-done study” with “no flaws in it” ?!? Is there such a thing as a “flawless study” when dealing with the complexities of human behavior, learning and expression? No self-respecting analyst would ever use that language; which begs the question… how much respect does this study, or any to follow, warrant? I’m sure these “researchers” wouldn’t find my opinions statistically significant anyway.
The way I see it, this DOE study could very easily be a reflection of the lack of training and resources invested into the integration of technology into curricula.
As a K12 ICT Coordinator hell-bent on integration across all facets of learning, I have never been more excited about what technology has to offer learners (remembering that we are all learners). With the School2.0 movement (the Read/Write Web or, the web as a platform…), we are finally beginning to realize technology’s promise of truly differentiated learning experiences. BTW, it appears this study was based on software in the classroom – basically using newish things in old ways – very School1.0 methinks. How about webware in the school/home?
Also the language used in the Report to Congress above wreaks of bias – “…not significantly higher…” But was it statistically significant? Perhaps that’s what was meant, either way, the sentence could easily have been worded as, “Test scores were not significantly lower in control classrooms than those in classrooms using the reading and mathematics software products.” – now that’s significantly different!
The parallels between this “study” and what we think we are able to “measure” inside our schools should seem quite obvious. What continually amazes me in education is how we think we can use paper and pencil tests to reveal anything more than our students’ ability to put pencil to paper.
When given the chance to step back a little and reassess our assessment practices, we usually realize the absolute futility and disservice of our ways. In the same way I question the study’s ability to come up with anything useful, I also question current assessment practices in our schools. I’m not just referring to methods of acquisition, but also purpose, design, implementation, analysis and how we use the perceived results.
The years of training it takes to generate somewhat meaningful surveys underscores how difficult it is to “read” human experience. So why do we still devote so much student and teacher time and energy into gaining “results” that can never reflect or enhance the learning experience? I’ve started writing more on this subject at unlear.ning.com
P.S.
Keep those polls rolling in, Dave. They really are a great hook – you got me good. And thanks for putting me onto Blogflux. My students have already started using it to integrate the math strands of data analysis & statistics into meaningful subject-wide mashups. Cheers from Kuwait.
Paul McKenzie
Powerful point. It’s the “what’s really best for the kids” debate 2.0. We had this debate when standards and standardized tests came out. Long-standing educators would say, “Yeah, but what about getting the kids outside? What about field trips? What about hands-on learning?” A lot of those things got thrown to the side with our major focus on standards.
My opinion is that if we can do job of getting the content through to them while also teaching them valuable technology skills, all the better. I wonder if this study doesn’t make a powerful point FOR the use of technology. It certainly didn’t hurt to have the technology in the classroom.
This goes to the heart of what learning is about. To my mind, it all revolves around the teacher/student relationship. What really ignites a studen’ts imagination and focus in school is always related to a specific teacher, his or her inspiration, and how it serves to motive the student.
You can’t have a relationship with software, and you can’t have a relationship with a piece of paper or a video or a textbook. All of these technologies can support the learning process. But the fundamental key to learning is the teacher, and the student’s relationship with the teacher.
Is this measurable? Well…. it might be worth surveying people with a passion for learning (where do you find them? start with grad students, perhaps) and find out how many of them had an amazing teacher in elementary or high school who propelled them into their career path. I think you would be astounded at your results. No one would say that it was that fab software program, or that amazing interactive multi media thingie that propelled them into science, or creative writing or stem cell research. No, I think you’d find that it was the teacher who served as the true motivator and catalyst for growth.
Hi, Dave –
I had to sort of fudge on the answer to the survey. Could you scientifically measure the impact of my most significant teachers? Absolutely. Would that impact show up in the narrow measure of standardized test scores? Probably not. However, the career I have enjoyed for nearly 30 years is largely due to the influence of a small number of teachers that I can list – Art, Laurie, Dee, and Aurora.
I’d ask a different question. Can we scientifically demonstrate that books impact test scores?
Conn
I’d ask a different question. Can we scientifically demonstrate that books impact test scores?
or pencil and paper, or a whiteboard vs a chalkboard, or ….
These are all tools that are used for learning. Can any one impact test scores more than any other? I think the difference is, with access to information as one gets in a 1:1 environment or with the ability to access computers–a student becomes more in charge of his or her own learning…thereby allowing them to learn deeper—I do agree that a good teacher can have immeasurable impact on learning, and for me the best teachers were the ones that allowed me to do that…take charge of my own learning, causing me to learn deeper.
The problem with analysis of anything is that of quantification. I mean, any measure we have is arbitrarily defined. It is up to human beings to delegate significance or meaning to such measures.
Apart from the difficulty in quantifying “learning”, “impact” (and other such concepts) and actually assigning causational factors to such things, there is also the problem that we are all continually learning no matter what anyone says. So if you wanted to qualitatively assess the learning I achieved or the value I found in something (say, by asking me) – right after a lesson, I will tell you one thing, a week later another and 5 years later another. Those things may not be different, they may encapsulate and broaden one another…each of those things may affect the “thing” in the future… and some days I may become more aware of what factors impacted me most and some other days I may disagree with or modify my own conclusions…
I believe in scientific research (hey, I’m in the lab everyday – and we’re always faced with the age-old problems of quantification and correlation/causation issues) and maybe it is somewhat important for making big decisions with budgets and whatnot… but doesn’t it undermine the individual student in a way that “we” are supposedly trying to avoid? A teacher may say they can or cannot notice the difference in the classroom, themselves, their students… a student may say they can or cannot notice the difference in the classroom, themselves, their students… but how accurate are our reflections and perceptions when assessed with scientific rigor?
OK now some of you might want to punch me in the face because I obviously don’t know how educational research is conducted or how things are measured in this field. Oh well!
Also – reading and math? Is the direct improvement of those test scores really what technology in the classroom is about?
That’s easy – there is no way to quantify the influence really great teachers had on who I am today! Overall, it was a love of learning and an attitude that drew me. I was enraptured not by quantifiable facts or degrees or accomplishments (though those things were present) but I became enraptured by their appetites for constant learning. How does one put that in a bottle and measure it? There have been so many in so many fields….Mr. T., Nancy C., Ken S., Dusty, Mr. Sherrin, Bonnie (I went to a Quaker HS & all where called by first names)
Anyone who could claim unequivocally that “there are no flaws in my study” is no kind of scientist, and I would tend to mistrust their results.
Hi Dave,
Another interesting post…
– No “scientific study†involving computer-aided instruction can account for the “teacher role†in the learning process. Even if the same teacher facilitated all of the instruction, what was his/her aptitude regarding optimizing the learning experience using this software?
– Anyone who claims that their study has no flaws in it better take a good hard look at themselves and their research pedagogy.
– Computer software alone does not lead to a valuable learning experience…. Was there creativity involved, student project creation, etc.?
And finally, I ask, “What did the researchers really evaluate? What were they really measuring?†Studies regarding learning outcomes are highly complex and the addition of software into the learning process adds another level of complexity. Perhaps the researchers measured keyboarding skills in their study? Or perhaps their results even include a student’s ability to navigate the MacOS or WindowsOS interface? Cause and effect…suspect in this case.
I often say in my ed tech presentations that what I learned in the process of earning my advanced degree could never be quantified. All the calculus, biochemistry, and genetics that I studied created one absolutely invaluable outcome- a love of learning that has transcended my initial field of study. Priceless. Not Quantifiable. And so it is to my mentors that I owe much gratitude…not the Macintosh, the email account, or the Plato learning system that I used in 1986.
It shouldn’t be a surprise that test scores were not significantly higher in classrooms using the reading and mathematics software products than those in control classrooms. A product does not teach a student how to think or encourage higher level thinking. It takes teachers facilitating inquiry-based lessons using technology and generating a student-centered classrooms to create critical thinkers. In my fourth grade eMINTS classroom I’ve seen the results that technology paired with inquiry-based, constructivist strategies can do to raise test scores and more importantly boost thinking skills. One of my favorite quotes is †“Give students a thought and they will learn for a day. Teach them to think and they will learn for a lifetime.†Software products cannot do that.
I would quantify the effectiveness of my teaching by counting the number of children who leave my classroom thinking, “You know, I think I’d kinda like to be a historian or a history teacher when I grow up.” or “I’d like to be a doctor, or a biologist, or a nuclear physicist, or a writer, or a mathematician, or a coach, or a…” If kids are leaving our classrooms thinking that what they’ve learned is important, and they continue to talk about it, and they want to learn more, then we are sending out 21st century-ready learners.
What made my favorite teacher influential on me? Well, I immediately think of 2 teachers – Ms. Newton, my 6th grade teacher (self contained classroom) and Mr. Greenwalt my high school biology, biology2 and anatomy teacher. What made them so great to me?
For Ms. Newton it was her super classroom management skills. I remember her class as a place I could learn. She didn’t stand for any horseplay and we were constantly working. I learned so much that year. I loved that even though we had some pretty rough kids who had in the past been able to get teachers off track, she didn’t allow it. She kept them on task, too. It wasn’t a terribly student centered, problem solving, collaborative atmosphere, but it was a place where learning was the main objective and Ms. Newton was not about to let anything stand in the way of our learning.
Mr. Greenwalt was one of those teachers you knew was really concerned with whether or not you “got it”. He could look into our faces and know if we understood or not. And if we didn’t, he stopped the lesson and found a new way to help us understand. Another reason I learned so much was that his classes were so hands-on. Dissecting frogs, fetal pigs, cats…looking through microscopes, growing plants…so much more.
Now that I am a teacher and a teacher trainer, I can’t help sometimes thinking that some people were just born to teach. They just have a way of nurturing learning that can’t be quantified. The teachers themselves often cannot even explain what it is they do in their classrooms. And that’s why mentoring and peer learning is so important. We have to watch these teachers in action, work collaboratively with them, and have them reflect and try to find a way to say what it is they do so that it can be shared with other teachers.
Like Paul, I have a prior career in public opinion polling and market research (more on the former), so I’m always looking at stuff like this carefully. I thought the engagement study was the one that brought up more questions for me, and they are just starting to occur to me now. For instance, a lot of structured instructional methods include very quickly changing activities (teacher talk, talk to student partner, report out to class) in cycles of 1-5 mintues with 1/4-1/3 of the time where students talk to each other. Would that get recorded as engagement? Is it engagement? I have more questions now about what I’m doing, and what that study looked at than when it first came out.
This is a wonderful question, and the debate even better. I wonder if we would get the same responses from people who did NOT have positive school experiences and therefore don’t hang out with the love-learning crowd that frequents this blog. I hypothesize that those who never actually experienced that amazing teacher who zinged them with the cognitive equivalent of biting down on aluminum foil would advocate more for “measurement” because that s all they ever saw and felt. For them I am truly sorry.
A new question: What percentage of the U.S. (or the world) do you think can name a teacher who really “did it” for them? I hope the answer is very high….But I really can’t say, since I know only the places and people with whom I have learned and taught.
I wonder what the difference is between this study and the ones done by eMINTS? I know that the studies that support their program do show increased test scores, and it makes me wonder if they somehow focused not only on the technology or software, but also on the amount of inquiry and collaboration- the way the teacher facilitates learning? What kind of research questions would really give a more balanced and accurate picture of the kind of learning that takes place in a tech classroom? Hmmm….
I can name three teachers that really “did it” for me- and in a 70’s kind of way it was because they recognized differences, provided differentiated curriculum opportunities, and most of all really truly cared about every kid.