Flickr Image (LHC Tunnel) by Mario Alemi
I’m on another of those wonderful stretches at home catching up with family, trying to catch some movies and mostly spending every spare minute trying to get as much office work done as I can (some writing and work on Citation Machine) with no time to read and blog.
But this post by David Wiley at Iterating Toward Openness was one of those sneaky reads that tricked me into wonder if I’m actually wrong about something. In The LHC and Education, Wiley started with his interest in the Large Hadron Collider. To say that the LHC was incredibly expensive is a sad understatement, and the machine does little more than generate data. But with that data, scientists will map realms of the universe that most of us can’t even imagine, much less see.
Whiley loves data. I love data. But he switched contexts, lamenting that…
The data that we, educators, gather and utilize is all but garbage. What passes for data for practicing educators? An aggregate score in a column in a gradebook. A massive, course-grained rolling up of dozens or hundreds of items into a single, collapsed, almost meaningless score. “Test 2: 87.”
It’s one of the reasons that “data driven decision making” doesn’t make my heart flutter the way that it does for others. It’s that, even in the best of situations, the data is scarce, shallow, grainy, and awfully expensive to collect — not to mention that the only people who can make much use of it are the data dudes that school sytems have been hiring over the past few years.
Then he totally chaffed my soul by suggesting (and rightly so from some points of view) that, “..using technology to deliver content is not improving the effectiveness of education…” but that another way of using tech might. Whiley continues,
I believe there is (another eay). I believe it so strongly that for the first time in several years I am opening a new line of research. I believe (and I fully admit that it is only a belief at this point) that using technology to capture, manage, and visualize educational data in support of teacher decision making has the potential to vastly improve the effectiveness of education.
I believe that I have written recently (Where Obama is Getting Education “Wrong”) that I think we should be teaching students to capture, manage, and visualize data as a basic working skill. It seems to me that ushering it away to the central office to worry over data as an educational concern may actually be detrimental to the learning our students need to be engaged in. Limited resources will cause us put undue emphasis on what can be easily measured at the expense of those important skills and knowledge that can’t.
But Whiley compellingly inspired in me a willingness to reconsider and I found my problem. It isn’t that I object to using data to inform better instructional decisions. It’s that the data is so lousy — ..scarce, shallow, graining, and awfully expensive to collect.
What if all of our students were doing all of their content and content processing digitally. What if all of the information transactions of learning, besides the most appropriately open conversations, was done with abundant, networked, and digital content. That would be an enormously dense, rich, and seductively meaningful mass of data that could be analyzed and visualized in a wild variety of ways. I’d be happy with that — especially if students became partners with us as self-analyzers and self-assessors, mastering their own skills as information artisans.
Powered by ScribeFire.
In Florida we just completed the first of three rounds of FAIR testing http://www.justreadflorida.com/pdf/FLAiRBrochureVer3.pdf . This is an assessment to create data to predict how students will do on their FCAT reading assessment http://fcat.fldoe.org/ . Now on the face of this it should create some usable data for teachers to use to guide their instruction. Unfortunately when the test went live earlier last month the infrastructure was not in place to handle almost every public school accessing the web based assessment. For most of the month of testing students taking the test were presented with denial of service and had to re-start the test or were not able to complete the test. Now when principals and teachers should be able to view the data gathered from the first round of testing superintendents received a letter from the FDOE http://blogs.tampabay.com/schools/2009/10/more-computer-problems-with-fair-testing.html that the database used in FAIR testing would not be accessible for an indeterminate time.
So I wonder after all that how useful will the data be.
I don’t know that we need to use technology to achieve all of what you’re suggesting, mostly because I think education practice should reflect the real world (lots of technology, but not all). I very much agree with Whiley’s senitment that you echo about the data being lousy. Alot of it is, and alot of it is just evidence of how good someone’s memory is, and how we’ve applied our memory. I think many of the issues with the data come from poor assessment in the first place. Are we trying to measure learning, knowledge, recall, outcomes? It seems to me that alot of assessment suggests it is measuring learning, but it really isn’t, it’s measuring evidence of learning, which isn’t the same.
Anyway, I can only poke sticks at the problem really, as my suggestion of qualitative analysis is often considered far more expensive and not what numerically oriented management want to know about. Hmmm… perhaps the problem is that we don’t think measurement is important enough to ascribe it an appropriate portion of our budget and time? Maybe that’s why the data is lousy?
The question is why is the data “.scarce, shallow, graining, and awfully expensive to collect”?
Multiple choice is great for creating data, but horrible at creating quality data.
We can go to all qualitative analysis, but still translating student work to numbers is difficult, expensive, and often subject to subjectivity.
I have always been a fan of individual teachers doing a lot of research in their own classroom and using the data with that same class.
If enough teachers can do that on a regular basis and upload the information to a central site the possibility of making use of the data in a general sense could become a reality.
The other can of worms is which data do we collect? Lots of disagreements there too. Quality is often in the eye of the beholder … and also tends to be driven by the agenda of the collector. At my school this is an issue. What the primary grade teachers want to use to collect “Reading” data is different than what the upper elementary wants (which is understandable actually) but 1) the powers at be want all our data to be from the same assessment tool so that “the data is consistent” and 2) the way assessments are usually packaged you have to buy the whole package (k -6 or even K – 12) which means you’d have to buy two expensive programs … there is no money (hence part of why “data is expensive” .) so we are stuck with a difficult assessment that gives unclear results that we have to translate into Edusoft instead of it being done digitally which is time consuming and frustrating … teachers have less planning time and aren’t very sure that he results they are using are really accurate … there is talk of “fixing things” by next year. Ugg!
As an individual interminably interested in data, I love working with numbers because numbers are automatically quantifiable. I know that 300 is always bigger than 200. The challenge is determining which of those numbers is better. If all I have is that one student got a 200 and the other a 300…what if those numbers represent the number of questions incorrect? The other trick is to recognize that percentiles and grade levels are both based on averages. If you took the scores on a nationally standardized test, the median score would be the 50th percentile and either the average or median score would be exactly on grade level (depending on how they compute grade level)…because the 50th percentile is ALWAYS the median score (by definition) and the grade level is based on either the average or median of the students enrolled in that grade level. It’s simply impossible to set that up any other way.
While, from a teacher perspective, I would love to have good qualitative data but, in the end, it will come back to quantitative at some point. Whether it’s in the creation of the assessment instrument or in the analysis of the results. That is what the public wants to see…how does this school compare numerically with that school. Look at everything that gets based on that…votes on new taxes often times are based on the current results; home valuations are tied to the results of the local schools; in general, quality of life in a particular sub-region is based on how well the school perform. If you don’t believe that, I’m guessing you don’t live in an area with multiple schools or school districts…just look at newspaper articles on education from metropolitan areas.
I live in a school district that is starting to do well again…I expect that my home value will go up faster than those houses in the district down the street that is going downhill (if they go up at all). All because of the numbers related to the recent state assessment and certification cycle.
How do we change that trend? I think it would be as hard as moving to the metric system in America. Americans love their numbers…we’ve been conditioned to.
DAvid-
with kids practicing math facts online we are going to be integrating this data. I know its baby steps, but this seems to be the right direction.
I agree with David about making students partners if not totally responsible for tracking themselves digitally. I just had a training yesterday and am very anxious to use a new software program called Classworks (it may just be new to me). The way I understand it, the system uses dated collected from students’ MAP scores to target areas of weakness. Another way the system can be used is by selected the state standard you’d like to address. The teacher can individualize assignments for each students or all classes and anywhere in between. Each session begins with a mini lesson followed by several exercises–including some games–and then a quiz. The student has to pass the quiz (the teacher enters the percentage to be considered passing) before going on to another section. The trainer did mention that it would move slowly if you use the same browser that your school is on, and he suggested downloading a less common one (Mozilla was mentioned) to avoid traffic. I will use the program tomorrow and see if it lives up to my expectations. I believe it will make it much easier to track student progress by generating individualized and class data by MAP units or state standards.
Dave,
You hit it right on the head when you talk about data dudes. The real problem, I think, is that administrators and teachers collect data, but they are ill prepared to deal with it properly. We can look at means and distributions, but frankly, if we don’t know how to evaluate with inferential and multivariate stats, we really can be drawing conclusions that are totally false. I think it unlikely that most teachers know an ANOVA from a chi square to a MANCOVA, and probably shouldn’t. Teacher training programs are about preparing teachers to facilitate good instruction and assessment, not become masters of quantitative or qualitative analysis. Professional development also should focus on instruction and evaluation. There’s only so much time. The quandry, in my mind, is how do we develop the expertise necessary to handle this information in a MEANINGFUL way. I am most concerned about those who “think they know” what the data says, but in fact they may be wrong and don’t know it. Imagine decisions for change based on false premises. I don’t have the answers, but think about this often.
Thanks for an interesting piece. I am now tasked to collect data for over 50% of the reading instructional time in a year ( for a 1st grade)and well over 30% of the entire year in every area. That’s the kind of data driving me right now.
What I do is record how much time, instructional time, is taken from 1st grade and turned into assessments I’m ‘mandated’ to do during the day. As well as tracking time on assessment pieces mandated like benchmark tests, District required tests, reading series theme tests and the like. Once I started tracking that it became clear I could at least reasonably conclude the issues that we see in our reading program might indeed be partially attributed to an excessive amount of time that the teacher was required to give weekly, monthly, daily in assessments.
Why we “lost” time teaching to testing-and that was done to the tune of 4and 1/2 months worth. Bet you find that exaggerated. Well, there’s the data and it’s a truth I’ll let speak for itself. It’s hard to fully explain to me why assessment in reading that’s purpose is to “inform instruction” needs to occupy 50% of the time we allow in a year to do the instruction we are informing . I’d also like to know why that’s better in any way than instruction that was informed previously by processes less “mandated” or standardized.
By all means, let’s collect data. Another thing I track is positive feedback on my work given at work. I didn’t need a full sheet of paper last year for that. I had ten entries.
I keep data on how often I’m tasked to do something without adequate planning, support system, funding….I keep plenty of data.
But the older I get, the more I am finding that while 300 may be more than 200, that probably won’t mean I’m going to be able to dent thinking allowing testing to predominate as the focus of my work-in minutes of course. Or the mind blowing realization that a reading program that REQUIRES and gives 300 minutes a day of information is alloted 110 minutes by my District in a “pick and chose” approach-so that right from day 1 they did not have the program supported with time. It’s one of the most powerfulthings you an ever do as a teacher. Track time. Chart it, look at it. Look at what programs require, look at what testing pieces take away, look at how much time goes to what. There you’ll find interesting things that are undeniable.
I found a lot in looking deeper.
Nor can I ignore what tracking behavior over twenty years has shown me about the kinds of things that are happening within student responses to choices we have made in our design.
I’m 50. If education is really all about data would you please answer for me why it is that when we think about school, recall it, discuss it with our children we usually are not standing there with a giant file folder of scores, charts, graphs, and numbers. And yet…that’s how I’m required to engage a fully grown parent about the thing dearer to them than their life, their child. It has a place, data. But if you can show something with it…often you are not asked to speak… But it is only a piece of the great and exciting work of helping children grow into happy, healthy adults.
Thanks again for the blog, helped me during difficult days.