I have had a fantastic week. It started with Pinckney School District in Michigan, where I had an opportunity to speak to their very appreciative audience, and then engaged in several conversations with their tech staff (all arts and humanities backgrounds) and instructors, much of it podcasted (see They’re Out).
And yesterday! Who would know that school administrators could be so much fun to learn with. During my demonstration of RSS, one school principal actually jumped up out of her chair flailing her arms in an ecstatic frenzy. Seriously. Ask them. And I met so many people who helped me to think about so many things. I hope to explore this more in my blog later, but today I have to work on PiNet Library. Moving it over to my new dedicated server has caused some problems for some teachers, so I need to spend time today crawling around in the claustrophobic caverns of code.
I do want to explore one idea very briefly. The other day I wrote an entry called Am I Getting this Wrong? In the article I question a study that recently came out of the National Center for Education Statistics sharing data related to the digital divide. My concern was with the age of the data used in the study, a survey conducted in 2003. Stephen Downes described the study as using pre-internet methodologies, and I think that this is a fair characterization, especially considering that the topic of interest was what youngsters are using the Internet and how they are using it, a picture that I suspect is constantly changing.
Scott McLeod, the director of the UCEA Center for the Advanced Study of Technology Leadership in Education (CASTLE), very fairly described the Department of Educations NCES’ methodologies in a subsequent comment. He says…
I’ve done a lot of work with NCES reports and data. This is their normal time lag. They take a lot of time to collect, clean up, analyze, and report their data because they almost always follow very rigorous research protocols and because their data is so large-scale. This data is probably very accurate, just not very timely. However, it’s the best data we have at the national scale. Welcome to the feds!
This brings me back around to a conversation I had this summer with media specialists about the source and value of information. The NCES is, and remains, a respected source of information and their work is thorough and professional. As a source, it is practically without question. Yet, what is the value of this report, within the context of our struggles to effectively prepare today’s children for their future. For these tasks, I would probably value more casual, anecdotal observations, and casual conversation about today’s experiences than professionally collected and compiled data that is three years old.
Technorati Tags: education, literacy, technology, warlick
Data that is current, but accurate to only +/- 20 percent, is data we can use, with some caveats.
Data that is three or more years old, that is accurate to +/- 2 percent, is useless.
(Ironically, polling companies can give us data that is accurate +/- 2 percent, 19 times out of 20, and is also current. So I don’t buy this whole ‘quality control’ argument.
Casual anecdotal observations and conversations should be depended upon only for qualitative (how it feels, etc) data and for concrete facts (there is a bridge), etc., but only if substantiated (the reporter’s rule – verify your information).
Casual anecdotal observations and conversations should be *never* depended upon for statistical generalizations, ever, because people commit various statistical fallacies (and especially, generalizing on the basis of only a few instances, generalizing without examining counter-instances, asserting higher probability to more specific generalizations, etc) that have been observed and recorded over time.
p.s. this goes also for causal generalizations (if you do A, then B will happen) for the same reasons – people do not have a good natural ability to derive causal relations, which is why it took 3000 years for natural science to develop, and why people today still trust horoscopes and the like.
In early August, our Michigan Gates Grant program, LEADing the Future , conducted a Grand Finale to celebrate the ending of the four-year grant during which over 4000 Michigan school administrators received a Palm handheld computer, specialized software, and project-based professional development. So our team has had lots of “a ha” moments with administrators in both the upper and lower peninsula!
During the Grand Finale, we helped 113 Michigan school principals and superintendents set up their own RSS feeds and populate them with blogs and news of interest to them. In addition, participants created personal start pages using Protopage. A Zoomerang survey at the close of the day revealed that 71% of attendees would “probably use their new RSS feed;” 27% revealed they would “need to learn more.” In addition, 68% rated RSS feeds as “very valuable.” The survey responses to Protopage were very close to those for RSS. Not only was it a rewarding day for our team, but it was a rewarding four years!
One of the more pressing problems we face is that policy is rarely set by the anecdotal. I can show my principal the value of social tools and get permission to use them in the classroom, but without the vetted research, including those tools in the larger curriculum is unlikely.
Are we doomed to “policy lag” as the trusted sources take too long but the valuable sources aren’t trustworthy enough for systemic change?
Isn’t it ironic that in the day of RSS feed that we would have to wait 3 years for a study on the internet? I bet they could process the study in India for a fraction of the cost. Just think of it one less dinosaur in education.
A couple of comments. First, while the report came out in 2006, the data was collected in October 2003 as part of the Current Population Survey (http://nces.ed.gov/surveys/cps/). This the is monthly survey conducted by Census which gives us the unemployment rate for the country. Census includes periodic supplemental questions to help measure other trends as well.
As with other government statistics, there is often more data available than time to analyze it. As a result, the data is often made available (http://www.bls.census.gov/cps_ftp.html#cpssupps) and various government agencies develop reports in the subsequent years. For example, this supplemental has been used for a number of reports including a preschool analysis:
http://nces.ed.gov/pubs2005//2005111.pdf#search=%22Current%20Population%20Survey%2C%20October%202003%20School%20Enrollment%20and%20Computer%20Use%20Supplement%22
and a broadband reports by NTIA in 2004:
http://www.ntia.doc.gov/reports/ anol/NationOnlineBroadband04.doc
As another commenter pointed out, NCES has a rigorous process in not just developing the report, but placing it through a technical review to be sure the analysis was correct. Most likely NCES only received funding to do the analysis a year or so ago and this was the most current data set. It’s not perfect, but most of the government indicators have some lag to them (even the monthly employment and unemployment rate numbers go through monthly revisions of the presious month).
It is far from perfect, but probably provides a more accurate picture than many of the other surveys conducted simply because of the size.