Flickr Photo by Jukebox909 ((Jukebox909, . “Polls show distrust of public opinion.”Flickr. N.p., 16 Nov 2006. Web. 12 May 2010. <http://bit.ly/7Xouzr>.))
I have long felt that the greatest value of the social web is in the content that it generates. I suspect that the content’s value compared to the value of “nearly now” ((A term coined by Stephen Heppell)) social idea sharing depends on the person. I’m not a chatter. I procrastinate phone calls. But I love to mine the conversation for ideas, knowledge, and resources that I need right now.
An interesting example of this comes from a Carnegie Mellon University study (pdf) indicating that analyzing data from Twitter posts can yield the same results as conducting a public opinion poll, perhaps costing less and irritating far fewer people.
According to the Mashable blog post I learned this from,
A CMU team from the computer science department looked at sentiments expressed in a billion Twitter messages between 2008 and 2009. The researchers then use simple text analysis methods to filter out updates about the economy and politics and determine if the overall sentiment of the update was positive or negative. The CMU team found that people’s attitudes on consumer confidence and presidential job approval were similar to the results generated by well-reputed, telephone-conducted public opinion polls, such as those conducted by Reuters, Gallup and pollster.com. ((O’Dell, Jolie. “Could Twitter Data Replace Opinion Polls?.”Mashable. 11 May 2010. Web. 12 May 2010. <http://bit.ly/cZa2y8>.))
CMU Assistant Professor Noah Smith thinks that for at least some topics, this kind of passive information gathering could work. Mashable blogger Jolie O’Dell quotes Smith as saying, “With seven million or more messages being tweeted each day, this data stream potentially allows us to take the temperature of the population very quickly.”
Twitter data tends to be noisy, as any tweeter out there knows. But so too is even the most carefully polled data. Researchers learn to filter out the noise, the extraneous data, and round out the results to reveal trends and indicators.
Twitter, as a source for opinion trends, certainly isn’t going to work for just any topic, and the data collected via Twitter tends to fluxuate more on a daily basis than does formally polled data, as discovered by the study. But I often make the point that we will continue to need to refer to authoritative, scientific, and formally vetted information to solve many of our problems. But in a time of rapid change, we need to also develop the skills to cull out timely, experiencial, and community shapped information to answer some of our brand new questions and solve some of our brand new problems.
Added Later: From this, one might say, with an increasingly conversational and participatory web, who needs public opinion polls? Certainly the issues involved are far more complex than that. But I can’t help but wonder if teaching and learning might come to take place in a more networked, digital, and info-abundant environment, and we might continue to develop data mining capabilities, if we might reach the point where the obvious question would be, “Who needs tests?”