There are few things more upsetting to sociologists than poorly used survey data. It’s frustrating to see data interpreted without any context or consideration of how the survey method might affect results. A very serious example of this is evident in the ‘Regnerus Affair.’ The bigger parts of the story are certainly questions about the review process, Regnerus’ relationship with the project’s funder, and the political implications of the work being published. But, I’d guess most sociologists who read the piece after it was published reacted initially to the study’s just plain bad operationalization of ‘same sex families’ and the uncritical thinking about the causal relationship between family socialization and children’s outcomes. That measurement of same sex families wouldn’t have passed my undergrad Methods course, and it got published in a respected journal.
On the face of it, it just doesn’t make sense to operationalize ‘same sex families’ as one child’s recollection of one parent’s, perhaps one time, dalliance. While I’m not familiar with the family literature, I’d be surprised if any scholars of family would accept this definition, and it doesn’t resonate with ‘common sense’ notions of family either. In undergraduate research methods classes, we’d use this as an example of questionable (like, it’ll get you a D on a good day) validity of measurement. Second, even if this was a reasonable way to measure ‘family,’ Regnerus’ explanation of children’s outcomes makes little to no effort to consider how our society’s institutional and cultural response to ‘same sex families’ might be implicated. Our major institutions don’t treat same sex parents equally. Our popular culture, by and large, doesn’t integrate same sex families into its primary narratives about what is good and acceptable. Even if Regnerus’ data provided solid evidence of negative outcomes for kids raised in same sex families, which they don’t, it would be bad sociology not to consider how these other factors might matter. Undergrads hopefully learn this in all their classes. So, Regnerus’ bad use of data is really infuriating for a lot of reasons, not the least of which is that it’s low quality sociology from a powerful sociologist.
Now, less than careful use of data is very common in the media. Usually, public opinion data are used by journalists as a hook for a story which ends up drawing on other sources for its primary claims. And really there are a lot of public opinion polls that are done just for this reason. The companies that do them mostly just want their data, and hence their company name, in the public eye. It gives them some legitimacy, and it gets them more clients. Almost any public event these days will end up being fodder for a public opinion survey, and then we’ll read stories about how congress is less popular than Nickleback. The crime here isn’t really that it’s ‘bad’ use of survey data (even though it is), but mostly that it’s just boring. Careful thought about what the results might mean actually makes for interesting reading, but journalists are often more concerned with deadlines and paper sales than thoughtful analysis.
All of this was in the back of my mind a few weeks ago when sociologist and hockey fan Avi Goldberg approached me with a fun project. He writes for this Montreal Canadiens blog, and was doing a bit on whether or not hockey fans would return after the recent NHL lockout. He had stories on three different public opinion surveys from early on in the lockout that showed fan interest waning. More than waning, really, the data were interpreted to suggest that fan anger might be the death blow for Canada’s beloved league. Avi asked for my thoughts about the data and their meaning, and I had a really fun time writing it up. He took the best of what I had to say and put it into this piece for the blog. I don’t know if others would agree with my assessment, but I do know I carefully applied my sociological imagination to make sense of the data.
That’s what we sociologists do, right? It can perhaps be forgiven when journalists, given their goals and pressures, present relatively thin takes on what public opinion data mean. That’s one of the main reasons sociologists should have a public presence on forums like blogs. We really should be responding to bad use of data whenever we see it in the media, but would there be time? It’s also, in my opinion, an unforgivable problem in that Regnerus study in Social Science Research. In that context, such bad use of data isn’t just annoying, it brings into question the legitimacy of the peer review process, and essentially the power structure of our discipline itself.