I love superhero stories. There is something awesome about fighting against the forces of evil and I’m frequently inspired to wear leotards and capes (but that’s unrelated). The call of being a hero has a strong pull on all of us and there are a whole bunch of obstacles to the fight that drag us down along the way. Sometimes though, we are our own worst enemies when our urge to do good leads us to undermine the progress we have made.
In 1997, the World Wildlife Fund issued a dire warning that deforestation was occurring and an alarming rate noting that “two-thirds of the world’s forest cover has been lost”. (The Skeptical Environmentalist) This number was repeated in reports and press releases, but the reason for the alarming figure (more than three times the nearest estimate of 20%) was due to an error that led to the exclusion of North America in a comparison. In 2017, the Crime Prevention Research Center published a study overestimated illegal immigrant crime in Arizona because their researcher conflated categorization in the ADC dataset leading to an overestimation of illegal immigrant crime by 3 times the amount found by most criminologists. (https://www.cato.org/blog/fatal-flaw-john-r-lott-jrs-study-illegal-immigrant-crime-arizona)
In both cases, the culprit for the result was an error in calculation, but the reason all three results went to press despite being wild outliers from other data points was that no red flags went up by the institutional referees. Nor are these the only times that this happened, there are entire sites devoted to retractions and critiques of studies. This makes two things clear to me: 1. Illegal immigrants in Arizona are responsible for deforestation, but more importantly, 2. Advocacy research is a scourge on research credibility.
Advocacy research leads to the type of errors above because results sought in the promotion of a cause increase the likelihood of biases, errors, and manipulations going undetected by reviewers and the organization. When a result supports your goals and beliefs, it does not merit the skepticism that a counter-intuitive result does. However, a drum of discredited studies, bad methodology, and conflicting results that arise has the effect of lowering credibility in the next person with a tie saying “according to research”. The gap between scientists and the public is huge with scientists overwhelmingly endorsing the safety of GE crops (88%), but the public highly skeptical (37%). EMBO reports that the gap is likely increased by the frequent publication of highly exaggerated claims (which are common enough in traditional research and compounded by the desire for readable headlines).
Advocacy research is not limited to activist groups or government, advocacy research is also an increasing problem in academia. In the social sciences, many researchers have trended away from objective research to conduct projects and pursue advocacy. Entire fields of study are established to study and promote a belief such as courses in and studies of advocacy causes. Most egregious are fields that were the focus of the Sokal Squared Hoax where researchers successfully published hoax papers that aligned with the beliefs of those fields sometimes referred to as “Grievance Studies”. These papers, like the examples from WWF, the CPRC, all revealed the same blindness to results that should have raised red flags. (https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/)
In my experience, advocacy research justifies its existence based on a few key arguments:
- Objectivity is impossible and those who claim it are dishonest
- Objectivity sustains the status quo and is an enemy of progress
- Injustices are too great to overcome due to historical conditions and advocates are leveling the playing field.
I have too much to say on each of these points, but I don’t think that advocates are wrong on any of them. I struggle with objectivity and it hurts my brain to see both sides of issues that I know well and even those who strive for objectivity frequently fall short. History is replete with evidence of resistance to change creating unfair obstacles for the truth to prevail. The answer to these questions is not to say “wrong!” but to acknowledge the weaknesses they point out are true.
My questions going forward on advocacy research are:
- Is objectivity itself a worthy goal even if is not perfectly attainable?
- What are the consequences of cashing in the credibility of research as “knowledgeable and scholarly” and are they worth the price for progress?
- If advocacy research is a problem, what are the solutions to it?
*Correction: In an earlier version of this blog I cited an example of advocacy research from the Washington Post editorial on the CDC. On further research, this was not a case of advocacy research leading to an outlier and I have removed the error. My thanks to Jayson for pointing it out!