A Third-Party Standard for Statistics, Marketing or otherwise?
I’ve written before about people blindly and mindlessly quoting “statistics” of dubious validity or sharing the latest story even when it is later shown to be misleading, but this problem will only get worse. Attention spans are approaching zero, and with the ever-increasing ease of simply liking, sharing or retweeting something with a catchy headline without even reading what’s behind it, be prepared for far far worse.
Today I regarded a very misleading infographic that showed a decline that was, in fact, probably statistically significant. It was, however, for data from a time period from September 2012 through March 2013 and had no seasonality adjustment. Of course, the reader retweeting the tweet of the article written by someone quoting the infographic is not going to dig back into the data and pull out the seasonality as an issue. Heck, they are probably not even reading the article let alone seeing the infographic let alone looking at the ultimate source of that data (which could be wrong too).
Social media and startup people seem to like badges. I believe we need some kind of human and machine-readable public responsibility “badge” that is cryptographically signed and authenticated, that is bestowed by a third party upon articles written by journalists that includes any kind of math or statistics that could be misleading and where the subject matter is of significant public import. This already applies to some extent to academic articles, where peer review is common, but that system has its own issues. No, this would have to be something that can happen far more quickly since the turnaround time on articles will need to be far quicker than for an academic piece. Some of the articles reference academic papers, so as long as there are not material misstatements of the findings, this could actually work quite well for work that is from a known entity.
It doesn’t mean the end of stuff like the guy who does a “survey” of 5 of his friends and then exclaims that the fact that 4 of 5 of them don’t intend to go to NBA basketball games anymore means the league is doomed, but it does mean that we would have a big, simple filter to at least know to cast a more skeptical eye on “non-certified” outlets or articles.
Because stupid and misleading information can spread far further, more quickly, the responsibility of those that originate it should increase too. Hopefully someone smart is already working on this problem out there in news/data-land.