Is a ratings system for journalists a workable idea?

We live in an age of ratings systems. On any given day, many of us interact with any number of schemes that involve the rating of movies and restaurants, Uber drivers and Air BnB owners, online retail transactions from Amazon to Ebay, and professional service providers including doctors and professors. Sometimes we are the ones being rated, sometimes we are the ones doing the rating, but more often we use the crowdsourced ratings to guide our behaviour and our choices. Some of these systems are better than others, but for better and for worse they have become part of our social infrastructure.

So it was interesting last week to see what happened after Elon Musk took to Twitter to suggest that he was going to start a ratings system for journalism:

“Going to create a site where the public can rate the core truth of any article & track the credibility score over time of each journalist, editor & publication. Thinking of calling it Pravda…”

Truth be told, Musk was grouchy. He’s been having a tough time of it the last while, largely because of ongoing production and safety problems at Tesla. He wasn’t happy with some of the coverage, and he took it out on “the media” who, he seemed to be claiming, had an agenda that went beyond telling the truth:

Problem is journos are under constant pressure to get max clicks & earn advertising dollars or get fired. Tricky situation, as Tesla doesn’t advertise, but fossil fuel companies & gas/diesel car companies are among world’s biggest advertisers.

The reaction from journalists was extremely hostile, to put it mildly. Most of the responses were straightforward ad hominem attacks on Musk, the most popular being versions of: Musk is ignorant (because this isn’t how journalism works); Musk is a hypocrite (because he’s profited enormously from a fawning press); he sounds a lot like Donald Trump (because he’s attacking the media instead of dealing with his own problems).

Aside from the fact that accusing someone of sounding like Trump is the new accusing someone of sounding like Hitler, these responses were for the most part correct. Yes, Musk was having a hissy fit. Yes his motivations are suspect. Yes, he doesn’t understand how journalism works. But they are also largely irrelevant to the larger question of whether a ratings system for journalists is workable, and if it is, if it is a good idea.

And so a couple of days ago, the Toronto tech entrepreneur Dan Debow wondered on Twitter:

Unlike most, I trust and respect journalism and the media. I get their role and job.  But I don’t understand why journalists & journalism would not benefit from a well designed public ratings systems (despite flaws) that have improved transparency and service in other domains…

It’s a pretty straightforward question. And it is also one that I don’t think the journalists and academics responding to him (and to Musk) have done a good job of answering. I have seen four general lines of objection: First, any proposed system could be gamed or abused. Second, even if a workable system could be designed, it would be dumb, pointless, or unnecessary. Third, a workable system would be dangerous to journalism. And fourth, journalism is simply not the sort of activity that admits, even in principle, of being rated.

Regarding the first objection, it seems to me this is just a deflection. A ratings system doesn’t have to be perfect for it to be useful, and the question of whether a useful platform can be developed is an empirical one, surely. At the very least, it is certainly a worthwhile exercise to explore Debow’s challenge by accepting the premise: assume a well-designed system. How could it hurt?

One response is that it won’t hurt, it will just be of no use. This was the approach taken by the tech writer Lance Ulanoff in a widely-circulated piece on Medium:

As I tried to explain to him [Musk] on Twitter, third-party ratings systems that try to define and increase credibility have fallen out of favor. Klout couldn’t survive, no one cares about Foursquare’s reward system anymore (I’m the major of “The Media!”) and journalists will not care about a Pravda credibility score.

So, even a workable system would be useless. Except as Ulanoff goes on to write in the very next sentence, the problem isn’t that no one will care, it is that people will care way too much:

If anything, such a site would cause endless anxiety and trouble for reporters trying to tell difficult stories. The truth often hurts. Angry readers or those who think they’ve been maligned by a factual-but-less-than-favorable review or piece of investigative journalism will take to Pravda and try to destroy the reporter’s credibility, but with Pravda’s carefully crafted ratings system behind them. It’ll be much more effective than trolling on Twitter and far more damaging to journalism in general.

Amazingly, this is the exact same bit of heads-I-win/tails-you-lose line run by the media professor Keith Richburg in an interview with CNN in which he asserts both of the following: On the one hand, Musk’s plan is dumb. There is no need for such a ratings system, because the media already does its own fact checking and error correction. On the other, Musk’s plan is dangerous because it will only serve to amplify the already widely-held view that the media is lacking in credibility.

It is hard to see this sort of double talk as anything more than unalloyed special pleading by people who don’t like a proposal but can’t for the life of them think of a non-selfish reason why.

***

The most interesting response to the Musk/Debow proposal is the suggestion that journalism is not amenable to a ratings system because of its particular, perhaps even unique, position in our cultural firmament. The idea is that because journalism isn’t a profession, it doesn’t usefully admit to being rated. One version of this claim comes from David Skok, responding to Debow (I should note at this point that both Skok and Debow are friends. I’m trying to be air to both their positions but it’s hard to reconstruct Twitter exchanges in a way that doesn’t make a blog post unreadable. Feel free to read the original exchanges). Anyway, here is Skok’s reply:

Part of the conceit in these proposed system is that they imply a “professionalism” to what is, fundamentally a “craft.” Once we go down that road we are at odds with the founding constitutional liberties of free speech/first amendment. (2nd amendment in Canada).

The Globe and Mail columnist Doug Saunders made a similar argument on Twitter (Again, I’m putting together a few disconnected tweets here in a way that I think does justice to his views. To see the whole thread, go here).

The assumption of Musk et al is that journalism is a pseudo-profession with a measurable forensic premise: “Let’s determine the truth.” But at its best it’s a narrative practice with a civic premise: “Let’s discuss how much more complex and less self-evident things are”

The idea of “ratings” imagines a sort of 1950s US model of journalism in which a “story” is a collection of quotes bound to a declaration. That was terrible journalism, and it helped create a damaging myth of closed professionalism within the craft. Not to be encouraged.

There’s a real danger in attempts to treat journalism as if it were a profession, not least in creating a public sense that it is an elite or restricted trade that couldn’t be practiced by just anyone.

So Skok suggests that treating journalism as a profession whose activities can be judged will raise the spectre of free speech infringements. Similarly, Saunders seems worried that a ratings system will become a de facto professional accreditation system.  I have to confess that I don’t really follow the reasoning of either Saunders or Skok here. To begin with, it is not clear to me why the fact that something isn’t a profession implies that its practitioners can’t be evaluated or graded. Yes, some sites or platforms explicitly rate professionals (doctors and professors in particular) but running a taco stand isn’t profession, and being an Uber driver isn’t a profession, and we seem to have no problem rating these sorts of activities.

But more to the point, there’s a slippery bit of reasoning at work here. Both Skok and Saunders begin by noting that journalism isn’t a “profession” in the same way that being a lawyer or doctor is a profession, and conclude from this that any attempt to name someone as a journalist and evaluate them according to some relevant set of disciplinary standards risks undermining the fierce independence of the business. But Saunders goes further, and seems close to denying that there is any such thing as journalism:

Of course there are objective facts. And there is correct spelling. But some of the worst and most irresponsible journalism gets the objective facts 100 per cent right and spells everything properly. And a code of conduct is just human ethics — that’s not special to journos

You will find, if you examine the code of ethics at any outlet, that “the practice of doing journalism” is identical to “the practice of human communication.” There is no special thing called journalistic ethics. There is just ethics.

There are a couple of problems with this. The first is that it is at odds with how journalism actually treats its own practices. The second is that it is at odds with how the law in Canada treats journalists.

***

It is true that unlike chartered accountants or architects, journalists have resisted creating a guild-style accreditation system or body that would be responsible for setting and maintaining standards and determining who qualifies.  Journalism also lacks many of the characteristics that are typically invoked in defining a profession: there is no esoteric body of knowledge that journalists need to master; they are not credentialed or licensed; strictly speaking they have no clients; and there is no mechanism for excluding non-journalists from acting as journalists.

All of this is true. But it doesn’t follow from this that journalism cannot be usefully distinguished from kibitzing on Twitter or dinner party chit chat, and there are a handful of important ways in which journalism behaves like a profession. For starters, journalism serves a number of vital social and political functions, including providing the information that allows citizens to make informed political choices, giving a coherent analysis of events, an investigating the uses and abuses of concentrated power.

Because journalism serves these (and other) vital functions, and because journalists are typically held to account for how they perform these functions, most news organizations have taken steps towards professionalization. Virtually all have internal codes of conduct or ethics guides that spell out the ethical obligations of their employees. (A typical one can be found online here )

Most of these codes focus quite explicitly on matters of professional conduct (e.g. when to identify yourself as a reporter; how to handle anonymous sources, what is meant by “on background” or “off the record”) or provide guidelines for avoiding conflicts of interest, real or perceived. And it is simply disingenuous to suggest, as Saunders does, that these codes are not materially different than the codes that govern “human ethics” in general, or apply to all forms and formats of human communication.

And it is precisely because journalism has itself adopted quasi-professional practices and guidelines that in many jurisdictions journalists are afforded special legal status and privileges, including enhanced protection against state surveillance and the right to protect sources. This includes Canada, where a bill to this effect was passed last year, and Quebec, where a complementary Bill is in the last stages before getting royal assent.  And crucially, both bills explicitly define who counts as a journalist. (The Federal law is here, the Quebec bill is here).

These quasi-professional codes of conduct, and the growing sense that journalists as a class are entitled to protections that the kibitzers on twitter are not, is a simple acknowledgement of the power and influence that journalism has in a democracy, and signals the way journalists are in a special position of trust. So we can say that journalists are professionals in at least the following sense: the are in some sort of a fiduciary relationship with respect to their fellow citizens. This means that the morality of journalism is a role morality, insofar as it is functionally determined by the role journalism plays a democratic society.

***

I do not say that a Yelp-for-Journalists is a good idea that can be well-executed. I do say that I’m not persuaded of its lack of merit by the arguments that have been advanced so far. 

 Ultimately, I think I’m with Siva Vaidhyanathan, a media studies professor at the University of Virginia who told CNN that it’s not a crackpot idea, though Elon Musk probably shouldn’t be the one to run it. But if it was carefully designed, and was run by an independent journalism foundation, it is possible that it could be of some use. 


Comments

Is a ratings system for journalists a workable idea? — 9 Comments

  1. It seems to me that a rating system tracking accuracy presupposes answers to the question “Why isn’t journalism good?” finding faults in the dispositions and practices of individuals and institutions. What if the problem is the economics of the industry favor consolidation, closing of foreign bureaus, overworked generalists whose primary goal is the chase clicks? Even in a more ideal scenario where journalists with expertise report on car manufacturing and testing (like what Carl Zimmer and Ed Yong do for biology, a field with more latent audience interest) that’s not going to stop generalists banging out nonsense. If news was healthy enough they could afford to provide a loss leading public service about important but less popular topics not dressed up with controversy, but it’s not?

    There’s also a sense in which experts see error where audience dictates simplification. There are many important details the uninitiated should know to understand a topic that aren’t going to get across in the space limitations, particularly when there isn’t the time or money to write, edit, rewrite.

  2. I like that this post takes Musk’s claim seriously and takes aim at bad arguments that it would be in your own interest, as someone deeply involved with the profession of journalism, to let slide. That’s great.

    A simple illustration of the idea that rating journalism can work is that there are already sites that do this. Here is one, for instance, that’s good! https://www.healthnewsreview.org/reviews/

    It’s a different model from Yelp (all the reviews are done by experts, and there are gatekeepers guarding who gets to post a review), but it’s certainly possible and useful to review journalism!

  3. I don’t think a rating system needs to be tested IRL, first. It could in theory be tested experimentally. Professors for example get an idea of the implicit rankings that students give to various sources of info by looking at the sources they cite in their essays. It’s not hard to think that this could be formalized: students could be given access to a curated list of sources, and they could be asked to rank them. Those rankings could be compared to rankings given by journalists, experts, etc. Not a perfect proxy, but it would begin to tell us whether the armchair criticisms of the very idea of a ranking system have any merit.

  4. I rate this posting I for Interesting and L for Let’s hear more of your argument than just a dashed off last line

  5. Assessing the credibility of journalists is a sub-issue of something I consider more important: assessing the credibility of any internet commenter, whether or not he/she is a journalist. There are thousands of journalists, but there are millions of commenters out there that aren’t journalists and some have a larger following than most journalists.

    Assessing the credibility of journalists is a sub-issue of something I consider more important: assessing the credibility of any internet commenter, whether or not he/she is a journalist. There are thousands of journalists, but there are millions of commenters out there that aren’t journalists and have a larger following than most journalistis.

    I proposed a solution a couple of months ago (See https://kenfromottawa.wordpress.com/2017/10/19/internet-author-credibility-the-missing-piece-of-the-puzzle/).

    The high points:
    1. Everyone that publishes has a public credibility rating:
    2. That rating is crowd-sourced, in that there is no central registry. One’s credibility is a combination of the assessments of other internet denizens.
    3. Most importantly, those assessments are weighted by the credibility of the assessor.

    I envisioned a marketplace of ideas where credibility is an attribute of every commenter and low-credibility sources can be filtered out, until their credibility reaches a certain threshold, or not.

    Ken, from Ottawa

  6. Call me cynical. But to understand why a ratings system would be a bad idea you need look no further than the comments section of any online news outlet.

    A journalists job is to inform the public, which necessarily means to tell them things that they didn’t already know, and in many cases don’t want to hear. What constitutes “good” journalism is inherently a loaded question.

    “but running a taco stand isn’t profession, and being an Uber driver isn’t a profession, and we seem to have no problem rating these sorts of activities”
    This is a terrible analogy. If I go to a taco stand I can rate the quality of the food and the speed of service, because it is directly relevant to the transaction taking place. I can rate an Uber driver based on how long it took to get to my destination, the cleanliness of the vehicle, and the rudeness (or hopefully lack thereof) of the driver himself. In both of these cases, I am rating an inherently subjective experience, but one where the definition of “good” quality would have a relatively low variance across the vast majority of the general population.

    The same absolutely cannot be said for information generally or journalism more specifically. People have very different preconceived notions of “truth”, and are surprisingly tolerant of belligerence so long as it squares with their preferred ideology. The problem is not that a ratings system would be open to abuse, but that abuse is inevitable given the huge diversity (to use a euphemism) of political opinions and interests… as opposed to, say, the number of ways in which you can expect an Uber to take you to your destination. You would have to wade through a lot of crap just to get to anything even remotely resembling useful information. So the contention is not that it can’t theoretically be useful, it’s that the likelihood of abuse greatly outweighs any possible use for such a system.

    Now, that said, one way something like a ratings system could conceivably portend credible journalism is if the people doing the rating have an incentive to be honest, which would likely mean something like a professional organization or licensing body with a clear set of rules and standards, as is alluded to later on in the article. But that would open the door to accusations of bias and political meddling based on who happens to be responsible for nominating the people doing the rating, or worse yet, “elitism” from a body dictating alternate facts.

    Truth does not necessarily follow from credibility, nor vice-versa. This is especially true when that credibility is dependent on the trust of the median intelligence.

  7. Ofcourse it is possible to rate news articles. Healthnewsreview.org prooves how and is the result of the right design specifications. If a news article is the product / outcome of a “journalist”, and this product is consistently being rated poorly, then their might be a correlation. If a chef’s special is consistently not ok, then there might be something wrong with the chef’s skillset. On some topics I would like to be able to visit “a Michelin three star restaurant”, for example when I’m going to vote, buy medicine, a house, a car, going out on a date. Where do I find such high quality news articles written by “journalist rock stars” (refering to the status some restaurant chef’s have these days)?

Leave a Reply

Your email address will not be published.