The Problem with Unvarnished

There is a lot of controversy right now about a new service called Unvarnished, a system that offers Facebook users the ability to provide personal business feedback for other individuals.  There have been good discussions of Unvarnished (UV) at the LA Times, CNET, and TechCrunch (among many others).  The service works by allowing users to set up profiles for other people and then allowing reviewers the chance to provide feedback on the quality of service or general reputation of that person.  The feedback is anonymous and unsolicited, which seems to run the risk of the site serving as a forum for character assassination and gossip. Unvarnished claims that it takes specific steps to guard against this possibility and to keep the quality of the feedback high.     

Here is how Unvarnished defends its service:

• Because users have to log in through Facebook, the worst of the abuses are limited.  Basically this is the claim that reviewers are only semi-anonymous – that they are known to FB and, presumably, UV.  UV’s argument is that this semi-anonymity will keep them from slander and unhelpful feedback.

• UV monitors for the worst of the worst – they claim on their site that active defamation or slander will be removed.

• UV gives raters a dynamic score reflecting the utility of their feedback.  Other raters rate the quality of each other’s feedback, allowing the credibility of each reviewer to emerge over time.

These arguments are pretty weak.  Semi-anonymity is pretty anonymous, and people can attack others in many subtle and not so subtle ways, even if their identities are known.  To use terms from the blogosphere, UV may remove the posts of the true trolls, but this isn’t much of a defense against the concern trolls (the ones who sound like, “I’m concerned that Scott has integrity issues…”). These trolls sound just like people who have real concerns, but their agenda is simple attack. The rating system for raters sounds good in practice, but at best will end up like Amazon’s community reviewer system – much less useful than Amazon ranking to determine the quality of the book.   

The deeper problem that UV doesn’t address: selection bias.  I’ve been coaching people using feedback for 10 years and the key variable isn’t the anonymity of the raters or providers – its their motive for participation.  If you have an open process in which anyone can provide feedback but no one is asked to do so, participants come from the extremes: those who are angry at the person to be reviewed or those who consider themselves true allies.  Neutral observers tend to sit the process out (or if they do participate, their voices are drowned out by the shrill voices at the poles).  This problem is much more severe than the ones that UV claims to “solve.”  

The future will bring feedback to the web for the public user – this isn’t the system that will do it well.