Zillow released its home value assessment tool recently. It is a tantalizing concept: they claim to have put a dollar value on over 40 million homes across the country. I rushed to the site and was satisfied with the results for my house. Then I was overjoyed to find that the new bathroom we are adding in the basement will increase our home value by $85,000. Nice! Better yet, I found that if I just add five more bathrooms, I can double the value of my house. I guess buyers would agree with me: it is nice to have a bathroom nearby when you need it.
Numbers like these have made some people suspicious. A recent article in the Washington Post criticized Zillow for its inaccuracies:
Offering automated property valuations via the Internet turns out to be much harder than it seems -- especially if you expect them to be accurate. But after running extensive tests on this ambitious national real estate service, I found it to be so inaccurate that it’s not useful.
The founder, Lloyd Frink, fully acknowledges the problems, but believes more information is better. It can only help, he argues, to give people more information in the confusing home buying or selling process.
Here’s the problem (one I’ve run into many times in the world of analytics): if you present something with precision, your audience will believe your numbers are accurate. Particularly if you are backing it up with language like:
We compute this figure by taking zillions of data points — much of this data is public — and entering them into a formula...[it] is incredibly robust and sophisticated...Hundreds of home details feed into the formula and the home characteristics are given different weights according to their influence in a given geography and over a specific period of time.
There is a related phenomenon in software development -- The Iceberg Secret -- described by Joel Spolsky:
If you show a nonprogrammer a screen which has a user interface which is 100% beautiful, they will think the program is almost done.
If the front end looks nice, most people assume everything behind the scenes works well.
I feel for the statisticians at Zillow. Creating a database with a majority of home values within 10 or 20% of reality is a monumental task. Unfortunately, even that isn’t good enough. It doesn’t take many wildly inaccurate estimates to undermine the credibility of the whole tool.
I’m reminded of a story passed around in the consulting business: Imagine sitting down in your seat on a flight and noticing that the seat belt sign above your head doesn’t work. The fact that some little light isn’t working doesn’t imply there is anything wrong with the airplane’s engines, navigation system or anything that truly could impact your likelihood of arriving at your destination. But that little failure can make you nervous.