Juice Analytics

A reader wrote to us today:

I seem to have spent the last few days (not including the week-end I must add) trying to get to grips with ’Analytics’. If [my boss] comes in wanting a 5 word anaswer to his question “what exactly is an analytic?” I think I’d still be at a loss as to how to define it.

It’s a great question. Analytics (along with its sister/twin term Business Intelligence) gets thrown around without much clarity as to its meaning. You might think with the word in our name, that we’d have long ago nailed down a definition. Not so. (Although we do have a good understanding of what “Juice” means?)

Below is my take on a “map” of the analytics world.

I used a couple of dimensions to help frame all the parts and pieces:

• Purpose. A concept of “exploration vs. control” highlights the difference between analysis and reporting. Analysis is about digging deep into data to discover relationships, find causation, and describe phenomena. Reporting, in contrast, is used to track performance and identify variation from goals.

• Timing. Most analytics is backward looking — in an attempt to understand what has happened, and therefore be equipped to make better decisions in the future. Alternatively, analytics can focus explicitly on predicting future performance or, in the a few cases, provide information to support decisions in real-time.

I’d really appreciate any comments on this map — whether I’ve missed/misgrouped/misrepresented concepts or alternative dimensions to describe the space. The more clarity we can provide in describing “what is analytics” the more palatable the concept will be.

Zillow released its home value assessment tool recently. It is a tantalizing concept: they claim to have put a dollar value on over 40 million homes across the country. I rushed to the site and was satisfied with the results for my house. Then I was overjoyed to find that the new bathroom we are adding in the basement will increase our home value by $85,000. Nice! Better yet, I found that if I just add five more bathrooms, I can double the value of my house. I guess buyers would agree with me: it is nice to have a bathroom nearby when you need it.

Numbers like these have made some people suspicious. A recent article in the Washington Post criticized Zillow for its inaccuracies:

Offering automated property valuations via the Internet turns out to be much harder than it seems — especially if you expect them to be accurate. But after running extensive tests on this ambitious national real estate service, I found it to be so inaccurate that it’s not useful.

The founder, Lloyd Frink, fully acknowledges the problems, but believes more information is better. It can only help, he argues, to give people more information in the confusing home buying or selling process.

Here’s the problem (one I’ve run into many times in the world of analytics): if you present something with precision, your audience will believe your numbers are accurate. Particularly if you are backing it up with language like:

We compute this figure by taking zillions of data points — much of this data is public — and entering them into a formula…[it] is incredibly robust and sophisticated…Hundreds of home details feed into the formula and the home characteristics are given different weights according to their influence in a given geography and over a specific period of time.

There is a related phenomenon in software development — The Iceberg Secret — described by Joel Spolsky:

If you show a nonprogrammer a screen which has a user interface which is 100% beautiful, they will think the program is almost done.

If the front end looks nice, most people assume everything behind the scenes works well.

I feel for the statisticians at Zillow. Creating a database with a majority of home values within 10 or 20% of reality is a monumental task. Unfortunately, even that isn’t good enough. It doesn’t take many wildly inaccurate estimates to undermine the credibility of the whole tool.

I’m reminded of a story passed around in the consulting business: Imagine sitting down in your seat on a flight and noticing that the seat belt sign above your head doesn’t work. The fact that some little light isn’t working doesn’t imply there is anything wrong with the airplane’s engines, navigation system or anything that truly could impact your likelihood of arriving at your destination. But that little failure can make you nervous.

Ripped from the headlines:

To help offset gasoline prices, Budget Rent a Car is imposing an additional $9.50 charge on all vehicles driven fewer than 75 miles…”

“The new charge is aimed at renters who drive short distances and don’t fill up their tanks before they return because the gas gauge still reads “full,” even though the tank is a few gallons short. In the past, Budget filled the tank and billed the customer the highest rate. But now, Budget will impose the $9.50 charge even if the renter tops off the tank before returning the car. The charge will be removed only if customers show their gas receipt to a Budget agent, one traveler has already reported, slowing travelers often rushing to catch flights.”

“This is a convenience and time-saver for our customers,” said Susan McGowan, a spokeswoman for Cendant Corp., Budget’s parent company. “This is being done to recoup the cost of lost fuel.”

Tom Asacker’s definition of brand is “the expectation of someone or something delivering a certain feeling by way of an experience.” What feelings are Budget customers going to have about their experience? Four-letter feelings.

Budget’s mis-step here feels like analytics gone wrong–a case where a spreadsheet exercise say “go, go, go!” while any sensible person would say “stop!”. As we wrote earlier today, focusing excessively on analytics means you focus less on customer service, innovation, branding.

Thomas Davenport published an article in Harvard Business Review entitled “Competing on Analytics.” He concludes the article with a checklist of ten key points he feels are important to creating a analytics-based business.

We disagree with quite a few of these points and even where we agree, we want add real-world nuance.

The challenge of analytics is communication and creating a shared understanding. It’s about focusing on high impact areas, moving forward one step at a time, being skeptical, being creative, searching for the truth. Any company can compete on analytics, and you certainly don’t need to satisfy a checklist to do so.

Here’s Davenport’s checklist, with Juice commentary. We’re putting together a list of practical steps anyone can take.

1. You apply sophisticated information systems and rigorous analysis not only to your core capability but also to a range of functions as varied as marketing and human resources.

Analytics is hard. Analytics takes resources. It takes effort for an organization to create and assimilate learnings from analytics. You need to focus your analytics at the key leverage points of your business. As Davenport points out in the HBR article, UPS focuses their analytics on knowing where packages are, Marriott focuses on revenue management. If you try to do everything, you won’t do anything well.

2. Your senior executive team not only recognizes the importance of analytics capabilities but also makes their development and maintenance a primary focus.

Of course analytics are good. But so is branding, innovation, operational excellence, customer focus. Companies are defined by what they don’t do just as much as what they do. If you’re going to make analytics a primary focus, you will need to make sacrifices elsewhere. Which of the above are you willing to de-emphasize?

Capital One, oft cited as the credit card king of analytics, aren’t customer service champions nor are they particularly innovative.

3. You treat fact-based decision making not only as a best practice but also as a part of the culture that’s constantly emphasized and communicated by senior executives.

This is hard to argue with. However, it’s easier said than done. In our experience, getting to a culture of decision making requires your business to have real, solid wins using analytics to make people care from top to bottom.

4. You hire not only people with analytical skills but a lot of people with the very best analytical skills—and consider them a key to your success.

The problems raised by the Mythical Man Month apply to analytics. Just as doubling the number of programmers on a project won’t halve the time it takes to complete a project, doubling the number of analysts won’t make your company twice as smart.

What you need are well placed and versatile analysts – analysts that are in constant communication and debate with key decision makers.

5. You not only employ analytics in almost every function and department but also consider it so strategically important that you manage it at the enterprise level.

What does this mean?

One thought: This refers to having a Chief (Analytics|Knowledge|Data) Officer. This may be a good idea. Here’s an interesting interview with Usama Fayyed, Yahoo’s Chief Data Officer about the value of having a chief data herder at a data intensive company.

If, on the other hand, this means centralizing analytics and building a single data warehouse, we disagree. For most companies, building a big “atomic baloney slicer” for analytics is not going to work out. These approaches take too long, are inflexible, and don’t adapt to your business.

6. You not only are expert at number crunching but also invent proprietary metrics for use in key business processes.

Why is “proprietary” a good thing? What you do want is to develop a few metrics which are core to the success of your business. If you are in a well established industry, it’s likely those metrics have been defined and are well understood. There’s a lot of value in well understood metrics that everyone in your business understands. The challenge with analytics is communication and creating a shared understanding.

7. You not only use copious data and in-house analysis but also share them with customers and suppliers.

Insight is not measured by volume. As for sharing with customers and suppliers, it’s a rare company that has evolved that far (e.g. Toyota). Focus analytics where you have the most leverage to change your business.

8. You not only avidly consume data but also seize every opportunity to generate information, creating a “test and learn” culture based on numerous small experiments.

There’s lots of ways to build insight from data. It can be test and learn, it can be customer visualization, it can be scoring systems.

9. You not only have committed to competing on analytics but also have been building your capabilities for several years.

Yes. Analytics is a learning process – a journey, not a destination. The best companies have been working on learning for a long time. You can compete on analytics without having worked on it for years. Just get started.

10. You not only emphasize the importance of analytics internally but also make quantitative capabilities part of your company’s story, to be shared in the annual report and in discussions with financial analysts.

You risk hypocricy if you follow this advice. Culture starts with internal stories. External stories will arise naturally and organically from internal stories. If you focus on external stories the best you can hope for is to find yourself in a Harvard Business Review article.

The best businesses connect with their customers. They build intimate relationships, learn, and extend their products using this knowledge. After Apple learned that customers were using iPods to save addresses and data, they incorporated this feature into their next release. Intuit heard their small business customers saying, “I need to keep the books without the complexities of accounting” and QuickBooks was born.

Many companies have a different story. For them, technology has been a killer app—it’s killed the ability of individuals in the company to see their customers as individuals. Customers are a list to be manipulated, a total in a spreadsheet. They aren’t seen as people, much less as potential innovators. Dependence on big information systems is a source of the problem. These technology solutions are built to be comprehensive; built for speed; built for anywhere, anytime access. They aren’t built to understand individuals one at a time.

Sometimes the inability to understand customers stems from a business’ impatience and short-term focus on ROI. Tom Asacker pulls out an early marketing guru to make his point:

Abraham Lincoln on chopping down a tree: “If I had six hours to chop down a tree, I’d spend the first four hours sharpening the axe”.

Instead, what do most marketers do? They take a whack at the tree, put down the axe, measure the cut, pick up the axe, whack the tree in a different spot, and repeat ad nauseum. Exhausting, to say the least.

If you are in an information-rich business with many customer interactions—you can know your customers intimately. You can look at individual customer behaviors and start to recognize important and startling patterns. It will take some time, but Abe would say it is time well spent.

Here’s a common problem we run into: An organization wants to understand the dynamics of their customers as they interact with marketing channels, change products, and move between active and inactive. Presenting this type of information is tricky and cumbersome, despite the light it can shed on how a business works.

In systems dynamics-speak, this is the world of “stocks and flows.” As entities move through a system, they are either in a particular stock (aka bucket, status, state of being) or flowing to another stock. By measuring the speed of flows and levels of the stocks, you can begin to understand how to manage and optimize a system.

For us, the challenge is in finding an elegant way to visualize this dynamic data. I haven’t seen an easy or established way to handle this problem. Excel isn’t very good at it (though we wrote about how it can do the job if pressed). Here are a couple more examples of ways we to tackled the problem:

• You can show stocks and flows in a simple and intuitive way if you are willing to constrain yourself to a couple snapshots in time. The graphic below was a way we displayed the inflow, outflow, and flow between products for a client. The visual language is straightforward: size of balls represents the number of customers, size of arrows shows the magnitude of the flows.

• On another project, we tried something completely different: we created a
  movie (Windows Media only) of the movement of customers into, within, and out of the business. To make the movie, we represented each customer as a point, then took daily snapshots of each customers’ “location” (with a little extra marching between locations to make the flows come alive). It was a fun way to show dynamic data, if nothing else.

These were each custom solutions. I’ve been looking around for analytical tools that address this problem. No luck. Here’s a few interesting things I found along the way:

• Visitorville is a web analytics tool that shows data in the context of a virtual city with people (site visitors) moving around between buildings (web pages).

• Information Aesthetics is a great blog to see innovative examples of data visualization. In this post, a reference to Chaomei Chen, information visualization guru and his Top Ten Unsolved Information Visualization Problems. “Number 8: paradigm shift from structures to dynamics: towards time-varying datasets, data streams & immediate trend-detection”

• Processing is “an open source programming language and environment for people who want to program images, animation, and sound.” We’ve played around with the idea of using this as a way to visualize dynamic data.

• Systems dynamic software like iThink provides mechanisms for modeling stocks and flows. In my experience, these packages are more about creating simulations rather than reporting of historical information.

FADE IN:

INT MINIVAN

CHRIS:

(lazily looking out window, spots a hay bale

Hale of bay? Why are they throwing

JENNIE

(puzzled)

Why’s the "of" in it?

CHRIS

What are you talking about, "of in"?

JENNIE

Why are you calling me "oven"?

FADE OUT

That’s a real conversation. Thankfully, my wife and I aren’t verbally dysfunctional all the time. My personal pet peeve are meetings that exhibit a similar sort of verbal confusion. Does this sound familiar?

JIM

We need X.

AMY

You can’t have Y.

JIM

X is really important.

AMY

We’ll never be able to get Y done in time.

This is a great way to blow half an hour before Jim and Amy discover that they aren’t even talking about the same thing.

This language barrier is particularly acute when business folks try to talk to IT folks. We’ve run into this problem a number of times. Here’s a good conversation on the topic. No solutions today, just venting… and laughing.

Batchgeocode.com has put together a clean-looking front end to Yahoo’s geocoding service if you need to map small numbers of addresses.

Here our Excel-based tool that performs similar duty; it takes a list of addreses, geocodes and maps them. We’ve used this tool to map thousands of names. It’s quite a bit quicker than the batchgeocode web tool, though not as flexible in its mapping output. Enjoy.

Note: I’ve just posted an update of the Excel geocoding tool here.

This past week, I caught an interview on The Daily Show with Robert O’Harrow Jr., author of “No Place to Hide.” The book is a potentially frightening report on personal information collection by corporations and the federal government. Mr. O’Harrow offered a scary description of the dangers that await ordinary citizens caught in this shadowy experiment (e.g. jailed for a crime you have yet to commit, a la Minority Report).

As is typical, Jon Stewart asked a very insightful question (I paraphrase):

Amazon is always wrong with their recommendations; what makes us think that the government will be able to do anything with all this data?

That’s precisely the question that comes to my mind when I hear stories of data collection. From what I’ve seen, gathering data is easy enough. It is making sense of the data that is hard. The challenge is to find relevant patterns of behavior, then determining causation with important outcomes.

Jeff Jonas, now chief scientist at IBM Entity Analytics, invented a data-mining technology used widely in the private sector and by the government. He sympathizes, he said, with an analyst facing an unknown threat who gathers enormous volumes of data “and says, ’There must be a secret in there.’ “

But pattern matching, he argued, will not find it. Techniques that “look at people’s behavior to predict terrorist intent,” he said, “are so far from reaching the level of accuracy that’s necessary that I see them as nothing but civil liberty infringement engines.” – from Hagerstown Free Army blog, Intercepting Irony

Getting beyond gathering data to actual insight is a surprisingly common problem in the corporate world. There is a common progression that I’ve seen:

• company wants to be data-driven,
• company puts hooks into its customer-facing systems to gather data,
• data piles into data warehouse,
• first generation data warehouse proves unusable,
• new, better data warehouse is commissioned,
• new, better data warehouse comes online (a year and a few million dollars later)
• value of new datawarehouse is diminished by new business direction,
• money runs out for analytics projects

Even the companies that have the stamina to squeeze value from their customer data aren’t quite as sophisticated as we imagine (fear?) them to be. The reigning king of data-driven decision making, Capital One, drops a credit card mailing to me on a weekly basis even though I haven’t responded in 10 years.

Mr. O’Harrow is probably right to sound the alarms about what could be accomplished with the growing mounds of personal information — or how personal data may be misinterpreted. That said, I’m skeptical that any organization, in particular the US government, is likely to effectively use such a big pile of data.

The Wisdump blog recently did a design critique of Odeo. They made some good points but specifically thought that the sign-up form was too simple. 37signals did their own critique of the site but arrived at the opposite conclusion.

These are two intelligent and experienced teams (it’s not like any schmuck straight out of college can get his own blog) with an above average sense of what makes a good user interface for a website. But they both saw the same site and disagreed. I think a big reason why this happens is that it’s hard to separate the elements of design related to organization and the elements related to aesthetics.

Joel makes a point about this in his series on good design. However, I disagree with his point that aesthetics can only enhance a design and not take away from it (imagine if your Ipod was puke green). He’s on the right track: good design is a two dimensional problem. One dimension is related to organization and engineering and the other is aesthetics.

Shouldn’t the engineering aspect of it be more objective? If UI is engineering, than it should be more than just a variation of “hot or not“.

One of the main elements that lead to good design is the issue of prominence. What parts of a website are the most eye catching to a user and what elements belong in those prominent locations. As I see it, there are four elements that go into how prominent something should be:

• Value to the user: How much does this feature enhance the users experience and interaction with the site?
• Value to the site: How much does the site need this feature to function properly?
• Simplicity: How simple is it for the user to learn or use this feature?
• Attractiveness or convenience: How much does this feature engage the user with the site?

Next step: Is there a way to quantify these factors in order to look at UI in a more subjective way?