Juice Analytics

I was working with a client recently who had sketched out a fairly standard marketing dashboard. The company has multiple marketing channels (online, direct mail, phone, etc.) and wanted a simple report that would track leads, conversion rates, and cost per acquisition by channel. These were seemly valid and important things to know… but surprisingly slippery information the more we discussed it.

In their business, many potential customers are likely to have multiple interactions across marketing channels as they explore their options. One of these leads might first be drawn in through online search and later receive a piece of direct mail or a phone call. As a result, evaluating individual marketing channels in isolation delivers deceptive results. Which channel gets credit for the new customers? The better unit of measurement may be unique channel combinations.

It gets messier: cost per acquisition suggests to most dashboard readers that another dollar spent on that marketing channel delivers a specific return on investment. In reality, marketing channels are prone to diminishing returns as you dig deeper and push harder to find the next lead. In search engine marketing, spending more can drive competition for limited space and lead directly to a higher cost per lead. Conversion rates might better be understood as a curve rather than a single data point.

This a common situation that reinforces a few of the themes we like to harp on:

• Analytics is a journey. A simple dashboard is a fine place to start, as long as the users commit to learning from the results and push for a more accurate representation of reality.
• Ask the next question. Good analytics should stoke curiosity and push people to think more deeply about their situation.
• Analysis before reporting. Premature reporting can lock down a mistaken version of reality.

I spent the first half of our weekend morning stroll blissfully prattling on about the technical aspects of photography. I was attempted to explain to my wife, drawing on a high-school class and my flawed memory, the relationships between lens aperture, shutter speed, film sensitivity, focal distance, and light.

In the middle of this revery, my wife turned to me and explained how little she cared—and would it be too much trouble for me to stop talking about it? My semi-educational chatter was ruining the peacefulness of the walk.

I was confused. I know she loves to take pictures, and we’ve been having real problems with indoor pictures coming out blurry. Didn’t she want to know why this was happening and what it would take to fix it? Turns out she’d prefer if I’d just adjust the settings so the camera would work. Just the answer, please.

My wife is an attorney and has an amazing mind for the intricacies of legal problems. I don’t. Or more accurately, I have a visceral reaction when the conversation turns to torts, habeaus corpos, and subject-matter jurisdiction. I debate in my mind whether to try to follow the logic or simply space out. Like her, I just want the answer—not the journey.

All of which raises a question for business analysts: Is some of the resistance we encounter to data-driven decisions perhaps just a general queasiness with the detail? Perhaps the culprit isn’t some organizational culture for gut-instinct decisions or distrust of the data. It is just the journey (i.e. the presentations filled with charts and graphs and numbers) that causes people to disengage from the discussion.

On this blog, we’ve often argued that better data presentation can make your analysis more accessible and impactful. But maybe the answer can be less data presentation—until it becomes absolutely necessary.

There are many legitimate reasons for presenting the details of an analysis; there are also some poor ones. Consider whether you have explained too much because…

* you are self-conscious about your credibility?

* you want to showcase all the hard work?

* you assume the audience is like you?

* you want our presentation to lead up to a dramatic conclusion?

If I could influence the future of business intelligence tools (wait, maybe I can), I would put a premium on “tangible” data manipulation. I’d design interfaces that let users touch, play with, and sculpt data as an object.

Many data crunching applications, particularly those focused on statistics (e.g. SAS), tend to separate the user from the act of data manipulation. The user defines a set of scripts or formulas, points to a data set, and let’s the application take over. For a programmer, this type of abstraction works. For non-technical business folk, it limits our ability to understand what is happening and why the result turned out differently than we imagined.

Here are a couple interesting examples of computer interfaces that attempt to merge real-world touch and feel with digital-world manipulation of information:

Via Information Aesthetics

What if BI interfaces brought an artisan’s mentality (I’m imagining glassblower for some reason) to data manipulation? Data is the tangible raw material. When there was something odd or imperfect in the raw material, it would be obvious on visual inspection. We’d have access to a variety of tools, some for broad and crude actions, others for a more delicate and subtle actions. These tools would be put in physical contact with the data to shape it. Finally, we could add a final aesthetic finish to our creation. Analysts could take pride in creating digital objects that could move and influence others.

Related thought: can we blame the poor visualization of analytical results on the lack of visualization in the data analysis and manipulation process?

Zach and I grew up in Lincoln, Vermont, a town of 900 people tucked away in the Green Mountains. At the center of this no-stoplight village is a general store. Vaneesa, the proprietor for more than three decades, greets her friends and neighbors at the counter everyday. She has grown to know each of their habits and needs and can tailor her stock and service in response. Everyone in town appreciates it.

This type of customer intimacy has long been lost as companies scaled beyond personal relationships. In an attempt to rebuild this bond, companies pile customer data — a digital representation of customers — into customer relationship management and business intelligence databases. Storing this information does little to get your business closer to understanding customer needs. Traditional data analysis falls short by aggregating behaviors and depending on the business to ask the right question. Surveying, another approach to staying in touch with customers, is hampered by customers’ imperfect knowledge of their own needs and by their limited memory of their own actions.

We discuss a way to solve this problem in an article on the Business Intelligence Network published today.

Open up your CRM reporting tool. If it looks anything like the tools I’ve seen, it’s all about filters and metrics and bland line graphs. Now imagine it worked a little more like the new Google Finance dashboard:

or like the Baby Name Wizard:

or had an interface like News Map by Marumushi:

Why is it that consumer data visualization tools are so much cleaner, easier to understand, more polished, and engaging than enterprise reporting/analysis tools? I have a few theories:

1. Reporting comes last. Reporting is typically the last thing considered and created in development projects. Visibility into the data becomes less important that ensuring data entry, integrity, management, and access. Alternatively, it is a matter of the developers playing to their strengths. Design and data visualization clearly aren’t passions – or they’d be unable to launch with these tools.
2. Reporting trumps analysis. There is too much emphasis on delivering raw numbers, and not enough emphasis on presenting information in ways that allow users to understand what is going on. Put another way: success is defined as delivering accurate data filtered as the user has defined it. What if success was about delivering new insights about what’s going on in the enterprise?
3. Comprehensiveness is king. I can just imagine the list of requirements stacked on the desk of the team tasked to build the reporting interface. It’s a long list and shoehorning all of it into an interface is hard. The irony is, from what I’ve seen, really useful data views are left out (like time series and animation).

The examples above offer smooth-animation, mouse-over discovery and highlighting, and a consistent color palette. The baby name explorer and NewsMap are also specific tools that address one specific need very well.

Sometimes success metrics can create unexpected, misguided, and counter-productive behaviors. I heard a great anecdote recently from a client, Celia. She is a former marketing head at an airline, so she knows of what she speaks:

The other week Celia was rushing to catch a United Airlines flight in Pittsburgh. She arrived late to the gate and found the United employees were intent on closing the door to the airplane ten minutes in advance of the flight. She had to argue to get herself on the flight. At United, it seems, success is measured by the percentage of flights that push-back from the gate before the scheduled time. These employees were perfectly willing to slam the door in Celia’s face rather than face having to fill out paperwork and other repercussions tied to missing the success targets.

No doubt push-back time was considered something employees could control and reasonably correlated with on-time arrivals. In addition, push-back time is straightforward to measure — unlike the seething anger of a customer like Celia.

Analytics truism: everyone wants a dashboard (a.k.a. key performance indicators (a.k.a KPIs), success metrics, scorecards). Managers want a barometer of performance, a hammer to use on their subordinates, and a straightforward quantification of their business. Below are a few of the guidelines we use when we take on this task:

1. Actionable metrics. Ask yourself: what would I do if the metric is out of line? Do I have the levers that can impact it? Measures that track final outcomes like revenue or total customers don’t give you much time to react or guidance about what to do next.

2. Less than five. When I first started at AOL, a friend of mine pointed to the dozens of reports flying around the organization and remarked (I paraphrase): “This many ’important’ metrics just indicates that nobody really understands this business.” If you struggle to boil down, you should spend more time defining success and understanding the factors that drive performance.

3. Simplicity over comprehensiveness. We don’t agree with Thomas Davenport’s call for more proprietary metrics:

You know you compete on analytics when…You not only are expert at number crunching but also invent proprietary metrics for use in key business processes.

In our experience, you’re better off if you choose metrics that can be understood outside your corner of the world. One common trap we’ve seen is a desire to create a single comprehensive metric; this metric is often an index that combines a number of factors into an overall measure of performance. The result: numbers that are meaningless without a lot of context and difficulty in interpreting deltas.

4. Presentation matters. Your dashboard should be easy to understand and provide enough data to give your audience context. I’ve seen many dashboards that stubbornly show only the current state of a metric and the change from the previous week. Why so stingy with historical data? At Juice, we always show trending and try to give users a means to “cut” the data – by business line, customer type, month, etc. Check out our template for creating a success metric dashboard (more info below).

5. Evolve to goals. Metrics without goals can be a waste. Unfortunately, getting people to agree to specific targets can be painful. After all, goals start us down a slippery slope toward clear accountability. Here’s what I’ve found works: start by focusing your energy on getting people to buy-in to the success metrics. Get clarity on definitions, show trending, and incorporate them into the organization’s vernacular. Be patient: one day someone will raise their hand in a meeting and ask if there are targets for the metrics. Pretend to act surprised by the cleverness of this suggestion.

The success metrics template makes it easy to quickly put together a top-line report for your organization. The ’data’ worksheet gives you a place to put in your weekly (or daily, monthly, etc.) metrics. Add in dimensions where appropriate. Saving and re-opening the spreadsheet will refresh the pivot tables. Shoot me an e-mail if you have questions.

The Dashboard Spy has a great blog about business dashboards.

One of our current projects it to design or re-design the user interface for the next generation of one of our client’s products. While talking with the designers and engineers, you wonder why the people with the deepest understanding of the product aren’t always the most qualified to design the interface.

The biggest challenge of designing the UI of a complex system is that as you become comfortable and knowledgeable of a design, the harder it is for you to accurately assess how simple and intuitive it is. How do you account for this? By constantly bring in fresh eyes at all aspects of the design process. The longer you’ve been toiling over a feature, the less likely that you yourself notice a glaring design flaw.

The Wisdump blog recently did a design critique of Odeo. They made some good points but specifically thought that the sign-up form was too simple. 37signals did their own critique of the site but arrived at the opposite conclusion.

These are two intelligent and experienced teams (it’s not like any schmuck straight out of college can get his own blog) with an above average sense of what makes a good user interface for a website. But they both saw the same site and disagreed. I think a big reason why this happens is that it’s hard to separate the elements of design related to organization and the elements related to aesthetics.

Joel makes a point about this in his series on good design. However, I disagree with his point that aesthetics can only enhance a design and not take away from it (imagine if your Ipod was puke green). He’s on the right track: good design is a two dimensional problem. One dimension is related to organization and engineering and the other is aesthetics.

Shouldn’t the engineering aspect of it be more objective? If UI is engineering, than it should be more than just a variation of “hot or not“.

One of the main elements that lead to good design is the issue of prominence. What parts of a website are the most eye catching to a user and what elements belong in those prominent locations. As I see it, there are four elements that go into how prominent something should be:

• Value to the user: How much does this feature enhance the users experience and interaction with the site?
• Value to the site: How much does the site need this feature to function properly?
• Simplicity: How simple is it for the user to learn or use this feature?
• Attractiveness or convenience: How much does this feature engage the user with the site?

Next step: Is there a way to quantify these factors in order to look at UI in a more subjective way?

We do a lot of data “bashing” around here. That’s our preferred term for cleaning, manipulating, matching, and analyzing large chunks of data. Such a macho word is no doubt an attempt to compensate for our soft, white-collar work.

Even so, there is something to be said for tangibility in your work. I like the feeling of fashioning finely honed insights from the raw material of ones and zeros. Unfortunately, the tools for data bashing frequently lack a hands-on feel. (This desire for substance may be the reason why so many people are inclined toward 3-D charts.)

This desire for tangibility crystalized for me when I was introduced to a mapping tool by the folks at GeoWise. Their InstantAtlas geographic presentation software does a great job of making data easy to explore and tangible for the user. In the screenshot below, I can click towns (in red) and the data shows up in the graphs and map. That is tangible, easy-to-use data exploration.

Another data tool I like to use is called JMP. It is an intuitive and straightforward statistics/analysis package that handles larger data sets (south of 500k records). One of its best features is the ability to quickly see frequency distributions of variables – and relate them to distributions of other variables. In the image below, I’ve clicked on “very satisfied” in the first distribution (dark green) and I can see where those data points show up for the other survey categories.

Like InstantAtlas, you can pick a subset of data and see the characteristics of that data. This simple capability can make the sometimes dreary business of exploring numbers much more engaging.