Juice Analytics

Sometimes a simple metric isn’t enough. It can’t fully describe a behavior or performance of a system. That’s when you need a Franken-measure: a made-up metric monster that creates a comprehensive composite to capture complex concepts.

Franken-measures go by many names—indexes, scales, ratings, composite or compound measures—and show up in all sorts of places:

Web analytics has an ongoing discussion about a measure of visitor engagement; the famous Google PageRank measures the “importance” of sites using a complex and mysterious algorithm.

Sports have embraced Franken-measures to evaluate player and team performance, e.g. passer ratings, Rating Percentage Index for college basketball, and judging of Olympic events like gymnastics, ski jumping, and ice dancing.

Economists loves indexes, e.g. Consumer Price Index, Consumer Confidence Index, Gross Happiness Index.

Marketers use “scores” to simplify their lives, e.g. Q scores measure the familiarity and appeal of popular culture entities and credit scores judge your value as human being.

Why would I want a Franken-measure?

You are probably already up to here with measures, so why would you want another one—much less one that is going to need extra effort and explanation? Here are a few things Franken-measures can offer:

A short-hand way to communicate about a complex concept. For example, a concept like customer loyalty may encompass everything from share-of-wallet to frequency of interactions to average sales amount.

A mechanism to operationalize a complex concept. Systems can take action on a single number more easily than an array of variables.

A definitive weighting of factors. Rather than constantly bickering about the relative importance of various measures, a Franken-measure can lock down the weighting, avoiding individual biases (in exchange for a systematic bias).

A balance of components. By combining multiple measures, variation in one measure doesn’t unduly bias the results.

What does it take to design an useful Franken-measure?

Not all Franken-measures are effective at achieving these benefits. There are at least four elements that contribute to a good design: completeness, concision, measurability, and independence. These factors can be combined into the Franken-measure Effectiveness Index (FEI) using Juice’s proprietary weighting model.

Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept.

Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users.

Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile.

Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.

What can go wrong?

Finally, here are a few of the pitfalls to avoid when setting out to create your perfect Franken-measure:

Complexity. A complex calculation can confuse and infuriate your audience because it is hard to understanding what is driving performance and why the measure is moving. Leigh Steinberg, famous NFL agent, said of the NFL passer rating: “Other than one attorney in our office, I am unaware of a single human being who has the capacity to figure a quarterback rating.” The formula isn’t quite as inpenetrable as that, but it isn’t for the weak of heart:

Changing the baseline. There will be inevitable pressure to change the franken-measure formula which automatically invalidates historical performance.

In search of comprehensiveness. A desire to be comprehensive can hamstring the effort. Take Eric T. Peterson’s Engagement Model. He is clearly striving for completeness but at the risk of feasibility, in my opinion.

Black box and credibility. For the people impacted by a Franken-measure, it is important to understand what is going on under the covers. And if it is impossible to share the algorithm or approach, credibility of the creator is all that remains. PageRank succeeds to the extend that people trust that Google has an objective, well-intentioned algorithm. A whiff of agenda or bias would undermine it in the eyes of the audience. Take the National Review’s “Liberal Rankings” which have managed to label the last two Democratic Presidential nominees as the “Most Liberal Senators.” Coincidences like that can undermine credibility.

For more information:

• Compound Measures, University of South Carolina
• Composite Measures, Wikipedia
• A Primer on Metrics, Gary T. Smith
• Choosing the Right Metric, Juice Analytics
• 5 Rules for Successful Success Metrics, Juice Analytics

Here’s a little predictive analytics:

About a year ago, I took a swipe at the “$80 million supercomputer to analyze NYC student achievement.” It smelled more like a super sales job than a super useful analytical tool.

At the time I had said:

Teachers are underpaid, hardly appreciated, and overworked. I can only wonder what the half-life is of a system that asks teachers to log on to get information delivered by the “chief accountability officer.”

Well, it appears that things haven’t gone that smoothly with the supercomputer. Today, I received a link from Leonie Haimson, a NYC education advocate, to a story entitled SCHOOLS COMPUTER AN $80M ‘DISASTER’.

Not only has the supercomputer struggled to gain much traction with users (“The school system’s new $80 million computer super system to track student performance has been a super debacle, teachers and principals say”), it has coincided with severe budget cuts.

We see these data warehousing problems all the time with our clients, and the NYC supercomputer displays all the hallmarks:

• Delivery delays: Nearly six months after the Department of Education unveiled the “first of its kind” data-management system, the city’s 80,000 teachers have yet to log on because of glitches and delays.




• Bad user experience: Many principals have complained that it runs slowly, lacks vital information, and is often too frustrating to use.




• Complicated training and set-up: School officials were hoping to have everyone hooked up and trained within months… delays in creating IDs and passwords for teachers
• Trying to do too much, delivering too little: The principal added that she preferred to get student information from a combination of old data systems “rather than wait for ARIS to churn and churn and churn and maybe give me half the report I need.”
• Massive cost: Complaints about the expensive system—on which nearly $35 million has been spent so far—have gotten louder since the city unceremoniously chopped $100 million from individual school budgets last month.
• And yet, few success anecdotes to justify the investment: ARIS had already enabled her data team to analyze the performance trends of the school’s many English-language learners.

It does offer one thing that I haven’t seen before: a Chief Accountability Officer.

175+ data and information visualization examples and resources

There is little ‘r’ reporting and there is big ‘R’ Reporting, and the difference is vast:

reporting is the stuff that comes straight out of your reporting solution. It gets created by choosing a few parameters and typically shows up as a table of data with an accompanying chart.

Reporting is something altogether different. It is concise set of graphics and data that tell a focused story. It is crafted to focus on the key information and exclude everything else. It may come in the form of a single dashboard or a 20=page deck, but it is always audience-friendly. It is informed by context and provides explantation. Reporting is not about the numbers, it’s about what the numbers tell you.

By analogy, what if we didn’t make a distinction between a raw fish pulled out the sea and a prepared fish dinner? When the waiter slapped a still-squirming sea bass on my plate, I probably wouldn’t take much consolation in getting a deboning knife and a hot plate.

In the wild, the two species of reporting are often confused. To help you identify one from the other, I’ve put together a couple of examples with tell-tale signs:

The difference comes down to a gut-feeling: Was this document created to address the questions of a specific audience with a specific problem?

This may be a distinction that is implicitly well-known. My concern is more about explicit acknowledgement of the gap between them. And in the process:

1. Avoiding passing off reporting as Reporting. In particular, vendors who offer reporting tools think they are delivering the ability to communicate performance, when in fact they are mostly providing the raw materials.

2. Recognizing the level of effort required to transform reporting into Reporting. Analysts spend a huge amount of time filling this gap; it is one of the wasteful backwaters of modern enterprises.

This has been a common theme in my recent client discussions. People are sick of slogging through their reporting tools to build useful information for management. Ultimately, developing great Reporting requires an understanding of problems, the audience, and thoughtful design. But that doesn’t mean it should be so painful to construct. We are working on a solution to help, but in the meantime here are a few general things we do:

• Gather data in its cleanest form (CSV instead of heavily formatted XLS, or in the worst cases PDF)
• Automate data cleaning and manipulation steps using Excel macros and VBA
• Create repeatable and documented report building processes
• Try to convincing executives that less reporting can be more valuable

We don’t tend to agree with Microsoft when it comes to data analysis and presentations. In fact, we’ve even been critical of them for misrepresenting data, excessive visual “flair”, missed opportunities to improve Excel, forgetting their power users, subpar presentation tools, and wasteful slide masters.

With all these past differences, I was a little surprised to find that we do share some common ground. Check out the comments (from an article in Internet News) by Peter Klein, CFO for Microsoft’s Business Division in describing the world of business intelligence:

“I’ve talked to a lot of customers about business intelligence and the one thing that they tell me is it’s really hard to use,” said Peter Klein, during at the Credit Suisse conference.

“‘I’m not getting the value out of the investment that I made,’” Klein said customers had complained. “‘I have invested a lot in my back-end systems, and today 10 percent or less of my employees actually touch it, or get access to the data. I’ve got six different BI solutions across multiple different departments, none of which talk to each other. And they’re hard to use, so I’ve got to send people to training for two weeks to learn how to use it.

Finally, we are speaking the same language. Now, I’m curious to see what they are going to do about it.

I recently ran a few training sessions about how to visualize and present complex data. The high point was a series of “extreme slide makeovers” in which I honed the message and cleaned up visuals from existing presentations. Here are some ideas to tame busy, confusing slides.

1. Simplify your slide master, make room for content. Fancy borders, elaborate fonts, and background images do little to impress your audience. They leave little room for communication, either. For those saddled with frilly corporate slides, you’ll have to take on the Brand Standards Police.

   It may help to get quantitative. Consider this PowerPoint standard slide master. Less than 50% of the total slide area (highlighted in green) is available for content.

   
   

2. Say something once, why say it again? The Talking Heads sang: You’re talkin’ a lot, but you’re not sayin’ anything / When I have nothing to say, my lips are sealed / Say something once, why say it again?

   Wordy slides can be confusing and tedious. The author is using a lot of words—and often lots of qualifiers—in hopes that the core point lies somewhere within. The burden of synthesis is shifted to the audience. That’s not fair.

3. Make one point per slide. The take-away sentence on your slide should clearly state your point; the data on the slide should support that point. Any information that is tangential to the key concept can be pushed to an appendix or supporting slide.

4. Redundancies cause unnecessary repetition. I was surprised in my slide makeovers how often I found information that could be consolidated to simplify the slide. Redundancy came in many forms: multiple graphs repeating the same legend, axis labels that are described in a chart title, restating the same point.

5. Christmas is over, take down the decorations. Clear out clip art, “screenbeans”, and other images used to dress up the slide. Most effects are less “dazzling” than you might think. Eliminate gradients, shadows, 3D effects, and most animations. These design effects were exciting 10 years ago. But if they don’t help you communicate, move on.

   On the other hand, consider using full-screen photos as a way to convey a idea or theme, accompanied by few words. Here’s an example from a presentation I gave a few months back:

   

6. Reduce chart-junk. Excel and PowerPoint charts come pre-packaged with a heaping helping of chart-junk (“unnecessary or confusing visual elements”). Here are a few things I change in a default column chart: no shaded background, grey gridlines, no chart outline, no y-axis line, no column outlines, turn off auto resize text, change column colors to increase contrast. If you want to save yourself from chart-junk induced carpal tunnel syndrome, check out Chris’ chart cleaner Excel add-in. Sometimes charts aren’t necessary at all. If you’re using a pie or stacked column chart to show a single data point, the number alone will do the job more clearly.

   
   

7. Delete your “Text-junk” too. Text can contain “chart-junk” too—visual distractions in text that dilute your message.

   • Title Capitalization or Other Excessive and inconsistent use of Capital Letters. Title caps doesn’t make sense to use and is more difficult to read.
   • Underlining. If you want to emphasize a word or phrase, use bold or italics.
   • Don’t use bullets when there is only one item or sentence. People have become so accustomed to using bullets that they’ll use them when they are totally unnecessary.
   • Bad fonts: The worst is Comic Sans MT, as the LMNOP blog describes: “These days, just like an e-mail from an “@ aol.com” address has a distinct lack of credibility, an e-mail written in this font makes the sender seem ridiculous and out of touch.”

8. Simplify style and formatting. Inconsistent colors, fonts, font sizes, and other styles are a subtle distraction. Limit yourself to three font colors (emphasis!, normal, low-emphasis), three font sizes, and three font styles. Here’s an example.

   Three colors

   

   Three sizes

   

   Three font-styles

   

   Putting it all together

   

All these points can be summed up as: Make everything on your slide serve your story. Best wishes for 2008!

Find and Download Great-Looking Excel and PowerPoint Charts

Chart Chooser is an online tool that answers two questions we commonly get:

1. What type of chart should I use to show my data?
2. How can I make good looking Excel or PowerPoint charts?

Chart Chooser is easy:

1. Check the boxes on the left that best describe your objective
2. Select the chart that you want to use
3. Choose from Excel or PowerPoint downloads to get a formatted chart template

A few notes about Chart Chooser:

• Thanks to Andrew Abela of Extreme Presentations for inspiring Chart Chooser with his “Choosing a Good Chart” post and for working with us to put this tool together.
• We’ve tried to make the charts both Tufte-compliant (i.e. minimal chart-junk) and visually attractive (thanks to Google for the color scheme).
• Feel free to suggest other types of charts that you’d like to see in the Chart Chooser. Send an example to chartchooser@juiceanalytics.com.
• If you’d like a customized version of Chart Chooser for your organization, write us at chartchooser@juiceanalytics.com or call me at 202.251.7750.

What do you do when you’d rather be out driving your BMW rather than sitting in your corner office? Make a business dashboard that looks like your car dashboard, of course. You’ll want to have lots of tachometers, temperature gauges, and traffic lights. It’s the ultimate business-driving machine.

It isn’t controversial to complain about the ineffectiveness of “gauges” for data visualization. In fact, even some of the worst offenders admit that gauges aren’t ideal:

Dr. Robert Alison of SAS in showing off a new easy graph procedure for creating gauges says:

“I know, I know … gauges have lots of drawbacks in dashboards. But hey, the other philosophy is ’give the customer what they want’ … and try to make it work as well as possible. So, as far as gauges go, these are pretty decent.”

Here’s the example he uses to show off “one of the sharper-looking dashboards I’ve seen”

The folks at Business Object’s Xcelcius admit that gauges shouldn’t always be used in their article entitled “The Use (and Misuse) of Gauges”.

That doesn’t stop them applying a triple-coat of carnauba wax while neglecting their rule to always label the endpoints.

In the end, they primly note: “Despite some recent bad press, a gauge isn’t inherently a poor graphic.” Bad press, is it. If only gauges had better PR.

In my opinion, warning about potential misuse isn’t firm enough. Gauges shouldn’t be used except under the most severe threats from a client offering enough money to buy absolution.

Stephen Few, a man who doesn’t mince words on information visualization, says:

“If you squint really hard, you can barely make out some of the values. But who cares, because if you’re an executive who likes to pretend that you’re driving a car while sitting at your desk rather than actually managing your business, then having a dashboard that is truly informative doesn’t really matter.”

Charley Kyd says:

“Using dashboard gauges for management reporting typically is a mistake. Gauges hide information that managers need and consume significant space in a report.”

Let’s break down the problems with gauges:

Gauges hide trends. For all the focus on how a value is performing, you’d think people would care about the historical trend.

Circles aren’t good for showing differences. Like pie charts, circular gauges aren’t the best way to show size or changes in values—bars are a more straightforward, if less sporty, approach.

Space eaters. Often gauges are used to show a single value. All that decoration for a single value must send Tufte into a tizzy. Attempts to cram two values into a gauge can be confusing. How do you read this one?

Difficult to read. The values can be obscured by all the attractive accoutrement:

Ranges can be tricky. By the analogy to a car dashboard, gauges are expected to have a static minimum and maximum value. What happens when a value goes beyond the pre-set range. Here’s an example of the “right way” from Xcelsius with the label: “This gauge shows a retail store’s progress against a daily revenue target.” We can only presume the maximum value is $45,000. What happens if I go beyond $45,000?

Traffic lights are contradictory. I may be getting nitpicky, but I can’t both have my traffic light look like the real thing (red on top, green on bottom) and abide by basic data visualization assumptions (better is higher).

Lastly, there are so many better options. Here’s a beautiful data display (courtesy of Mr. Few) that could have been done with gauges, but mercifully was not.

“The last mile” is a term that often is applied in the telecom industry in reference to “the final leg of delivering connectivity from a communications provider to a customer.” It is an expensive and complex step due to the challenge of pushing information from centralized, high capacity channels to many diverse end-points where information is ultimately used.

We think there is a “last mile” problem in business intelligence too. This critical bridge between data warehouses and communication of insights to decision-makers is often weak or missing. Your investments and meticulous efforts to create a central infrastructure can become worthless without effective delivery to end-users. “But how about my reporting interface?” you wonder. That’s a creaky and narrow bridge to rely on for the last mile of business intelligence.

Listening to our clients, we are confident the last mile is a real problem. The ultimate source of this failure is less clear. Here are a few of theories:

1. The engineers who built the data warehouse build the interface. No offense to the talented individuals who can push around, clean, normalize, and integrate data—but they may not be ideally suited to designing a user interface for non-technical users. A designer wouldn’t create charts that look like this (our favorite example of chart-based encryption):

In the worst case, developers are dismissive of user experience. I’ve met with IT folks who felt confident that providing a massive data table would provide a suitable solution for delivering information to users. “Hey, they’re getting their data. Is there a problem?”

2. Reporting is considered the fundamental mechanism for working with data. Here’s a framework we’ve started to consider in thinking through the multiple approaches for getting value from data:

• Reporting lets you monitor things that are well-understood and relatively predictable.
• Exporation or analysis helps you understand new processes and erratic and shifting behaviors.
• Presentation is about communicating insights and understanding, often building on both reporting and analysis.

Many people assume that a reporting tool is sufficient to do in-depth analysis and communicate results. That’s like trying to build a deck with a screwdriver.

3. Poor fundamentals in information display. Despite the efforts of folks like Edward Tufte and Stephen Few, general literacy in this area is still low. Shiny, 3D pie charts are still acceptable, even desirable in some places. Particularly disturbing is the persistence and pervasiveness of this problem in Excel where there still remains some confusion as to why this is bad information display:

You don’t have to go any further than the Dashboard Spy to find examples of the visual muck that is commonplace.

Have you run into this problem: you have a list of phone numbers and associated values which would be best shown geographically to see patterns, but there isn’t a clear way to put the data on a map. Maybe you’d like to see a map of customer service calls by call duration or inbound sales by average order size.

I wanted to share how to MacGyver a solution with a piece of twine, bubble gum, Excel, and a free online map tool. To me, this is a nice testament to the simple but powerful data visualizations that can be accomplished without programming skills or expensive applications.

1. Pull out area codes

First I pulled the area codes from my list of phone numbers using the formula below. This simply checks if the phone number starts with 1, then grabs the appropriate three digits for the area code.

=VALUE(IF(LEFT(E7,1)=”1″,MID(E7,2,3),MID(E7,1,3)))

2. Convert area codes into states

For my purposes, mapping the phone numbers by state was sufficient. Ideally, we would map the phone numbers to precise latitude and longitude coordinates by doing a reverse lookup of addresses then using the Excel geocoding tool.

First I needed a lookup table that could link my list of area codes to states. I wasn’t able to track down a good data table, so I grabbed the data from All Area Codes and cleaned it up. Here is a lookup table of area codes by state.

An aside: I have a pet peeve with people who sell data that feels like it should be publicly available. You’ll run across these businesses when looking for basic information about ZIP codes, MSAs, or area codes. Here is an example of one of these parasitic businesses.

3. Create your summary data set

I used a pivot table to summarize metrics by state.

4. Create colorized map of the US

Our friend Ducky Sherwood has generously put together a online tool called Mapeteria that will generate a colorized overlay of US states. In Ducky’s words: “Want to make a choropleth thematic map (i.e. coloured based on your data) for Canadian provinces, U.S. states, or French départements?” This overlay can be viewed in either Google Maps or Google Earth.

Here’s where it gets a little tricky. You will need to provide Mapeteria with a URL to a properly structured CSV file. Posting a CSV file to a web server isn’t trivial if you aren’t running your own web site. I found one free service called FileDEN that did the job (other suggestions?). Beware all the advertising—and in all likelihood they immediately sold my e-mail address at registration. Nevertheless, you can upload a file here and it will give you a URL which can be used to create your map.

Here’s an example of the results: