Juice Analytics

via Dashboard Spy

Real-time dashboards — the kind that show up on a big screen in a call center — are an entirely different beast than your standard management dashboard. Their job is to support immediate decision-making. As a result, the information must be easy to interpret, alert users to problems, and make the next action obvious. In addition to key success metrics, real-time dashboards may show detailed data about the action “on the ground.” Here are eight characteristics that can make a real-time dashboard effective:

1. A summary status that indicates how things stand overall. Users need to be able to tell at a glance whether they should worry or not. Here’s a great example from the folks at Superblock. The “Is it going to rain?” site tells you the single most important thing you need from a weather report.



2. Reflect a well-understood structure of the business. By the time you design a real-time dashboard, you should have a strong theory for how the pieces of the business fit together (i.e. the relationships between key measures, drivers, and available actions). For example, in the call center business, there are clearly defined success measures (e.g. wait time), a mathematical relationship between these measures and their underlying drivers (e.g. call volume), and known levers to address problems (e.g. staffing levels).

3. Support quick diagnosis of problems. The data presentation should point directly to the likely source of the problem. Real-time dashboards aren’t the place for deep analysis or introspection into the drivers of the business.

4. Simple data presentation. In my view, real-time dashbaord’s aren’t the place for complex or advanced data visualizations. Imagine you were Napoleon and you had to use a half-completed version of this chart to make a battlefield decision in the next 5 minutes.



5. Granular view of the “unit of action.” Real-time dashboards are often about tracking activity. It may be useful to show the raw data around these events in the form of a ticker, scroll or RSS feed. We use at a real-time tracker for our website called Sitemeter. It does a nice job of tracking the basic unit of action — visitors.



6. Appropriate time window. Getting time right on an operational dashboard is critical. If the measures and trends represent too long a time period, users may not react to changes quickly enough. On the other hand, very small time windows encourage frantic reactions to changes that may not represent real trends. Ideally, the dashboard should offer the ability to configure this time range and “freeze” a moment in time.

7. Prominent but balanced alerts. Naturally, alerting users to problems is a central mission for real-time dashboards. The challenge (as always with alerts) is to balance between “the sky is falling” hysteria and “don’t worry, be happy” apathy. I’ve written before about alerts, but one item to emphasize is the need to show a sense of relative importance. Not all problems have the same impact on the business, and finding a way to communicate this relative importance is valuable.

8. Point to specific action. If real-time dashboards are about identifying and responding to issues, the tool should point users to what they can do about a problem. This can be as simple as displaying the phone number of the right person to call.

Real-time dashboards can be ignorable, create mayhem, or drive great behavior in an organization. Thinking carefully about the design and functionality will make a huge difference.

A couple years ago we released our first baby dashboard design. I’ll admit it was a bit rudimentary. It tracked only the most basic measures and offered little insight into your baby’s current mindset. I was a new father and had a relatively superficial understanding of the nuances of babies, not to mention actionable baby metrics.

With the arrival of my second child, I set to work designing a dashboard that would give a parent all the important information they need, presented in ways that let them react to baby data even in a harried household. Let me present the prototype of our new Baby Dashboard 2.0, modeled by my daughter Maya.

We use the same heads-up display technology as in our first release, but now with more sophisticated data collection techniques we’ve included a meltdown prediction chart and real-time translation engine.

There are a few features in here that I believe demonstrate important fundamentally design principles for great Information Experiences:

• Choose metrics and information that a user can act on. Information that is just interesting isn’t worth a random pile of ones and zeros. You need information that you can act on. In BD 2.0, we wanted to deliver news you could use, in the moment. The “meltdown fuse”, for example, is a way to measure how close your baby is to freaking out. As she gets tired, sick, or hungry, her fuse shortens to the point that a simple disruptive act — a loud noise, Mom walking out of the room — will set off a meltdown. You need to know how close you are to this threshold so you can minimize the smallest of disruptions.

• Draw attention to the information that is most urgent. While the dashboard provides detailed trend breakdowns, the most important thing for a parent is the current state of things. The top bar of the dashboard answers the most critical questions always on a parent’s mind: 1) How close is my baby to melting down? 2) Does my baby need any of the basics: food, sleep, or clean diaper? 3) What is my baby trying to say to me?

• Progressively reveal data as the user expresses interest. Like a busy executive, a parent doesn’t have time for all the information at once. They are on a need-to-know basis. If a parent needs to get a better sense of the potential meanings of a baby word (“daaah”), a single click will give a breakdown of the most likely interpretations.

• Different views for different audiences or perspectives. BD 2.0 provides distinct views for baby status and parent status. The parent status (not shown) was added because we recognized that the mental state of the parent was as important to a happy child as a clean diaper.

For those of you who expressed interest in licensing our Baby Dashboard 1.0 technology, please be patient while we work out the bugs in this next release.

Organizations have a personality, and it bleeds into everything from executive reporting to product offerings. A recent Fortune article entitled Microsoft without Gates offers this wonderful tidbit about Steve Ballmer, CEO of Microsoft:

Even though he never was a serious computer programmer, by all accounts Ballmer is just as good at math as Gates is. He lives and breathes data. “Steve has a computer in his head,” says Bob Muglia, a 20-year company man who heads the Server and Tools division. Ballmer expects his subordinates to be adept in math as well. He distributes 11-by-17 sheets filled with numbers detailing the progress of various operations. The numerals are so small that executives use transparent magnifier rulers to see them. But there are never any columns showing percentage changes. Ballmer believes people ought to do that in their heads. It saves space on the paper for more numbers.

Wow. If it is as bad as the author describes, Ballmer has designed the anti-dashboard.

The Presentation Zen blog offers another great example of organization culture as displayed in business artifacts:

Gates here explaining the Live strategy. A lot of images and a lot of text…Good graphic design guides the viewer and has a clear hierarchy or order so that she knows where to look first, second, and so on. What is the communication priority of this visual? It must be the circle of clip art, but that does not help me much.

Does it get more “Zen” than this? “Visual-Zen Master,” Steve Jobs, allows the screen to fade completely empty at appropriate, short moments while he tells his story.

The tendency with reporting, and information dashboard design in particular, is to cram as much information on the page as possible. It is a problem that Avinash describes with typical candor:

“This one of the core reasons why most dashboards are ’crappy’, i.e. they are data pukes that provide little in terms of context and even less in terms of actionable value.”

In the past, we have offered tools to make data presentation as clear as possible (chart chooser, Excel chart cleaner). Sometimes clean isn’t enough; a more dramatic approach is needed.

One alternative is to shift the focus from the full data to changes in the most critical data points. By pulling out the important exceptions, you can make it easier for your audience to digest what matters and take action.

Stephen Few says in his book Information Dashboard Design:

“The best way to condense a broad spectrum of information to fit onto a dashboard is in the form of summaries and exceptions…given the purpose of a dashboard to help people monitor what’s going on, much of the information it presents is necessary only when something unusual is happening; something that falls outside the realm of normality, into the realm of problems and opportunities. Why make someone wade through hundreds of values when only one or two require attention? We call these critical values exceptions.”

Alerts are one mechanism to turn the focus to the exceptions, outliers and data highlights. Whether embedded in the dashboard or presented separately, alerts can be the extra layer of abstraction that make a dashboard useful. Unfortunately, they are hard to get right. I’ve arrived at four C’s for effective alerts—context, cogency, communication, control. Here’s a checklist to consider as you build alerts into a dashboard or report:

• Are the parameters well defined? An alert is commonly defined by the following factors: metric (e.g. revenue), dimension (e.g. time), delta (e.g month over month change), scope (e.g. Northeast region, Peanut-product line), threshold (e.g. increase or decrease of 10%).
• Is the timing of the alerts actionable? One client explained to us that fluctuations in many of their metrics make monthly alerts too frequent—it would unnecessarily alarm people when, from their perspective, no significant trend had been established.
• Is the change statistically significant? This is of particular importance when you are measuring deltas. A doubling of traffic from a referring site doesn’t mean much when it is moving from one to two visitors.

• Can the alerts be described in simple terms that even an executive can understand? Alerts should have a real-world meaning that users are familiar with. If an alert is based on a complex metric, for example, users will be confused as to the implications.
• Is the alert actionable? In the best cases, alerts should point users to both the drivers of the alert and the actions that can address the situation. This system does neither:
  ![terror warning system]
• Are the alerts so granular and/or frequently triggered that users will get alert fatigue? Excessive use of alerts will undermining their credibility. We saw this happen at one client where an IT-designed system threw off alerts like they were going out of style. The application went out of style the next year when users decided it was more distracting than useful. Here’s another example of a system that seems designed to raise blood pressure.

(It appears that a 5% increase in brand attribute performance isn’t good enough to get you out of the yellow.)

• Is the alert placed in context? Google Finance does a nice job of putting news alerts within the stock chart.
  
• Is it clear what the user should do next? Give the user a clear path to more information so they can understand the full context of the alert.
• Does the sophistication of your alerts match the sophistication of your audience? I’ve found that it is better to start with some simple alerts so your audience can begin to learn what they mean and how to react. Over time, these alerts can become more refined and focused to capture complex situations.
• Does the alert draw the eye without being visually overwhelming or annoying? Here’s a article about how to “reduce visual noise” in dashboards.
• Is color used appropriately? Red means bad. Yellow is sorta bad. Green means good (but “good” things don’t need to be alerts). It isn’t particularly fair for color blind folks, but these conventions are deeply rooted.
• Have you found the best mechanism for presenting alerts? Alerts can be sent through e-mail, as SMS message, blasted over the office intercom system, or posted to the wall in the bathroom. What is the most convenient and appropriate medium?

• Can the user identify the important alerts for them, and avoid the others? As hard as you may try in designing the dashboard or report, you aren’t in the shoes of the users. They will learn what they want to pay attention to and what information is extraneous.
• Can the user adjust the parameters? With more sophisticated dashboards, you want to give users the ability to adjust parameters to hone in on the exceptions that really require action.
• Can the user analyze alert frequency and trends? I’ve never seen a system that does this, but having the ability to view and analyze alert history seems critically important to getting a holistic view of performance.

What do you do when you’d rather be out driving your BMW rather than sitting in your corner office? Make a business dashboard that looks like your car dashboard, of course. You’ll want to have lots of tachometers, temperature gauges, and traffic lights. It’s the ultimate business-driving machine.

It isn’t controversial to complain about the ineffectiveness of “gauges” for data visualization. In fact, even some of the worst offenders admit that gauges aren’t ideal:

Dr. Robert Alison of SAS in showing off a new easy graph procedure for creating gauges says:

“I know, I know … gauges have lots of drawbacks in dashboards. But hey, the other philosophy is ’give the customer what they want’ … and try to make it work as well as possible. So, as far as gauges go, these are pretty decent.”

Here’s the example he uses to show off “one of the sharper-looking dashboards I’ve seen”

The folks at Business Object’s Xcelcius admit that gauges shouldn’t always be used in their article entitled “The Use (and Misuse) of Gauges”.

That doesn’t stop them applying a triple-coat of carnauba wax while neglecting their rule to always label the endpoints.

In the end, they primly note: “Despite some recent bad press, a gauge isn’t inherently a poor graphic.” Bad press, is it. If only gauges had better PR.

In my opinion, warning about potential misuse isn’t firm enough. Gauges shouldn’t be used except under the most severe threats from a client offering enough money to buy absolution.

Stephen Few, a man who doesn’t mince words on information visualization, says:

“If you squint really hard, you can barely make out some of the values. But who cares, because if you’re an executive who likes to pretend that you’re driving a car while sitting at your desk rather than actually managing your business, then having a dashboard that is truly informative doesn’t really matter.”

Charley Kyd says:

“Using dashboard gauges for management reporting typically is a mistake. Gauges hide information that managers need and consume significant space in a report.”

Let’s break down the problems with gauges:

Gauges hide trends. For all the focus on how a value is performing, you’d think people would care about the historical trend.

Circles aren’t good for showing differences. Like pie charts, circular gauges aren’t the best way to show size or changes in values—bars are a more straightforward, if less sporty, approach.

Space eaters. Often gauges are used to show a single value. All that decoration for a single value must send Tufte into a tizzy. Attempts to cram two values into a gauge can be confusing. How do you read this one?

Difficult to read. The values can be obscured by all the attractive accoutrement:

Ranges can be tricky. By the analogy to a car dashboard, gauges are expected to have a static minimum and maximum value. What happens when a value goes beyond the pre-set range. Here’s an example of the “right way” from Xcelsius with the label: “This gauge shows a retail store’s progress against a daily revenue target.” We can only presume the maximum value is $45,000. What happens if I go beyond $45,000?

Traffic lights are contradictory. I may be getting nitpicky, but I can’t both have my traffic light look like the real thing (red on top, green on bottom) and abide by basic data visualization assumptions (better is higher).

Lastly, there are so many better options. Here’s a beautiful data display (courtesy of Mr. Few) that could have been done with gauges, but mercifully was not.

Misaligned goals, distorted behaviors, and a misguided sense of success… no, I’m not referring to college graduates. I’m talking about the problems caused by using the wrong metrics in your organization. You’ve probably seen examples like tracking average customer profitability and losing perspective on the variance in profitability or evaluating customer service reps on calls handled without regard for the quality of the experience. I’d like to offer up a quick-bake recipe for choosing the right metric.

Step 1: Set the context

Metrics generally serve one of two purposes. Start by understanding what you are trying to achieve.

1. Identifying problems. Defining the right metrics in this case requires you to do a little detective work: What is the data residue of a problem? What evidence can be found and how exactly does it show up?

2. Measuring performance. The right success metrics need to focus on measures that can be controlled and where improvement in the number is unabiguously a good thing.

Step 2: Balance the four dimensions of a good metric

Lots of metrics fail in at least one of these dimensions. A few examples:

• Common interpretation: We had a client who made a distinction between “leads” and “prospects” in their marketing organization. Prospects had theoretically expressed more interest in the service through their actions. Unfortunately the line between leads and prospects was always hard to decipher and the definitions were hard to communicate. On a related note, we got a kick out of Tom Davenport’s (author of “Competing on Analytics”) assertion that a company competing on analytics needs to “invent proprietary metrics for use in key business processes.” There is nothing inherently wrong with “invented proprietary metrics” but it sounds like something that is designed to confuse anyone outside of the inner sanctum.
• Actionable: Metrics are frequently too broad for the impact that a particular group can have. Customer satisfaction is a popular dashboard staple, but it is hard for most managers to see how they can have a significant impact on the number.
• Accessible, credible data: Sometimes the most valuable and obvious metrics are frustratingly hard to track. In the web analytics world, unique visitors is important to know, but user deletion of cookies has thrown a wrench into the works.
• Transparent, simple calculation: Top NFL agent Leigh Steinberg says of the famous quarterback ratings metric:”Other than one attorney in our office, I am unaware of a single human being who has the capacity to figure a quarterback rating.” I don’t know what kind of art majors he hires, but all they need to do is use the simplified formula: (83.33 * Comp %) + (4.16667 * Yds per att) + (333.333 * TD pct) – (416.667 * INT pct) + 25/12.

(Want a little validation of this framework? Avinash, respected web analytics guru, just published a post with “Four Attributes of Great Metrics” and he landed on a strikingly similar set of four: 1) instantly useful (i.e. actionable); 2) relevant (i.e. common interpretation); 3) timely (i.e. accessible); 4) uncomplex (i.e. transparent and simple).)

Step 3: Avoid the metrics bugaboos

Finally, here are a few traps that I’ve seen in deciding on appropriate metrics:

• Trending and distributions: Don’t always try to compress a metric into a single number. Often it is more revealing to show the metric across time or as a distribution to uncover variance.
• Edge cases: There will always edge cases where a metric may not mean what you think it means. These situations are worth understanding, but you shouldn’t allow the perfect to be the enemy of the good.
• Setting goals: Could you hold someone accountable for this metric without them throwing out a half-dozen reasons why it doesn’t make sense? It’s a decent test of the value of the metric.
• Self-serving: Be careful that you don’t select metrics simply because you know they’ll make you look good.

Charts are a great way to explore data. Here is some American baseball data showing player salaries over a five year period.

Charting this data with a line chart would allow us to see trends in salaries by team. However, when we use Excel’s default chart, we get something that looks like this:

That’s quite a mess. It would be a lot easier if we could create one chart for each row.

The OFFSET function is going to help. In its simplest form the OFFSET function works like this:

OFFSET(anchor, rows from anchor, columns from anchor)

That is, OFFSET will start with the anchor cell, go down a number of rows from that anchor and over a number of columns and return the value it finds.

We can use the OFFSET function to create cells that pull a single row of data out of the table dynamically. We create a new row atop of our data and create a series of OFFSET functions that all rely on a single cell (the big yellow one) for their row offset. So changing one cell will pull different rows of data into our fixed location.

Now, chart the data that doesn’t move.

After fixing the chart, we’d like to make it easy to change the value in the big yellow cell.

We can use Excel Forms to build a lightweight user interface. Bring up the Excel forms toolbar by rightclicking on any toolbar and choosing Forms. Place a scrollbar beside the chart.

Right clicking on the scrollbar allows you to Format Control. Link the control to the cell that is controlling all the row offsets. Now, moving the scrollbar will update the chart.

Now, the scrollbar controls the chart. Here is the baseball spreadsheet for you to play with: Baseball_offset.xls Have fun!

Note: this post is adapted from a presentation I gave at eMetrics 2007 in San Francisco.

Everyone wants a dashboard and the promise of a world in which the intricacies of your business are clearly laid out on a single page. Dashboards can make running your business as easy as driving a car, where slight adjustments and careful attention to warnings mean smooth sailing on the road to success.

I’m not so convinced. For someone who is, check out the mysterious Dashboard Spy. He/she has a massive collection of dashboard screenshots and describes these precious morsels as “simple to understand and impressive to look at, these scorecards are becoming ’must-haves’ for all enterprises.”

If we already live in a dashboard-centric world, we might as well do them right. I see at least three areas where dashboards need improvement: depth, information display, and storytelling.

Depth. Stephen Few makes a worthwhile distinction between dashboards and something he calls “faceted analytical displays” (FADs):

• A dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.
• A faceted analytical display is a set of interactive charts (primarily graphs and tables) that simultaneously reside on a single screen, each of which presents a somewhat different view of a common dataset, and is used to analyze that information.

We might consider dashboards a static version of FADs (or we could consider FADs a versatile dashboard). If that’s true (and I’m sure Stephen will step in to correct me), then who wants a plain dashboard? Why build something that only raises questions but doesn’t give the user any ability to drill down, explore, tweak parameters, or otherwise try to answer those questions?

Information display. Like most reporting, dashboards suffer from poor information design. Here’s our list of blogs that preach the right way and highlight the offenders. Here are two particularly misguided design approaches that I’ve seen recently…

Just because it is called a dashboard doesn’t mean you need to take the concept literally (via Dashboard Spy)

Just because you can make it shiny doesn’t mean you should. Crystal Xcelsius not only vigorously embraces pie charts, but they add a “reflective kidney bean” to further derail the information display.

Storytelling. Most dashboards are loose affiliations of charts—a hodgepodge of graphics on the same topic intended to offer a full view of a situation. It is the same problem so many people run into in creating href="http://www.juiceanalytics.com/writing/2006/03/presentation-checklist-always-simplify-never-screenbean/">PowerPoint presentations.

You want the information to easily slide into the viewer’s brain and stick when it gets there. The best dashboards have story-like features such as:

• Set the stage. What is the context? Who are the characters?
• Focus on only the important elements and themes; don’t try to be a comprehensive account of everything that happened. Ruthlessly cut extraneous content.
• Offer recognizable characters to spare the reader’s precious attention. There is a high cost to asking readers to learn from scratch. For dashboards this means terms, metrics, graphics, and metaphors that are familiar within the organization.
• Create flow and cohesiveness from chapter to chapter. Themes and characters reappear chapter after chapter. A good dashboard isn’t a bunch of disjointed charts, but a logical flow from one analytical examination to the next.
• Levels of detail. Some elements of the story span the entire experience; other details provide the insights and seasoning to keep your interest.

Here’s a good example of a dashboard (perhaps FAD) from Visual I-O that has many of these storytelling elements.

In contrast, the following dashboards (courtesy of Dashboard Spy) don’t attempt to explain anything to the reader:

If you’ve seen a worse dashboard, sent it our way and we’ll put together a gallery of the worst of the worst. Please redact any company-specific information.

We all know at least one GUI Jock. That one guy who knows how to, say, run a complex query on the content management system, or export data from the annoying sales database front-end or actually get new data into what qualifies as “the system” where you work. He is a master of tools that appear obscure, but are in fact just a pain in the neck. He is not writing firmware for the space shuttle; he is changing the background gradients in your marketing dashboard.

The GUI Jock is a paradoxical figure. Indispensable and yet undervalued, he owes his livelihood to the ferocity of the beast he tames. The sheer number and complexity of pull-down menus, check-boxes, obscure options, software bugs, and poor user interface choices created by an external software vendor. The GUI Jock conquers them all—he is a human compiler who receives requests in the loose and informal language of the outsider and compiles them to the standards demanded by expensive enterprise software.

But how did he find himself in this position? Ironically, he may have fallen into this unfortunate role by being good at a few ad hoc requests which he likely completed under the assumption that he would soon be moving on to more interesting work. But now he is stuck in a trap that he helped build and of which others are afraid. He is there to fall on the grenade that is lousy software, poor documentation, and bad process so the rest of the organization can go about its job without another hassle. The GUI Jock suffers so we do not.

What can be done?

In my experience the GUI Jock is usually not happy with his lot. If you know him you are probably aware that he can be a grouch and he has probably sighed in your presence more than once (if you don’t know him, he might be you). But can we set him free?

A typical response is training. Grab a conference room for a few hours, set up a projector and show the junior staff just how to hold that chair while taming the beast known as the “InsiteDynaMetrix CollaboStream(tm)”. The juniors sit and nod, happy to have such a big block of their day accounted for. In my experience, the success rate of this approach is woefully low. It can backfire, basically serving to train attendees to know who exactly the GUI Jock is and that they should funnel all relevant requests directly to his inbox.

To protect itself, the organization demands that the GUI Jock stay in his role. He is the only person who will save himself. He has a few options:

• Sucker a new employee into the role. New employees are eager to please and crave the recognition of value that comes with being a GUI Jock. They are also too naive to see the quicksand.
• Increase the friction for people who lean on him. Ask for forms to be filled out, demand detailed requirements, and delay in delivering results. With enough process, these people may decide to serve themselves.
• Apply to graduate school.

prairieFyre Software, a provider of contact center solutions, has created a reporting tool that takes a table of data and encrypts it in chart form. The original numbers and trends are virtually unrecoverable. Congratulations, prairieFyre, for this exciting new approach. This may be patentable, but I’m afraid there is prior art.

Beat that, Junk Charts.