Too Literal with Numbers

Data analysts always face the same white-knuckle fear when they present: Will someone derail my presentation by questioning the source of a data point, the quality of the data, statistical significance, or why two numbers don't align?

These types of inquires may appear innocent and within the field of play. I find them counter-productive. They implicitly undermine the analyst's credibility and worse, deny the rest of the audience the opportunity to hear the full analysis narrative.

I've suffered through enough of these train wrecks to wonder about the underlying cause of this disruptive behavior. Here's my theory:

People who have experience and comfort with numbers have the ability to abstract meaning from analysis for themselves—even when numbers don't line up, the data is unclear, or the analysis has minor flaws. They can ask themselves higher-level questions: What does this mean? What are the implications for the business? How else could these results be interpreted?

In contrast, people who are uncomfortable with analytics treat numbers literally. They are disturbed by surface level inconsistencies. They expect—even need&mdas;hthe numbers to line up in straightforward ways. The medium simply isn't familiar enough to abstract their own meaning.

Combine this uncertainty with the omnipresent pressure in business to express an opinion (any opinion) to appear smart—and you end up with wasted time spend discussing superficial irregularities.

Does this sound like the rantings of a data elitist? Fair enough, but I'd suggest that a similar phenomena is common across any field of expertise. Take art, for example. People who have little exposure to fine art are bound to ask for help in explaining the underlying meaning, remark on the superficial beauty, and find themselves attracted to the most obvious reflections of reality. Meanwhile, those who have experience can gather their own meaning, think creatively about what they are seeing and understand how it fits in context with other examples.

If this notion is accurate, where does that leave us? On the one hand, it highlights the need to understand your audience and anticipate and educate those people who are inclined to be disruptive. It also hints at some of the poorly understood realities of analysis:

  • Analysis is as much art as science. Analysis relies on the perspective and skill of the analyst. Managers might want an objective reality, but they should recognize that the "truth" won't tell an actionable story.
  • Analysis should strive for directionally-correct results, not precision and comprehensiveness. Good analysis impacts decisions—therefore speed matters.
  • Give people something tangible and incontrovertible to hang onto. This is one reason we really like the bottom's-up analysis approach. When you create visualizations of individual customers, people can't argue with this granular data—and it gives them something that they can fully understand and appreciate.
  • Finally, there is nothing quite as valuable as personal credibility to make an analysis valuable. If people trust the work you have done in the past, they are likely to avoid arguing the small stuff.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

8 comments | Show all comments only the last 5 are shown


September 16, 2006
Paul Scaer said:

Could you explain more about the "train wreck" situations that seem to be so damaging? As a teacher, I find that the opposite is the case when students do power points: people rushing to conclude something for which they have no credible evidence. Do you let your audience think about your data?


September 16, 2006
Zach said:

The train wrecks usually go down like this: Someone starts by asking if the data being presented is "statistically significant" -- 1) not knowing what this means; 2) implying that the analyst is looking to pull a fast one with some sketchy data; and 3) redirecting the discussion away from a productive discussion of the meaning and results of the analysis. This question opens the door to a series of questions about survey design, how a particular metric is calculated, and speculation about edge case exceptions. Before you know it, half the meeting is over and no one has learned anything new. To me, these are questions that are best dealt with offline (before or after the presentation) -- not when you are playing with the time of the whole audience

"Do you let your audience think about your data" feels like a leading question. I grant you, there are going to be students and even analysts in professional environments that have done sloppy work or worse, are looking to deceive with data. Perhaps the better solution is to separate these discussions: set up a time to discuss data quality and analysis methodology that is separate from discussion of results and implications.


September 25, 2006
Daniel said:

There's always the old trainer's trick of the "parking lot," an easel pad where you write down questions that are temporally inappropriate with a promise to address them before the presentation ends. Everybody gets to hear the presentation without a lot of distracting side-trips, and you get to answer when they've heard the entire presentation in context. Also, you'd be amazed at how often people decide that their questions is no longer important once the presentation is over.

DGF


September 25, 2006
Zach said:

Great point. I really like the parking lot trick.


February 25, 2007
» Truth in Advertising - Juice Analytics said:

[...] This isn’t easy, we all have people who can drive us crazy, who can derail a presentation with niggling questions or who ask for information they’ll never use. [...]

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment