segmentation

Survey Analysis Grows Up with SurveyVisualizer

Luc Girardin of Macrofocus contacted us in response to our post "When Will Survey Analysis Grow Up?" to point us to their SurveyVisualizer analysis tool. I had a little time this weekend to download and play with this application. There is a lot to like.

SurveyVisualizer is designed for surveys that have a hierachical or tree structure. Luc describes the relevant data structure in a background paper about the product:

The questions—also called quality criteria—are then aggregated into 23 quality dimensions (e.g. network quality, ticketing, cleanliness, security, reliability). They represent the level of satisfaction with a whole group of questions pertaining to a particular issue. The quality dimensions themselves are further aggregated into three different customer satisfaction indices, reflecting the different areas of responsibility.

The free download has multiple satisfaction "criteria" (e.g. friendliness of crew) roll up to "dimensions" (e.g. cabin crew) which fall under "indices" (e.g. index of flight services). This may be an appropriate structure for a satisfaction survey—but it isn’t one I’ve encountered before.

Despite this limitation, the analysis capabilities delivered by SurveyVisualizer are intuitive and innovative. For example, all your survey data is displayed at once in a kind of relational map. This lets users visually identify patterns in the full set of results. Each of the vertical hashes represents a question or roll-up of questions. Clicking on any one of these hashes highlights the hierarchical relationships. The "ghost" lines represent the results across questions for a multitude of dimensions or respondent types.

SurveyVisualizer 1

Users have the ability to select specific dimensions to identify patterns in the corresponding results. An easy-to-use interface lets you choose a dimension then apply a color to the line within the relational map.

SurveyVisualizer 2

Also, users can click on individual display lines to investigate the results (e.g. I wonder who had that particularly crappy score for flight delays?)

If your analysis requirements don’t fit this particular structure, Macrofocus has a more general-purpose tool called InfoScope.

When Will Survey Analysis Grow Up?

Malcolm Gladwell at TED

At the 2004TED conference, Malcolm Gladwell tells the story of Dr. Howard Moskowitz, a man who revolutionized the prepared food industry through a new kind of analytical thinking. Long story short: Dr. Moskowitz was one of the first people to argue that companies should pursue multiple products targeted at customer subsegments rather than try to create the perfect product for all customers. He realized that an attempt to create a "platonic ideal" —whether it was pickles, mustard, or pasta sauce—would be a suboptimal result for most consumers. Consumers are individuals with preferences that are better clustered than averaged. Mr. Gladwell states that this change in business thinking (spurred by Moskowitz’s study of pasta sauce) mirrors a more general scientific shift from a focus on universal truths to the study of variation.

The prepared food industry gets it—as evidenced by nine variations of Ragu sauce on the grocery shelves—but I’m not convinced that these lessons have permeated the rest of the business analytics landscape. In particular, I am struck by the inability of most survey analyses to reveal insights about respondents.

The tools may be part of the problem. Here’s an example of what WebSurveyor provides its users to help them analyze online surveys:

Sample Survey Analysis

Their site tells us:

"Each question is graphed independently allowing you greater flexibility in customizing the layout of reporting for each question...Filter results based on specific responses or cross-tabulate results from two different questions, giving you powerful tools for detailed analysis."

Powerful? Flexible? More like barebones. WebSurveyor is putting the analyst in a very constrained box that won’t help deliver an better understanding of respondents. WebSurveyor’s tool demands "question-centric" not "customer-centric" analysis.

Consider how this typical survey approach would serve you in an effort to understand the passengers of Noah’s Ark. A surveyor would ask each animal to fill out basic information about their height, weight, number of legs, food preference, etc. The results would then let us know that the average animal weights 23 pounds, has a height of 1.2 feet, 5.6 legs, 30% omnivore and so on. All of which would miss the essential insight about the animals on board: there are two of each.

Unfortunately, the kind of analysis needed to reveal personality / needs / behavior clusters in your respondent population isn’t well supported by out-of-the-box analytical tools. One approach is factor analysis—a statistical technique that is used in marketing to "identify the salient attributes consumers use to evaluate products in a category" (Wikipedia). Another approach is to examine individual visual representations of individual respondents—a technique that we term (rather clumsily): customer flashcards.

Judge customers by behavior, not fur color

To a stranger, my two dogs look alike. To me, they couldn’t be more different. They came from different dog shelters and are more than two years apart (that’s 14+ human years). Here they are: Maddie has her chin resting on Ally.

Dogs

Ally is twitchy, a mama’s girl, frightened of loud noises, and getting creaky. Maddie is confident, independent, curious about the loud noise, and energetic. Ally loves other dogs and distrusts new people. Maddie adores all people and is suspicious around certain dogs. Their features and personalities couldn’t be more different. I’ve had some time to get to know them.

When we meet a stranger on a walk (particularly one who isn’t a dog owner), we often get: "They must be related." Our denials don’t seem to phase these people as they point to the obvious evidence: "...but they are exactly the same size and color."

Superficial judgements are natural - a first level of defense to categorizing and manage a complex world. However, it’s unhealthy to not try to dig deeper. For some businesses, superficial characteristics are as far as the analysis goes when segmenting or profiling customers. A better approach is to look at customer behaviors which provides a much more accurate reflection of interests and needs. Jim Novo, marketing consultant, agrees:

Customer behavior is a much stronger predictor of your future relationship with a customer than demographic information ever will be

Simple customer characteristics can be easy to come by; age, income, zip code are probably part of your basic customer database. In contrast, behavioral segmentation is a more initimating analytical challenge. Here’s the approach we’ve used successfully at Juice:

  1. Create individual pictures of customers that visually show their behaviors over time. The trick is to create a "visual language" that represents actions and is intuitive
  2. With a dash of Excel, SAS, and python code, we generate thousands of these pictures of individual customers
  3. We visually scan for common patterns of behaviors and the associated success/failure points (e.g. repurchase, upsell, churn, etc.)
  4. Finally, we work backwards from our new understanding of behaviors to segment customers based on statistical measures of behaviors.

This approach differs from traditional data mining-based approaches that drill down from the top looking for patterns. We start at a very granular level and looks for patterns (using the power of the human visual system). It may sound a little crazy, but we’ve found that it can be both insightful and highly predictive.