Paul Graham, from The Power of the Marginal

"Rising up through the hierarchy of the average big company demands an attention to politics few thoughtful people could spare... I think that's one reason big companies are so often blindsided by startups. People at big companies don't realize the extent to which they live in an environment that is one large, ongoing test for the wrong qualities."

A while back, we got all over Thomas Davenport for his checklist of ways to tell if your organization is an "Analytics Competitor." To me, he had posed the wrong question. It asks too much and reveals too little.

I don't need to know, for example, if I can chip like Phil Mickelson and finish off a tournament like Tiger Woods. I just want to be a decent golfer who isn't embarrassed during a corporate outing. Organizations don't need to know if they can go head-to-head with Harrah's (the new popular case study, displacing Capital One); they just want to be smarter about their decision-making using the data they have.

Creating a useful analytics capability requires a number of pieces to come together. Here's our take on the important things to evaluate as you consider: Where do I stand with my analytics and how can I get better?

Strategy

• For each business function or line, how will analytics impact decisions?
• What decisions will not be impacted?

People and tools

• Do I have people who understand the dynamics of the business (i.e. can pull their head of the data and see the context for an analysis)?
• Do I have people who are skilled in basic analysis approaches and tools? (e.g. PivotTables, simple modeling, basic statistics)
• Do I have people who can effectively communicate the results of their work through simple, attractive data presentation?
• Are there analyses or reports that I cannot accomplish with my current toolset because the data sets are too large or the statistical requirements to heavy?

Process

• Do I have a process for working with the business lines and functions to understand their needs?
• Do I have a process for defining, developing, producing, and delivering reports?
• Do I have a process for managing the ever-expanding queue of requests?
• Do I have templates for data presentation, reports, common analyses, models, etc. that will make my work more repeatable and efficient?

Raw materials

• Do I have a low-friction means to access the data that is the raw materials for my work?
• Do I understand the issues and intricacies of my organization's data? Have I documented it?

Integrate into business

• Have I proven the value of analytics through visible “wins”, i.e. real, live (and successful) cases where reporting and analysis is driving business decisions?
• Have I achieved a seat at the table, i.e. genuine involvement in decision-making process?

If I could influence the future of business intelligence tools (wait, maybe I can), I would put a premium on "tangible" data manipulation. I'd design interfaces that let users touch, play with, and sculpt data as an object.

Many data crunching applications, particularly those focused on statistics (e.g. SAS), tend to separate the user from the act of data manipulation. The user defines a set of scripts or formulas, points to a data set, and let's the application take over. For a programmer, this type of abstraction works. For non-technical business folk, it limits our ability to understand what is happening and why the result turned out differently than we imagined.

Here are a couple interesting examples of computer interfaces that attempt to merge real-world touch and feel with digital-world manipulation of information:

Via Information Aesthetics

What if BI interfaces brought an artisan's mentality (I'm imagining glassblower for some reason) to data manipulation? Data is the tangible raw material. When there was something odd or imperfect in the raw material, it would be obvious on visual inspection. We'd have access to a variety of tools, some for broad and crude actions, others for a more delicate and subtle actions. These tools would be put in physical contact with the data to shape it. Finally, we could add a final aesthetic finish to our creation. Analysts could take pride in creating digital objects that could move and influence others.

Related thought: can we blame the poor visualization of analytical results on the lack of visualization in the data analysis and manipulation process?

Business intelligence expert Claudia Imhoff of Intelligent Solutions describes the end-game for business intelligence in something she calls the "Corporate Information Factory" (plus an "e" for Extended). CIFE is a comprehensive ecosystems of people, processes, systems and applications to deliver all the promise of business intelligence for an organization.

Here's the problem: most businesses looking at this picture are going to feel intimidated and inadequate, like presenting to Steve Jobs (entertaining for Martha Stewart?). Below is my view of the common reality of business intelligence—it is a far simpler picture, lacking the governance, data management processes, multiple data marts, and overall data discipline.

Of course, CIFE probably wasn't meant for comparison; it is a long-term vision. Even so, the gap between reality and this beau ideal of BI may be too vast to be useful for many organizations.

The real questions: How do I get started in the right direction? How can I make better use of my business data without all the large investments implied by the CIFE model?

Over the next couple of weeks, we'll share our perspective on low-risk, practical actions with immediate impact. Here's some of what you can expect...

Step 1: Where do I stand?—Asking the right questions to start the journey

Step 2: Love the One You're With—Making the most of Excel

Step 3: The Learning Journey—Investing in analysis before reporting

Step 4: Just Say No—Reducing the pain and increasing the value of reporting

Step 5: Small Victories—Getting the organization to care about analytics

Here's the third (and hopefully final) in my series on interesting podcasts. This one comes from the folks at IT Conversations. It is a recorded presentation by Ray Lane from the Software 2006 Conference discussing the shifting landscape of the software industry. (Ray's slides are here.) Ray is the former president of Oracle and current partner at venture capital firm Kleiner Perkins.

About 26 minutes in, he gets to the part that intrigued me. He provides a description of what it is going to take to develop the successful and valuable enterprise software of the future. His thoughts resonated with ideas we've had about the weakness of existing business intelligence solutions and what we'd do to fix it. Here is my favorite quote:

Enterprise software industry made one big, huge mistake in the late nineties. It focused on buyers and forgot the users. [They need to ask] How are the users really using the software?

When it comes to new software opportunities, he recommends that new entrants:

1. Target the white space. "Find the white space, the open space, the stuff that hasn't been done. Don't try to do it better than Oracle, Microsoft, SAP. Enterprise software hasn't done everything. There is still a lot of manual decision making, a lot of manual effort, a lot of cost that goes in."

2. Low effort improvement. "Improve what they have today but do it low effort." He mentioned an example of an innovative company that could install its software in a week.

3. Free now, pay later. "Free so i can try it and actually see the value very quickly, then pay for it later. Trust that the customer will see enough value that they will pay for it later."

4. Generate individual value. "We make different technology decisions at home that at work. Why? because there is something that says: the enterprise is bigger than us. Take the [Amazon.com, Google] mindset into the enterprise."

I'd add one more item to his list:

5. Solve specific problems. Too often, enterprise software is built to be comprehensive and generic so it can marginally solve any problem. Better to flexible and modular to allow rapid implementations that target the biggest pain points or opportunities for individual client situations.

Yesterday, Google rolled out new mapping features for Google Earth and Google Maps. Many of these features are behind the scenes in the APIs, but there are great new capabilities that you will start to see. One thing I'm excited about is that KML—Google Earth's format for building sophisticated map overlays—has come to Google Maps.

Google demoed this at their Geo Developer day yesterday using one of our Google Earth overlays that shows US census bureau data by county mapped as a heatmap. It looks like this.

Counties are displayed in a list on the left. When you click on a county, you get a nice popup showing statistics for that county.

There are a few limitations. Large KML files don't load in Google Maps, medium-sized files load very slowly—it seems Google is parsing the KML using Javascript.

The mapping toolkits provided by Google Maps, Google Earth, and Yahoo Maps beta are well on their way to becoming important business tools once developers figure out how to wire in your enterprise data.

Without further ado, here are some US census data maps for you to explore in Google Maps.

Population Density

Lighter is higher population density (white is 800+ people per square mile), Dark is lower population density (black is 2 or fewer people per square mile)

Median Age

Lighter is older median age (white is 46.0 years median age), Dark is younger median age (black is 29.0 years median age)

Male/Female Ratio

Lighter means more men than women (white is 55% men), Dark means more women than men (black is 45% men)

Science and art have in common intense seeing, the wide-eyed observing that generates empirical information. Beautiful Evidence is about how seeing turns into showing, how emipirical evidence turns into explanations and evidence presentations. The book identifies excellent and effective methods of presenting information, suggests new designs, and provides tools for assessing the credibility of evidence presentations.

Edward Tufte's Beautiful Evidence, is now available for purchase. This is the labor of years of writing, thinking, and discussion. The book is physically beautiful--cloth-bound, five-color printing--and has the physical and intellectual heft to beat down opponents of good design.

Tufte covers the following topics.

I'm particularly interested in the first five chapters, particularly in The Fundamental Principles of Analytical Design. The last three chapters seem to range over Tufte's personal hobby-horses--he is, after all, a sculptor.

His critique of PowerPoint should be well known and raises good issues. I wish, however, that Tufte took on more than the critic's role in his discussion of the poverty of some of Microsoft's products. People need more than rejection of the tools they have. Business people aren't going to make graphs in Adobe Illustrator, they need to make them in Excel. People need the tooling that turns Excel (or PowerPoint) into effective tools for information communication.

Zach and I grew up in Lincoln, Vermont, a town of 900 people tucked away in the Green Mountains. At the center of this no-stoplight village is a general store. Vaneesa, the proprietor for more than three decades, greets her friends and neighbors at the counter everyday. She has grown to know each of their habits and needs and can tailor her stock and service in response. Everyone in town appreciates it.

This type of customer intimacy has long been lost as companies scaled beyond personal relationships. In an attempt to rebuild this bond, companies pile customer data – a digital representation of customers – into customer relationship management and business intelligence databases. Storing this information does little to get your business closer to understanding customer needs. Traditional data analysis falls short by aggregating behaviors and depending on the business to ask the right question. Surveying, another approach to staying in touch with customers, is hampered by customers’ imperfect knowledge of their own needs and by their limited memory of their own actions.

We discuss a way to solve this problem in an article on the Business Intelligence Network published today.

There aren't many opportunities for business analysts to share their expertise and learn from the best. Here's one. Check out the 2006 Data Visualization Competition sponsored by the Business Intelligence Network. The data's available in a Excel spreadsheet. To win, you need to find clear and complete ways of showing the data to solve real business problems. It's very down-to-earth.

This is a chance to share your skills and improve the state of our profession. I think the winner gets, errr, a book, and glory, lots and lots of glorious glory. And adulation.

A little birdie told me that the Juice Analytics census data heatmaps were used at Google's Developer Day to show how Google Maps can now load Google Earth KML files. Very cool.

Google Earth KML files now have two important user interface features that I'm excited to try out. First up is progressive display of data. This means a KML file can show high level summary info when when a user is high above the earth and seamlessly show more detail as the user zooms in. This was only possible through network links in the current version of Google Earth and this will feel a lot more polished to users. The other important UI feature is folders can now support radio buttons (where only one thing can be selected at a time). The big deal here is it allows a user to explore points organized into multiple dimensions where you can only view a single dimension at a time. For instance, you might want to view your customers grouped by sales volume, types of products purchased, or industry. Choose which of these groupings you want to see and the others will be hidden.

Finally, viewing KML files in Google Maps is a potential home run. This increases the sophistication of what Google Maps can display and simplifies rollout of geographic information to an organization. Bravo, GE folks.

Below is an interesting exchange by the guys at RedMonk from one of their wide-ranging discussions (Episode 10, subscribe here):

Stephen O'Grady: "[We are often asked,] you give so much away, how do you make money? Particularly in a business that is predicated on authority and ability and relevance, the notion that giving content away for free somehow precludes economic opportunity..." [here O'Grady's audio cuts out]

Cote [Trying to complete Stephen's sentence]: "...is stupid"

James Governor [Trying to fill in for Stephen]: "...is horseshit"

Cote then goes on to talk about how success, whether in software or music, is all about building relationships that you can eventually find a way to monetize.

James Governor: "We can just keep giving it away, and we'll still have more insights. It's not about content, its about contacts. It's about the relationships and the community.

Stephen O'Grady [Quoting someone from Sun -- maybe Jonathan Schwartz]: "We are betting on what Sun can deliver on in the future, not what we delivered in the past...

"...If you try to...lock down only what you have at the moment, I think that misses out. Basically, at Redmonk, our argument is that you can certainly read some of our thinking for free, but you aren't going to be able to leverage the future thinking until you engage with us. The things we will come up with you in the future...those are the kinds of economic opportunities that are absolutely not threatened by releasing our content for free.

I like this concept of a relentlessly future-focused business. It is a model that forces innovations and puts the client relationship, not the business transactions, at the center of the strategy. Giving away slightly-aged insights or capabilities goes a long way in building goodwill with prospective customers. It also demonstrates confidence in your abilities. In the words of our new friend Henk van Ekelenburg: "In my experience, consultants who are secretive usually want to hide their lack of skills."

BusinessWeek put out an interested podcast interview with Angela Shen-Hsieh, CEO of Visual I|O as part of its series on 10 Cutting-Edge Designers. She describes her company as "allowing people to get insights or understanding or perspectives from information." (Hey, we do that too!) From what I've seen, Visual I|O has done some innovative things with presenting data. Check out this demonstration of their capabilities--a highly interactive tool to help a baseball manager decide if they should pull a pitcher from the game:

But enough advertising for a competitor. I thought Ms. Shen-Hsieh made some interesting points about the state of the business intelligence industry. For example:

BusinessWeek interviewer: "Do you find that you have to do some education with your clients to convince them that design, information design, visualization techniques are really a valuable tool?"

Ms. Shen-Hsieh: "I feel that is a problem in our industry in that the technical challenges have been so fierce, there have been so many of them, that the focus has been mainly on in the IT world on the collection of the data, the storage of it and the access of it. This next frontier is really at getting at the last 18 inches between the screen and your brain. So we don't really talk too much about design, we talk about the problems and how this addresses the problems."

Well put. We've written before about the failure of business intelligence to live up to promises.

I do take issue with Ms. Shen-Hsieh when she dismisses Excel as a tool for visualizing information (and the blatant set-up by the interviewer):

Interviewer: "Excel spreadsheets and pie chiarts just don't seem to be up to the challenge of solving corporate problems today, in part because they can't address all the complex parameters that some of these decisions involve. Is the static nature of those visualization tools also a problem?"

Ms. Shen-Hsieh: "I think the lack of interactivity is the problem; I think the lack of business-focuses, problem-focused, solution-focused visualization is the problem...In the most simpliest terms, if you think of Excel, you've got an x-axis and y-axis, so you basically have two dimensions of data that you are looking at at any one time. But a complex problem is going to have lots of dimensions that you need to consider. That's what visualization gets you. It gives you a much broader concext besides a flat, tabular, x-y.

It isn't fair to say Excel can't do more than two axes. There are many ways to show multiple dimensions, make Excel dynamic, even visually compelling. Granted, it doesn't come out of the box this way--but any visualization requires some work.

Dear Microsoft Excel team,

In the last all hands meeting, you probably heard that I’ve been temporarily assigned as program manager of Microsoft Excel 2007. Granted this is an extraordinary action, but these are extraordinary times, my friends.

In the spirit of openness, I will tell you we will be making some specific changes to the product. We must work quickly as my time here is limited. I’m not even an Microsoft employee and I'm sure HR will have some say in the matter once they figure out what's going on.

In my limited time we will improve one specific thing, Excel's charting and data visualization functionality. People live and, unfortunately, die on infographics and frankly, the situation in Excel 2007 right now is deeply...busted.

We will improve the situation by working on five projects:

Project 1: Improve the defaults: Making a basic chart in Excel with default settings is easy but the results are poor from a data visualization standpoint. For instance, Excel’s default colors do not provide good contrast and nearly all Excel charts are laden with chart-junk. Let's get out the brooms! I know many of you have read Tufte. Good infographics are within our reach. Let's not multiply choice, but gently guide users to a limited set of good choices.

Incidentally, 3D charts will be turned off globally, although users can easily turn it back on by going to Preferences, selecting the Charting tab, opening the Advanced Charting subdialog box, finding the "Pimp my ride" sub-sub tab (it's down there in the back, just look for it!), and choosing both the "3D" and "extra glossy" checkboxes. There may also be some permissions issues here, too.

Project 2: Provide guidance on chart types: Users choose from a large variety of chart types in Excel without any background in effective data visualization principles. As a result, users can select from charts that are known to be ineffective in displaying data. These chart types include: pie charts, stacked column charts, clustered bar charts, cone charts, pseudo 3-D charts.

Well, we're going to solve some of that problem in Phase 1 by removing ineffective chart types. However, we can do better. We're the intellisense guys, remember! We will intelligently introspect on selected data to present no more than three good charting choices to a user. If lots of series are selected, then we offer multi-line and parallel trellis graphs as options. If it's a single series with just a few values, then we show bar. If it's a lot of data with a category dimension, we can offer box charts. We can do this.

Project 3: Provide missing chart types: As the right hand taketh away (Project 1), the left hand giveth. Excel lacks a number of effective chart types including: box charts, bullet charts, dot plots, trellis displays. Tableau's got all this advanced infovis stuff, why don't we? This is our chance to exercise innovation and move the industry forward.

Project 4. Create a common packaging format for Excel add-ins: Mozilla Firefox provides an excellent example of how to build a lightweight and easy to use extension process. There, I said it. Let me say it again: Mozilla Firefox provides an excellent example of how to build a lightweight and easy to use extension process. Let's all say it together. Mozilla Firefox provides an excellent example of how to build a lightweight and easy to use extension process. Whew! Feel liberated?

Let's face it, using Excel add-ins is difficult, confusing and, at times, frightening, to users. Furthermore, add-ins from different providers are packaged differently. There's no consistent process to add functionality to Excel.

We must make it easier for ordinary users to innovate and add features they need to Excel. A side benefit of this project will be that this will damp down our need to pack the product with every conceivable feature under the sun. What's more, we can harvest the best of these add-ins as built-in features for future versions of Excel. That's what Firefox has done.

Project 5: Develop a community around Excel: There are a number of practitioners providing good technical advice for Excel charting. However, even this excellent technical advice is not reaching users. A recent user group meeting in Atlantic City, New Jersey featuring several Excel luminaries got only 50 attendees!When SAS puts together a user conference, they have thousands of attendees and hundreds of papers are presented. They have it in a convention center for goodness sake.

Users don't know what they don't know. They don't know that they need to keep learning and sharing what they know. We need to celebrate the luminaries among our users that help fill the inevitable gaps in our products. And we need to encourage innovation and sharing from our lead users.

That's it. Just five things. Five things that will make Excel a better product and the world a better place. Let's get to work. One way or another these changes must come to Excel even if I have to do them myself.

Megalomanically yours,

Chris Gemignani

Microsoft Excel 2007 Program Manager

Note: Concept cribbed from the temporarily ennobled Jonathan 'Wolf' Rentzsch. Thanks to Kelly O'Day for indirect inspiration.