Delivering Data in Excel: The DTP Framework

Here at Juice we build fewer Excel dashboards than we used to. Excel itself is a decidedly imperfect vessel for any serious development--it's simply too easy to veer off of the disciplined track onto the underbrush.

Even so, Excel remains a playground where we can do surprising things. For instance, check out our Excel lightbox and an Excel tagcloud. We could appropriate everything that you find on the webbiest of Web 2.0 websites and build our Uruk-hai equivalents.

The key to staying on the rails when building Excel tools--either dynamic dashboards or simply to explore data--is discipline. At Juice, we use a methodology that we call "DTP" (Data Tansform Present). The foundation of DTP is the rigorous separation of data from presentation. This is similar to a well-known approach when building computer user interfaces called Model-View-Controller. I'm going to cover some of the key principles and we'll follow up with an example later on the blog.

Data

Data is the raw material of any visualization or report. It needs to be easy to add data or change data without having to change anything else about your dashboard.

We store raw data with dimensions preceding metrics in blocks in separate worksheets. If you want to sound pretentious, you can call this "first PivotTable normal form". Key points:

  • Have one worksheet for each data source.
  • Call these sheets "Data", or "{Title} Data".
  • Place them at the end of your workbook.
  • Data is snug to the top left of the spreadsheet. This allows us to use dynamic ranges. Dynamic ranges let you add data and have it automatically incorporated in all PivotTables.
  • Ensure that column names are in the first row.
  • Place your dimensions before metrics. Dimensions before metrics

Transform

We use PivotTables to transform the data into the structure we need.

  • Call these sheets "Transform" or "XXXXXXX Transform".
  • Create one sheet for each issue that you are exploring. This doesn't mean that you will only create one PivotTable. You may have multiple PivotTables to support different views or perspectives on an issue.
  • Turn on "show items with no data" for row and column dimensions. Show all items
  • We are seeking predictability, we want to the PivotTable to always be the same size regardless of what the PageField filters are.
  • Place all the dimensions that aren't used as rows or columns in the PivotTable as page fields. Every dimension should have a home. All dimensions must have a home
    • Set all PivotTables to not store data and refresh on open. PivotTable settings

Present

The Presentation page copies data from the Transform page(s) and formats it for display. It also allows users to control what data is being displayed.

  • Build a user interface to interact with your data. There are many ways to let people interact with your data, but one of the easiest is to use a PivotTable as your interface. This is described below.
  • We use an in-house style guide for graphs that you can see in our Chart Chooser.
  • If the Presentation page is likely to be printed, preset the print range.
  • When copying data from the transformation page to the presentation page, blank values will come out as zeros. We use a simple formula, =if('Transform!A2'<>"",'Transform!A2', ""), to ensure that blanks remain blanks.

Using a PivotTable as your interface

A simple way to let people manipulate your data is place a PivotTable containing only PageFields but no data on the presentation sheet. A Visual Basic macro triggered to run whenever the PivotTable changes then pushes out any changes to the master PivotTable to all the PivotTables on your Transform sheet.

Here is the code to make this happen.

This drives our PivotTables in concert and ensures they stay in sync.


That's a basic overview of our DTP technique. You can try a simplified version of DTP here.

DTP Example.xls

We'll be back soon to talk through this example.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

14 comments | Show all comments only the last 5 are shown


February 26, 2009
Jacob said:

I am having trouble getting rid of zeroes from my charts. Where exactly does the formula =if('Transform!A2'<>"",'Transform!A2', "") go?

Thanks in advance


April 2, 2009
nicholas said:

You say that you don't use Excel that often anymore to create dashboards. What tools do you use or recommend these days to build dashboards?


April 2, 2009
Zach said:

Nicholas, Most of our dashboards are web applications using Flex and our open-source visualization library JuiceKit (www.juicekit.org).


June 10, 2009
Patrick said:

Wow - Thanks so much, I love it and this make life with Pivottables so much easier! Goes right into our weekly reports!
One question: I always thought I know Pivottables pretty good - but how do I add Pagefields without Data so that the blue frame does not show up like in the example file? Thanks for your help! I love your tools and have been an avid user of the Chart Cleaner for years now. :-)


December 12, 2009
shawnify said:

Typo in third paragraph: "DTP" (Data Tansform Present)

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment





Bubble, bubble toil and trouble

Recently we wanted to show how Concentrate, our new long-tail search analytics tool, could give you a view of search patterns across travel websites. As political junkies, we were inspired by this chart from our friends at the NY Times.

NY Times candidate word bubble chart

The first tool we tried, simply on principle, was Excel 2003. As expected, making a NY Times quality bubble chart in Excel 2003 is a hard problem. Here's a draft of how far I got before giving in to label fatigue.

Excel NY Times bubble

The bubbles themselves aren't tough, but getting the labels right is hard. I'd love to see a solution, so if any reader wants to tackle it eternal fame can be yours. Here is a CSV if you want to try.

travelpatterns.csv

Another of the tools we use at Juice is NodeBox, which we used to make this:

Concentrate pattern comparison

Here's the code that made the graph.

The power of a programmatic approach like this is that by changing a line or two, you can get the following. Click for a larger version. Click the text for the code..

With great power comes a great need to exercise restraint. Otherwise you end up like these poor chaps. Must... flex... restraint... muscles...

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

17 comments | Show all comments only the last 5 are shown


January 16, 2009
chip said:

Rob Bovey has an xy chart labeler that may have helped on the original Excel version. I use it a lot and it provides a good degree of flexibility on placement.

http://www.appspro.com/Utilities/Utilities.htm

The labels are not dynamic which is a drawback. It works on other types of charts too.


January 18, 2009
Andy Cotgreave said:

Hi Clint,
Yes, I did initially add the text. However, in Tableau it somewhat overwhelmend the circles. I did try to format the text to grey and shrink it, but the text only served to confuse things.


January 19, 2009
Chandoo said:

Hi Chris,

Good stuff...

I have tried the same in excel while keeping the labels right (I guess so). You can take a look at the chart and downloadable excel here: http://chandoo.org/wp/2009/01/19/excel-bubble-chart/

Let me know your comments


February 9, 2009
David Franta said:

Didn't really find another place to post this, but interesting article posted by Cringely (ZDnet fame) about how JP Morgan mangled a bubble chart recently -

http://blog.cringelysmortgage.com/2009/01/29/whats-wrong-with-wall-street/


February 22, 2009
Mike Chelen said:

How about using the Google Charts API scatter plot? http://code.google.com/apis/chart/types.html#scatter_plot
It allows variable bubble sizes, and has been used in some similar charts such as http://www.xefer.com/twitter

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment





Introducing Concentrate for Long Tail Search Analytics

We are pleased thrilled to introduce Concentrate™, an innovative long-tail search analytics tool. Concentrate is for SEO and paid search professionals who want to make sense of search keyword data and make the most of search investments.

Check out the demo here. Or try out the free version here (you’ll need admin access to a Google Analytics account).

We built Concentrate because we saw a fundamental conflict in the world of search analysis: On the one hand, search keyword data is terrifically interesting and valuable. It can tell you what your visitors and customers want and how they think about you and your products.

Juice Analytics keywords

Unfortunately, search query data is also big, messy, and hard to get your hands around. In a typical month, the Juice site gets over 10,000 visits from over 7,000 unique keywords.

Even if I could somehow wrap my head around our top 100 keywords, I’d only understand 25% of the visits. For people spending money on search engine optimization or paid search campaigns, that’s a big blind-spot to accept.

We want you to understand and act on all your search data. Concentrate ingests data from sources that most sites already have available (e.g Google Analytics, Omniture, Coremetrics, Hitwise, Compete, etc.), enhances this data by finding common patterns and query types, and visualizes search phrases for exploration and analysis.

Over the next couple of weeks, we will share examples of some of the interesting things you can do with Concentrate, including:

Pattern identification to condense the long tail into keyword phrases with similar structures. For example, here are some common search patterns from a cooking web site (the “[x]” represents a wildcard).

Patterns

Keyword visualization to show the connections between keywords and the relative performance of phrases. This wordtree shows the frequency of words within phrases (size) and average time spent on site (color).

Wordtree

Congratulations to Chris, Pete, and Sal for all their hard work, diligence, and creative problem solving to launch this solution.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

8 comments | Show all comments only the last 5 are shown


January 10, 2009
Daniel Waisberg said:

Looks amazing, I will implement it and start working for my own website. I think that for search marketing / SEO companies this will be a killer tool. It can add a huge value!


January 12, 2009
Bjoern Sjut said:

Hi,

has there already been testing with foreign languages? I could volunteer to integrate it with a German content heavy site to test the behaviour on umlauts, etc.


January 12, 2009
Bjoern Sjut said:

Oh, I can shed a light on this already: My most important keywords for our German sites are "error#" and "unicode error#" :-(


January 12, 2009
Pete Skomoroch said:

Bjoern,

Thanks for the feedback. I just fixed that unicode error for you and reloaded your list. Concentrate should run without errors on foreign languages, but some of the text processing components (stopwords, stemming, etc) are only fully supported in English at the moment. Let me know how the new results look and we will work on incorporating more international features.


February 27, 2009
Pauli Price said:

On the final validation stage, where I entered the bounce rate for my first keyword, the application met with an un handled exception because it couldn't find the google analytics keyword file. Perhaps because there were spaces in my site name? Unfortunately it also spit out all kinds of diagnostic information you probably don't want the casual observer to see. You really want to trap that unless the login is a privileged account.

Anyway, help doesn't go to a help screen or anything - it appears that clicking on 'help' brings one to the account page, so I figured I'd post my tale of woe here.

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment