Delivering Data in Excel: The DTP Framework

Here at Juice we build fewer Excel dashboards than we used to. Excel itself is a decidedly imperfect vessel for any serious development--it's simply too easy to veer off of the disciplined track onto the underbrush.

Even so, Excel remains a playground where we can do surprising things. For instance, check out our Excel lightbox and an Excel tagcloud. We could appropriate everything that you find on the webbiest of Web 2.0 websites and build our Uruk-hai equivalents.

The key to staying on the rails when building Excel tools--either dynamic dashboards or simply to explore data--is discipline. At Juice, we use a methodology that we call "DTP" (Data Tansform Present). The foundation of DTP is the rigorous separation of data from presentation. This is similar to a well-known approach when building computer user interfaces called Model-View-Controller. I'm going to cover some of the key principles and we'll follow up with an example later on the blog.

Data

Data is the raw material of any visualization or report. It needs to be easy to add data or change data without having to change anything else about your dashboard.

We store raw data with dimensions preceding metrics in blocks in separate worksheets. If you want to sound pretentious, you can call this "first PivotTable normal form". Key points:

  • Have one worksheet for each data source.
  • Call these sheets "Data", or "{Title} Data".
  • Place them at the end of your workbook.
  • Data is snug to the top left of the spreadsheet. This allows us to use dynamic ranges. Dynamic ranges let you add data and have it automatically incorporated in all PivotTables.
  • Ensure that column names are in the first row.
  • Place your dimensions before metrics. Dimensions before metrics

Transform

We use PivotTables to transform the data into the structure we need.

  • Call these sheets "Transform" or "XXXXXXX Transform".
  • Create one sheet for each issue that you are exploring. This doesn't mean that you will only create one PivotTable. You may have multiple PivotTables to support different views or perspectives on an issue.
  • Turn on "show items with no data" for row and column dimensions. Show all items
  • We are seeking predictability, we want to the PivotTable to always be the same size regardless of what the PageField filters are.
  • Place all the dimensions that aren't used as rows or columns in the PivotTable as page fields. Every dimension should have a home. All dimensions must have a home
    • Set all PivotTables to not store data and refresh on open. PivotTable settings

Present

The Presentation page copies data from the Transform page(s) and formats it for display. It also allows users to control what data is being displayed.

  • Build a user interface to interact with your data. There are many ways to let people interact with your data, but one of the easiest is to use a PivotTable as your interface. This is described below.
  • We use an in-house style guide for graphs that you can see in our Chart Chooser.
  • If the Presentation page is likely to be printed, preset the print range.
  • When copying data from the transformation page to the presentation page, blank values will come out as zeros. We use a simple formula, =if('Transform!A2'<>"",'Transform!A2', ""), to ensure that blanks remain blanks.

Using a PivotTable as your interface

A simple way to let people manipulate your data is place a PivotTable containing only PageFields but no data on the presentation sheet. A Visual Basic macro triggered to run whenever the PivotTable changes then pushes out any changes to the master PivotTable to all the PivotTables on your Transform sheet.

Here is the code to make this happen.

This drives our PivotTables in concert and ensures they stay in sync.


That's a basic overview of our DTP technique. You can try a simplified version of DTP here.

DTP Example.xls

We'll be back soon to talk through this example.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

14 comments | Show all comments only the last 5 are shown


February 26, 2009
Jacob said:

I am having trouble getting rid of zeroes from my charts. Where exactly does the formula =if('Transform!A2'<>"",'Transform!A2', "") go?

Thanks in advance


April 2, 2009
nicholas said:

You say that you don't use Excel that often anymore to create dashboards. What tools do you use or recommend these days to build dashboards?


April 2, 2009
Zach said:

Nicholas, Most of our dashboards are web applications using Flex and our open-source visualization library JuiceKit (www.juicekit.org).


June 10, 2009
Patrick said:

Wow - Thanks so much, I love it and this make life with Pivottables so much easier! Goes right into our weekly reports!
One question: I always thought I know Pivottables pretty good - but how do I add Pagefields without Data so that the blue frame does not show up like in the example file? Thanks for your help! I love your tools and have been an avid user of the Chart Cleaner for years now. :-)


December 12, 2009
shawnify said:

Typo in third paragraph: "DTP" (Data Tansform Present)

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment





Bubble, bubble toil and trouble

Recently we wanted to show how Concentrate, our new long-tail search analytics tool, could give you a view of search patterns across travel websites. As political junkies, we were inspired by this chart from our friends at the NY Times.

NY Times candidate word bubble chart

The first tool we tried, simply on principle, was Excel 2003. As expected, making a NY Times quality bubble chart in Excel 2003 is a hard problem. Here's a draft of how far I got before giving in to label fatigue.

Excel NY Times bubble

The bubbles themselves aren't tough, but getting the labels right is hard. I'd love to see a solution, so if any reader wants to tackle it eternal fame can be yours. Here is a CSV if you want to try.

travelpatterns.csv

Another of the tools we use at Juice is NodeBox, which we used to make this:

Concentrate pattern comparison

Here's the code that made the graph.

The power of a programmatic approach like this is that by changing a line or two, you can get the following. Click for a larger version. Click the text for the code..

With great power comes a great need to exercise restraint. Otherwise you end up like these poor chaps. Must... flex... restraint... muscles...

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

17 comments | Show all comments only the last 5 are shown


January 16, 2009
chip said:

Rob Bovey has an xy chart labeler that may have helped on the original Excel version. I use it a lot and it provides a good degree of flexibility on placement.

http://www.appspro.com/Utilities/Utilities.htm

The labels are not dynamic which is a drawback. It works on other types of charts too.


January 18, 2009
Andy Cotgreave said:

Hi Clint,
Yes, I did initially add the text. However, in Tableau it somewhat overwhelmend the circles. I did try to format the text to grey and shrink it, but the text only served to confuse things.


January 19, 2009
Chandoo said:

Hi Chris,

Good stuff...

I have tried the same in excel while keeping the labels right (I guess so). You can take a look at the chart and downloadable excel here: http://chandoo.org/wp/2009/01/19/excel-bubble-chart/

Let me know your comments


February 9, 2009
David Franta said:

Didn't really find another place to post this, but interesting article posted by Cringely (ZDnet fame) about how JP Morgan mangled a bubble chart recently -

http://blog.cringelysmortgage.com/2009/01/29/whats-wrong-with-wall-street/


February 22, 2009
Mike Chelen said:

How about using the Google Charts API scatter plot? http://code.google.com/apis/chart/types.html#scatter_plot
It allows variable bubble sizes, and has been used in some similar charts such as http://www.xefer.com/twitter

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment





Decorating data

An early Christmas present has arrived from the DabbleDB team for the 100 million or so of us that have to work with data on a day to day basis.

They’ve created a do-what-I-mean web tool that lets you show how you want data to be restructured and bang! it’s done. Check out the video.

Cleanup data in action

It’s a great idea and a elegant, easy to use interface. There are so many directions I’d love to see them take this tool.

Cleanupdata is a great name, but they’re really giving you better ways to restructure data. This tool won’t help you find and fix errors and anomalies in data. At least not yet.

I also hope they extend cleanupdata to let people automate these data restructuring operations. If only you could apply a cleanup created in cleanupdata.com to 1,000 Excel spreadsheets or to a database table.

If you like this, it’s worth checking out DabbleDB. They have rethought the database with a database/spreadsheet/web forms/visualizer platypus of a tool. It lets your data be pliable in ways that databases don’t allow, while retaining structure that spreadsheets don’t recognize.

Added: Avi Bryant, one of the authors of the cleanupdata.com service notes that the example in the screencast is motivated by this post on cleaning data in Excel. Compare and contrast. I know most people would prefer to avoid ="("&MID(H2,1,3)&") "&MID(H2,4,3)&"-"&MID(H2,7,4) in order to format a phone number.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

7 comments | Show all comments only the last 5 are shown


November 30, 2008
Bjoern said:

I agree with Tom. I'd love this to run locally, but transmitting the data to dabble db and then have it mailed back to me. I don't know about US regulations but in Europe I would get seriously smacked on the head for that, especially for data like social security numbers, names and contact data that was shown in the example video.


November 30, 2008
Chris Gemignani said:

Thanks for the clarification, Avi. I've updated the text. Congrats to the DabbleDB team. We love it.


December 4, 2008
Michel Guillet said:

Hi Chris,

I've been using birst, a comparable product, with some success.


December 9, 2008
Mike Chelen said:

Is the free version always available or only for a 30 day trial? Also, it would be nice if more that 15000 rows were supported, overall looks like a pretty cool site and services :)


December 10, 2008
Ted said:

Sorry, I much prefer a spreadsheet and formulas. They give you complete control (the code in this product is going to make assumptions that are wrong at times), they are very repeatable (i.e. if I do the same thing every day, I shouldn't need to repeat my actions in a browser and run batch jobs), and you can look back at a later point in time and see what you (or soemone else) did very easily. Maybe MID is complicated, but I don't think so.

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment





Analytics Roundup: TIps for showing, sharing, communicating

Developer's Guide - Google Chart API - Google Code
Beautiful stuff, particularly the Venn diagram.

Align Journal - BI Worst Practices
We often see articles on BI "Best Practices" here is an article telling us what NOT to do.

flot - Google Code
Attractive Javascript plotting for jQuery.

ongoing ยท On Communication
Interesting blog post about how different forms of communication rank for immediacy, lifespan, and audience reached.

The Excel Magician: 70+ Excel Tips and Shortcuts to help you make Excel Magic : Codswallop

SlideShare
Source for presentation ideas.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

0 comments | Add a comment

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment






Introducing Chart Chooser

Find and Download Great-Looking Excel and PowerPoint Charts

Chart Chooser is an online tool that answers two questions we commonly get:

  1. What type of chart should I use to show my data?
  2. How can I make good looking Excel or PowerPoint charts?


Chart Chooser


Chart Chooser is easy:

  1. Check the boxes on the left that best describe your objective
  2. Select the chart that you want to use
  3. Choose from Excel or PowerPoint downloads to get a formatted chart template

A few notes about Chart Chooser:

  • Thanks to Andrew Abela of Extreme Presentations for inspiring Chart Chooser with his “Choosing a Good Chart” post and for working with us to put this tool together.
  • We’ve tried to make the charts both Tufte-compliant (i.e. minimal chart-junk) and visually attractive (thanks to Google for the color scheme).
  • Feel free to suggest other types of charts that you’d like to see in the Chart Chooser. Send an example to chartchooser@juiceanalytics.com.
  • If you’d like a customized version of Chart Chooser for your organization, write us at chartchooser@juiceanalytics.com or call me at 202.251.7750.
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

23 comments | Show all comments only the last 5 are shown


November 27, 2007
Kelly O'Day said:

This is my 3rd and hopefully final try at getting the link to work.

http://processtrends.com/toc_chart_doctor.htm


December 20, 2007
Stef said:

Hey there, the website is still not visible from Switzerland! Gush....


February 27, 2008
Mike said:

Hi guys!

This is fu&%$ awesome! Thank you very much for this!


February 27, 2008
Tom said:

I love your site and have used several graphs to make myself 'look good' at work. Thanks.
I want to use the Waterfall chart but for the life of me I can not figure out how you remove/hide the color fill from the data points after the first one and leave it in for this one.

Thanks.


April 12, 2008
Priya said:

Hey thanks for this useful site... I was wondering if there is a write up for different type of charts displayed here, as in what type of data or steps / FAQs etc.

If I am missing something here, let me know

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment





Earlier writing