Dunning-Kruger effect - Wikipedia, the free encyclopedia

Mistakes in Experimental Design and Interpretation

Taxonomy of ways to fail your experimental design.

The design process is about whittling away distractions, making the obscure feel obvious, making the obvious feel implicit, and doing it without anyone noticing. To the untrained eye, your best work looks like you’ve done no work at all. If you’ve done a stellar job, then your design will feel utterly obvious. —Neil Mix from Paradox of Elegance blog post.

Neil goes on to say that "it’s easier to see the flaws than it is to see the elegance." That may be true, but a careful look at the best interfaces reveals the little and beautiful elements that make all the difference. These small features might not determine whether someone uses a piece of software, but they will determine if the user enjoys their stay:

Designer Bret Victor, who we first wrote about here, has developed a desktop widget for the SF-area train schedules. He allows users to change their query right in the description of an object—notice the red text.

While we are fawning over Bret's handiwork, here's another cool feature he built into his Click-Shirt site for customized design of t-shirts. This bar at the bottom of the screen tracks the history of changes as the user designs a shirt. Each time I make an edit, I get a visual breadcrumb trail to easily see my history and backtrack.

Google Finance stock charts have a nifty little device that lets you change the time range you are looking at. You can change both the size and the start of the time window using one adjustable object. Not to mention the embedded alert markers.

Some elegant touches are more subtle. Check out the search toolbar in Firefox. When I start typing, it fills out search terms both from my history (above the line) and from common searches (below the line).

The Safari browser for Windows offers a new approach for finding words in a web page. The browser greys the screen and highlights the target word. And you can to tab through the various instances of the word with the orange highlighting.

And while we are on the subject of Apple: sometimes the difference between clunky and good is simply about the quality of the images. A while back I wrote a break-up letter to PowerPoint—one reason was that the Mac-alternative called Keynote does a much job with the look of default charts. The chart on the right feels more professional, in part due to the anti-aliasing of the image. (Joel on Software has an interesting post about anti-aliased fonts here.)

Finally an infographic from the New York Times called the Sector Snapshot. The beauty of this presentation of information is in the careful use of contrast and skill at keeping the focus on the numbers.

“The representation of numbers, as physically measured on the surface of the graphic itself, should be directly proportional to the quantities represented.”

Edward Tufte calls violation of this principle the “Lie Factor”. The implementation of in-cell data bars in Microsoft Excel 2007 is a big offender.

Almost a year ago, I was surprised to discover that the Microsoft Excel 2007 development team didn’t understand what zero means. Their implementation of in-cell data bars showed a bar in a cell, even if the cell had a zero or very low value.

That was in the Excel 2007 Beta. Things haven’t improved in the current version of Excel 2007. The default setting for data bars in Excel 2007 is to scale to bars so that the smallest bar is based on the smallest value in the selected range and the largest bar is based on the largest value. It still appears that the smallest bar will be no smaller than five or ten percent of the width of the cell. Here’s a sample:

So, if you select a range that has values between 600 and 700, the 600 would have a little bitty bar and the 700 would have a full-width bar. Based on the bars, it would look like the 700 is ten to twenty times larger than 600. Outside of Redmond, this is generally regarded as untrue.

What’s more, if you create two sets of data bars side by side, each group of data bars scales itself independently even though they look the same. Take a look at this screenshot:

Notice the top seven cells have data bars that have one set of scaling and the bottom data bars have a different scaling. However, they look identical, and users should generally expect these bars to have the same scale.

Here are the rules:

1. Defaults matter! It doesn’t matter that you can do data bars correctly in Excel. The default should be to do it right and it should be hard to do it wrong.
2. The “right way” to make data bars is to make the length of the data bar directly proportional to the value in the cell. If one cell has a value twice another it should have a bar that is twice as long.
3. Remove the default gradient shading. The gradient makes it hard to tell where the bar ends, obscuring what you’re trying to show.
4. Continuous cells with data bars should all use the same scale. Use different colors to indicate ranges that have different scales.

Excel 2007 supports at least twenty-five different combinations of ways of specifying the length of the data bar.

Exactly one of those ways is correct. Base the shortest bar on the number 0. Base the longest bar on the highest value. Turn off the gradient. If you want to see bars based off percentile or some custom formula, then be explicit. Create a new column, create your formula, create bars on that column.

Please, guys, this isn’t rocket science. This is plain common sense. You would not ship Microsoft Word with a glaring bug in the way text renders. You would not ship Excel with a broken statistical function that people use everyday. Delivering deceitful-by-design infographics betrays your central role in democratizing the analysis of data. Until you fix this, in-cell ASCII art still remains the best way to explore data visually.

A disclosure: We do not currently use Excel 2007 at Juice Analytics. This is not due to a high-minded sense of moral outrage but is merely a reflection of our clients' environments.

Back in my consulting days at Diamond Technology Partners, I was known for my ability to bend PowerPoint to my will and fashion epic presentation-stories from lovingly-crafted slides. There was a term used when a client wanted a good looking presentation; they would ask if it could be "Zachified." Ah, the false glory.

Now I realize I was merely an amateur in designing presentations that could entrance and persuade an audience. I was going on instincts without much thought to the types of evidence, structure, and flow that would convince my audience.

Last week I had lunch with a man who has made a living from teaching others how to create effective presentations. His name is Andrew Abela and his blog is Extreme Presentations. Andrew has developed a thorough framework and training approach. He has a Doctorate and is a professor at Catholic University, so you know he brings an academic seriousness to the messed-up world of flufferpoint:

def (Withrop Hayes): A presentation that attempts to distract from the lack of substantive content or evidence with use of screenbeans, clip art, and other stock pictures or illustrations. A.k.a. clipterfuge (Todd Moy), clusterpoint (Cathy), The Macy’s Data Day Fluff Parade (Jamel).

Andrew gave me a quick backstage pass to his training methodology. Here are a few highlights:

1. Like a fool, I asked whether he preferred the sparse Lessig method or the more traditional, content-rich method. False choice. It all depends on the situation, just don't use the wrong approach at the wrong time. Andrew makes the distinction between "ballroom style" and "conference room style."

"Ballroom style presentations, like most typical PowerPoint presentations, are colorful, vibrant, attention-grabbing, and (sometimes) noisy. They typically take place in a large, dark room, such as a hotel ballroom. Conference room presentations are more understated: they have less color and more details on each page. They are more likely to be on printed handouts than projected slides, and they are more suited to your average corporate conference room. The single biggest mistake that presenters make is to confuse the two idioms, and particularly to use ballroom style where conference room style is more appropriate. I would estimate that upwards of 90 percent of all PowerPoint presentations use ballroom style, yet most of the time our presentation conditions call for conference room style."

That's from an article he shared with us called Achieve Impact through Persuasive Presentation Design (PDF)

2. It is important to mix data-based evidence with anecdotes. People need both of these types of information to persuade both the mind and heart (my interpretation).

3. Anticipate your audience's objections and build them into your storyline. What is better than having exactly the right slide next when someone raises a concern?

4. Good presentations require a lot of thought about their design. Andrew has defined five dimensions of an "Extreme Presentation": logic, rhetoric, graphics, politics, and metrics.

His blog offers a couple useful tools:

• A framework for choosing the right chart
• Slides that pass the squint test : "A good way to test whether your page is laid out properly is to apply what designers call the "squint test." Squint at the page, so that all the text is blurred and illegible. Do you get anything about the page without having to read the text? If you can see that the page is showing a process or two or three alternatives or a bunch of things converging, then your page passes the squint test."

Sharing what matters | Economist.com

This requires a new level of flexibility in the database. When building most databases today, programmers decide in advance what sort of questions users might wish to ask of the data, by defining what are known as the “schema”

Allowing Business Users To Program Your System Is A Recipe For Disaster

Parts of the solution make sense, but the problem is scare-mongering.

Spreadsheet errors, news stories about spreadsheets with costly mistakes

Ok, now who's going to measure the benefits?

Today I got this chart in an e-mail from the well-meaning alumni fundraisers at my school:

Aside from the crazy looking gridlines, monsterous size, oddly phrased title, and overkill for a single data point, its greatest sin is not giving the Young Alumni enough credit. They blew away their goal of 300 gifts, you just wouldn't know it from this chart.