methodology

Are you using information effectively?

Have you noticed that sometimes it’s hard to get your point across? Do you find you’re trying to do the right thing with your information, but the organization just won’t cooperate?

We think this is happening too often in the companies the Juice Community lives and works in. And we want to do something to change it.

We want to better understand if we’re helping you be more effective in your workplace as an information evangelist. To make this possible, we’d like to ask (yea, even beg) you to complete a short 10 question survey about how information presentation is making progress in your company and if you feel alone or supported by the info-viz pundits out there.

But you might ask "what’s in it for me?" Well, to begin with, we’re going to demonstrate how to summarize qualitative survey information. You’ll get some great examples of how to apply non-traditional charting styles to problems within your organization.

However, we can’t do it alone; we need you to complete the survey. And if we don’t get enough respondents, the results won’t lend themselves to what we have planned.

So what are you waiting for? Fill out the survey here and help us help you help us. And what does Gilligan’s Island have to do with information presentation? Well, you’ll just have to take the survey to find out!

Update: The survey is now officially closed. Thanks to all who responded. We’ll have the results out in a few days.

In Fogbugz 6.0, Future Results CAN be Predicted by Past Performance

Monte Carlo. It’s a car. It’s a song. It’s a casino. It’s a city in Monaco (near France, somewhere). It’s also a method of statistical simulation that is used to better understand the probabilistic distribution of known set of data. Great. So what?

I recently attended the Atlanta session of the FogBugz World Tour to hear from Joel Spolsky about the latest version of his bug tracking software, FogBugz 6.0. Joel Spolsky has been writing about software development, management, business, and the Internet at joelonsoftware.com since 2000. Three books have sprung from content on his site: User Interface Design for Programmers,Joel on Software, and Smart and Gets Things Done. They are good reads for software developers and business folks alike; smart, funny, and focused on the human face of software development.

Before starting his company, Joel was a Program Manager at Microsoft on the Excel team to provide programmability to Excel.

The new release of FogBugz has quite a few nice features to make bug tracking much easier, but I was most intrigued by the new capability called Evidence-Based Scheduling, or EBS for short. This new capability uses a Monte Carlo simulation approach to determine probabilities associated with the expected “ship dates" of the software release project.

The core premise behind this capability is that unlike in the financial realm, in the software world, future results can accurately be based on past performance. FogBugz remembers the original estimate and the actuals for each task for each developer, and for all the projects they’ve been assigned to. Here’s where it gets interesting. Based on this information, FogBugz runs a series of Monte Carlo scenarios where it randomly generates different plausible results for each member of the development team. It then assembles the results to create a distribution for probability of completion for each developer—and more importantly, for the entire project. The result of this analysis is a "Ship Date" probability curve that shows potential ship dates on the X-axis and probabilities of achieving those dates on the Y-axis. Ship Date depends not only on estimates for remaining components, but also on the probability that the individual developers will be able to individually meet their commitments. A steeper curve means the developer estimates are more confident and a flatter curve means estimates are less confident.

The classic software development process involves balancing three things: time, money, and features. Most projects of reasonable duration are not going to be able to effectively add staff mid-stream—at least not to accelerate delivery timeframes. If you assume the primary factor in determining "money" is tied to people, time and features are your only real variables. FogBugz 6.0 lets you experiment with changing these two factors. This is a useful tool. Consider a scenario where the business people come to the development lead because they need a project to be complete by a specific date. This tool gives the project lead enough information to understand the probability of making a specific date. Additionally, it lets you test out what will happen if you remove a particular feature. With this new information, the development lead has the visibility to provide back additional guidance to the project sponsor about the probability of making the requested date, as well as the effect that changing release date and scope have on the probability of on-time delivery.

So how does it know? The ship dates are based on each developer’s history of estimation accuracy supported by developer time sheets. Yes, that’s right. I said time sheets—the bane of all developers. But stick with me, it’s not as bad as you might think. Each developer (or responsible team lead) can turn on a timer that automatically tracks time spent on each of their cases.

Fogbugz plots a linear fit through the data points for each developer and then uses the calculated R2 value to determine how consistent the developer’s estimates have historically been. Based on this calculation, probability distributions for remaining tasks for each team member are determined, which leads the ship date probabilities. It’s then easy to see the long pole in the project (Milton Ritchie in this example). This doesn’t necessarily mean the most work, but only reflects the probability of on-time completion—and correspondingly, the most risk to the project.

Anyone who has followed Joel’s writings knows that he is keen on the idea of improving the process of software development. And, we all know that estimating an accurate time to completion, or “ship date," for any software project is a pain in the rear even after all these years of "practice." The unique approach that Fog Creek takes is not to predict the specific completion time, but rather to predict the probability of completing the project across a range of dates.

So, project estimating will likely never be a simple turn and churn process. But with this release of FogBugz, Fog Creek Software has shown great outside-the-box-thinking that could significantly improve how we deliver software projects.

Wanted: Smarty Pants, apply within.

Chris talked about customer intimacy last month and that kind of thing always gets my mental juices flowing. When an idea like that is laid out in front of you, it’s a head slapping "of course" and "weren’t we doing that already?" thought.

Being around really smart people like Chris and Zach gives you a Newtonian stance on ideas; you get to rest your mind on top of fully baked thoughts. You also get critical, constructive analysis of your own musings and ultimately both our clients and Juice reap the benefits.

But where do you find the kind of people who live for this stuff? It’s not like you can pop down to the mall and pick up a clutch of insightful, ultra-curious, Excel-wielding Python gurus. No, there are really just two solutions: You have to stumble on them or grow your own.

Stumbling on smart people really is tons of fun. Some of what works is really obvious and takes the form of talking to people at conferences, wooing people with great blogs, reaching out to some of the better user groups, and posting on places like Craig’s List. Actually that last one isn’t so obvious because unless you can articulate your company’s worldview in a few muscular paragraphs you’re just going to attract the wrong kinds of people. If your post is too wacky you’ll be treated to an interpretive dance during the interview to detail how a project was a success. And if your ad is too dour you’ll attract the living dead. Oh, the horror.

A few years ago popular belief held that there were oodles of highly trained, big-brained technology folks begging for work. That might have been partially true, but I do know a lot of carpetbaggers left the business to go back to whatever carpetbaggers are doing these days. I’ve been blessed with working with some amazingly brilliant people over the years. None of them have ever had a problem finding work. Ever, ever, ever. Those are the kinds of folks you want to go out of your way to stumble upon.

The second method of growing your own might sound like a leap of faith but it’s really effective if you can pull it off. Back in the mid nineties, I was King of the Internet for a rapidly growing software company. Much like everybody else, we suffered the slings and arrows of an outrageous job market, and finding top notch talent was an uphill struggle. The world had all but lost its mind and you’d find yourself seriously mulling the thought of shelling out $100k a year for an HTML "programmer." Plus signing bonus, of course.

No, that would never do. Something different had to be done.

That’s where working with brilliant people comes in handy. If you float what might be an out-of-the-ordinary idea they’ll actually think it over before voting either way. We had this crazy idea to take really clever people from outside the industry and train the living daylights out of them.

Boy, it worked like a charm. I still keep in touch with a few of these rather bright individuals and they’re still enjoying the heck out of their careers. One is still with the company, one with a smaller software outfit, and the third is a consultant for one of the largest consulting entities.

Whichever route you take, you can only squeeze out really juicy ideas from the right kind of brain. In our case we value creativity, chutzpah, a smart work ethic, dedication, and unbridled curiosity. It’s not enough to teach somebody to be effective with a toolset and, to paraphrase Potter Stewart, we know smarts when we see them.

Learning from the edge cases (Part 2)

In my previous post, I mused on the subject of edge cases and the learning opportunity they provide. I want to touch on how this applies to customer analytics.

This weekend I read a blurb in the Washington Post Food section. A customer by the name of Anne Monahan complained about the “dark blue menus printed in black ink" at a local restaurant. “In dim light, the menus were nearly impossible to read," she remarked. The restaurant co-owner said that she hadn’t noticed the problem before, and vowed to change the hue of the menus the next day.

It would be easy to dismiss Ms. Monahan as an outlier and a whiner. After all, this complaint was rare. An edge case. But the restaurateur decided to respond to the issue. Perhaps other patrons were bothered by it, but hadn’t commented.

This is not to say that companies should be a slave to the edge case. But don’t throw them out. Listen to what they have to say and be willing to respond, because:

  • They may be the canary in the mindshaft — telling you something that others haven’t yet realized

  • They may be an extreme case of common behavior that shows up more subtly amongst other customers. Recognizing this behavior can only help you better meet customer needs in the future.

  • They may offer new ways to think about the business or customers. As I said in my last post, edge cases help define the boundaries of reality.

  • Collectively, outlier customers provide a service: they stress test the product and highlight unrealized strengths or weaknesses.

Of course, there is also much to learn from the ordinary cases — the mainstream customers. I think most companies already understand the ordinary. In fact, the ordinary is already deeply embedded in the business’ assumptions.

Most statistical analysis tells you: watch out for outliers. They are the data points that can screw up your averages. Because of their rarity, they aren’t deemed worth focusing on. I disagree.

I hope to return to this topic as we find ways to apply it at our clients.

Too much data, too little focus

In his article Eyes Bigger Than Stomachs: Data Glut , Jim Meskauskas observes that there is “a frenzied gathering of bits, collecting any and all manner of jetsam and flotsam. There is a subconscious belief that the marketer will know what it is that they are looking for once they find it."

He recognizes that more data isn’t better. In fact, it can be worse. Vast expanses of data have a way of lulling organizations into a false sense of security. It is easy to believe that the answer to virtually any business question will be available because all available data has been collected.

This reminds me of a data gathering process I recently observed for a software release. I watched (and participated) as the organization collected more than 160 “reporting requirements" through brainstorming by a wide-range of stakeholders.

The repurcussions of this massive list aren’t good:

  1. Reporting requirements are often the first things to get dropped as software development schedules get tight. With little prioritization, these requirements get dropped in a relatively random order.
  2. No one was challenged to ask: what are the truly important measures of success? The assumption was that somewhere in these 160 data elements, the answer would be found at some later date. But is there any guarantee? The organization has an illusion of future flexibility. In truth, the resources necessary to manage the sheer bulk of data gathered reduces flexibility.

I couldn’t agree more when Jim says says: “you’ve got to have a thesis around which your experiments are structured or you don’t discover anything because you don’t know what you are out to discover."

Exchanging Methodology for Principles

We met with a prospective client today—the meeting went very well.

This client, a former consultant, first asked us: “so what am I buying here; what’s your methodology". Crafting and intellectually protecting a “methodology" is how consultants mass market what is essentially a one-to-one service. Methodology is a shield and a sword that allows an inexperienced MBA to go into a situation and start hacking and slashing.

We feel that it is better to have principles and experience. Two things, admittedly, that we are continuing to develop.

No, we don’t have a methodology yet. But we know in our gut that there is value in a light approach to analytics. We seen too many hours and too much talent wasted on analysis of the wrong problem. We know of frustration among business leaders because their business simply does not know what’s going on. We’ve seen the waste caused addressing the wrong questions. We’ve felt the loss of repetitive grunt-work. And we’re excited to show the world something different.

But this methodology; our approach as we learn it and craft it in the field is not a secret. Like RedMonk’s business principles, or the Agile software manifesto, we strive to openly recognize those who have helped us and the projects from which we draw our inspiration.

Analytics Ingredient List

James Governor draws a nice analogy between learning how to cook and learning to rely on your staff to develop and implement IT infrastructure. In either case, you can get pre-packaged solutions. But these solutions won’t solve long-term needs.

Next time a vendor pitches you a “solution" ask yourself what are they afraid of? Is this science or an illusion of science? And if you choose this “organic IT solution" what are you missing out on? What nourishment? What experience? What set of skills that you can apply in other areas?

I think he’s exactly on point about the slippery simplicity of buying comprehensive solutions. Naturally, this also applies to analytics solutions (like an Cognos or Business Objects implementation), instead of developing capability in-house.

It’s better to teach a man to fish than to give him a fish. And it’s better to know the data and how to manipulate that data, than to learn a “solution". It’s better to concentrate on solving your concrete problems than to implement a solution that solves all problems.

There are wonderful, cheap tools available now. The dynamic languages: Perl, Python, Ruby. JMP. Even something as simple as Excel with our DTP framework can do powerful things. At Juice, we’re learning ways to cook up concrete solutions out of cheap componentry and teach them to your staff so that the next time a need arises, your staff can cook for themselves.

Learning from the edge cases (part 1)

I’ve recently developed an interest in "edge cases" - the extreme situations or data points that fall far outside the norm.

It was first piqued when I read a post by James Vornov about the impact of extreme cases on decision making. He notes: "Studies of decision making have shown that people are strongly influenced by single, uncommon events. Even when the pattern of frequent events indicates one type of behavior, the uncommon event prevails."

Edge cases do more than create the deepest impression; they also offer rich ground for learning. Consider two catalysts for learning: 1) frequent but ordinary events and 2) extreme events. Each offers a different type of lesson. When we learn from the ordinary, we gain the ability to predict likely outcomes and put clear dimensions around expected results. Learning from the edge cases is wholly different: it helps us define the bounds of reality. It tests our assumptions and creates sharp contrasts.

Storytelling is just one example where edge cases are a teaching tool:

 

 

 

 

 

 

  • The legal profession uses the extreme cases to define precedents and test the limits of laws
  • Engineers conduct stress testing on materials or products to understand the limits of capabilities. Similarly, programmers test code by defining edge cases.
  • Individually, I think we learn most about ourselves in situations when we experience something new, unusual, and challenging.

Another way to view edge cases is that they test our common sense. Tato on Everything2 points out that "as science pushes our understanding of the universe and our selves, we are confronted with new complexities and edge cases where these instincts [common sense] are actually dysfunctional, or wrong."

Michael Feldstein offers a similar view in his blog when he says: "In any field of inquiry, the edge cases are where some of the most interesting work gets done."

In each case, edge cases help us understand the far reaches of the possible. They help us map out reality. In a future post, I want to talk about how businesses can use edge cases, in particular outlying customer data points, to better understand their products, customers, and marketplace.