Slow Down, Cowboy! How BI/Analytics Users can Avoid the Texas Sharpshooter Fallacy

It is better to know nothing than to know what ain’t so.  Josh Billings

A Texas cowboy fires several rounds at a barn. Looking at the holes riddled across the barn wall, he observes there happens to be some holes clustered together. He excitedly paints a bulls-eye centered over the biggest cluster of holes and proudly shows it off as proof he is a sharpshooter.

Texas-Sharpshooter-Fallacy

Source: youarenotsosmart.com

This old joke is the namesake for a type of logical fallacy known as the Texas Sharpshooter: to cherry-pick clusters of information, be it bullet holes or data, to support one’s own personal agenda.

Examples of the Texas Sharpshooter fallacy can be seen in public health investigations, politics (a politician might accuse an opponent of a “pattern” of poor actions), and pseudosciences like “intelligent design.”

I especially like author David McRaney’s description: “If hindsight bias and confirmation bias had a baby, it would be the Texas Sharpshooter Fallacy.”

But a Texas Sharpshooter fallacy can also be the result of the mistaken assignment of meaning to what is in fact a small subset of all related data. It then occurred to me: virtually every business intelligence tool has a feature to highlight a portion of a graph to focus only on a particular data cluster, enabling every BI end-user to “draw a bulls-eye” of their own! The risk of jumping to false conclusions based on such selective data selection can be very high.

So how can business intelligence users avoid unwittingly creating their own Texas Sharpshooter fallacies, and help debunk those put forth by others in the organization?

The problem occurs when a BI end-user draws a conclusion that is based on that isolated data cluster. Suppose a sales manager using a BI tool highlights a cluster of data points, all with a noticeably low gross margin compared to the other monthly sales. The sales manager then sees that those ten data points – all sales with low gross margins – are all deals closed by Joe Bloggs.

Is Joe Bloggs giving away pricing concessions to customers? Well, if the sales manager has poor analytical, managerial and/or interpersonal skills, or even just doesn’t like Joe, this data cluster may be all the “proof” (s)he needs to “have a word” with Joe, subvert Joe’s efforts by grousing over his lousy dealmaking skills with others in the hall… you get the picture. Again quoting David McRaney, “You commit the Texas Sharpshooter Fallacy when you need a pattern to provide meaning, to console you, to lay blame.”

Of course, the sales manager does not have “proof” Joe Bloggs is “giving away margin” any more than that cluster of bullet holes is “proof” the cowboy is a sharpshooter. What the sales manager does have is a hypothesis – defined succinctly by Wikipedia as “a proposed (repeat, proposed) explanation for an observable phenomenon.” That data cluster of low margin sales should be the beginning of the sales manager’s line of inquiry, not the conclusion.

A wise user of business intelligence tools will proceed like a scientific researcher, who will begin with a hypothesis and then try to disprove it. Here are two suggestions to do so effectively:

  • Use the word ALL a lot. Seriously, asking new questions of the data with liberal use of the word “all” is wise; for example, how do Joe Bloggs’ ten low margin sales compare with “all” of his deals for the month (both in terms of currency and quantity); “all” sales of the same products by “all” sales reps; “all” sales to those customer(s), etc. Thinking and asking questions of the data in terms of “all” will effectively erase the cowboy’s bulls-eye target, so to speak, by focusing on relevant data in its entirety and not the original data cluster in isolation, which may well prove statistically insignificant in light of “all” related data.
  • The longer time series of data you can utilize in your line of inquiry, the better. Here’s a real world example: A college finance and budget department recognized a common assumption among faculty that administrative spending was out of control, leaving faculty to fight for whatever was left. The budget and reporting director produced a trending analysis that convincingly proved that growth rates in spending were the same between faculty and administration… over the last 15 years! This diminished the conventional wisdom and led to fact-focused planning and prioritization discussions and enhanced trust and communication between departments. (Of course, a cherry-picked subset of the data would no doubt have yielded “proof” of disproportionate growth in administrative spending).

If a co-worker makes an assertion based on some isolated data set, ask whether their assertion is supported by the entire spectrum of related data.

One final thought: A highly effective business intelligence implementation will make it very easy for the end-user to compare a data cluster with “all” related data and readily observe the significance, if any, of a selected data cluster. Now there’s some real data sharpshooting!

If you liked this post, you may also like:

Business Users Can Learn a Lot from Data Scientists

For Success, Grit Beats Intelligence! Or: Use Business Intelligence Software to Achieve Grit Goals

Business Intelligence is a Diagnostic Tool, but is the “Patient” Willing to Listen?

What do Highly User-Friendly BI Tools and European Travel Expert Rick Steves Have in Common?

Source: Rick Steves Press Room

I am a longtime fan of Rick Steves, host of the long-running PBS television series Rick Steves Europe and author of an extensive series of European travel guides. His London guide helped me make the most of limited free time while traveling to the old world on business.  I hope to return sooner rather than later!

Always keeping an eye out for interesting business analogies, I suggest Rick Steves and highly user-friendly BI tools actually have a lot in common.   For openers, Rick Steves has a knack of making European travel as non-intimidating as possible for new travelers; very user-friendly BI tools help eliminate the ‘intimidation factor’ non-technical personnel might feel when trying to use an analytical tool. 

I think the analogy between Rick Steves’ brand of travel and user-friendly BI tools goes much deeper, after reading Rick Steves’ unique travel philosophy…  Continue reading

BI/Analytics is a Key Diagnostic Tool, but what if the “Patient” Won’t Listen?

Over the years I have found business intelligence author and consultant Neil Raden to be a welcome voice of common sense, from calling for simplicity for successful BIinsisting on avoiding hype, to a healthy disdain for BI buzzwords:

One particular Neil Raden blog article really struck a chord with me. The title speaks for itself: You Cannot Fix a Broken Company by Measuring How Broken it is. Neil Raden’s post goes to the heart of what Business Intelligence is – and what it is not:

The BI industry has sort of casually sent the message that BI makes companies better… But the question is, once you expose something that needs attention, what next? As a consultant and implementer of data warehousing and BI for many years, I never really came up with a good answer…

And here is an especially important comment:

Even when we did everything we set out to do [in our BI/DW consulting engagement], when approaching management about the next phase of the operation, to help the client start addressing the problems with employee morale, high turnover, inventory snafus, poor customer service, etc., the response was usually something like, “Neil, aren’t you the data warehouse guy? Shouldn’t we get McKinsey in here to work on that?”

I empathize with Neil and this condescending ‘You’re just the techie’ brush-off from management. It also raises much more serious concern: Why would management’s first response to serious business problems be to bring in outside consultants to handle it? Isn’t that their job? Was management basically admitting they’re not up to the job of addressing these issues on their own?

One more quote from Neil’s post:

In short, if you’re involved with BI, you may have as good or even better insight into what is going on in the company, but you clearly lack the portfolio to do anything about it…

Continue reading

Business Intelligence: Think Outside the Box by Turning it Inside-Out

Source: Lifehacker.com

I just read a clever tip on the Lifehacker website on how to easily reuse sturdy electronic device boxes for shipping or storage: turn the box inside out.  Many electronic device boxes use no glue or tape at all, making the process very easy.  You end up with a like-new, “fresh” box ready for easy labeling for shipping, storage use, etc.

In fact, a shipping box company has taken the inside-out box idea further with a reusable box design that turns inside out, by enabling easy removal and reattachment one of the glued box edges.

This tip suggested to me a business metaphor: Think outside the box by turning it inside-out! After all, that’s one thing that takes place when implementing a business intelligence system: data and insights previously not visible become available for many, if not all, people in the organization to see.  That can be regarded as a boon or a liability depending on one’s perspective.  Indeed, a significant hindrance to business intelligence acceptance is the perceived loss of control over the data, and therefore perceived risk of judgment and  reprisal, of one’s department, region, product line, etc.  In her book Successful Business Intelligence, Cindi Howson quoted a senior executive who said, “Some departments don’t like [their] data being exposed…others may see they are not doing a good job…” (p. 159).

Maureen Clary wrote an excellent article for the BeyeNETWORK on proactively addressing people problems that might derail a business intelligence initiative.

Continue reading

The Best Product Marketers Are “Intelligently Disobedient”

To help avoid a wide variety of business risks and disruptions, organizations should encourage employees to be “intelligently disobedient.”

This important trait is from Bruna Martinuzzi, author of some great business books including one of my favorites, The Leader as a Mensch. She explains:

I once worked for a technology company that encouraged employees to practice what they called “intelligent disobedience.”

The concept originates from seeing-eye dogs: while dogs must learn to obey the commands of a blind person, they must also know when they need to disobey commands that can put the owner in harm’s way, such as when a car is approaching.

Intelligent disobedience is not about setting out to be disagreeable or arbitrarily disobeying rules for its own sake. Rather, it is about using your judgment to decide when, for example, an established rule actually hinders your organization, rather than helps it.

That blind conformity is more likely to be prevalent in organizations practicing one-way, “top-down” business communication.

Bruna Martinuzzi offers a number of ideas to encourage cultivating an environment of intelligent disobedience, directly applicable to effective product marketers and product managers, including …

Be aware of mind traps that lead to blind conformity. Mind traps act as mental straight-jackets, preventing you from thinking creatively and rationally. These include, for example, the “herd instinct” – relying on the fact that “everybody else is doing it.” Another dangerous mind trap occurs when a group unduly defers to the “subject matter expert” rather than challenge long-held assumptions that may no longer be valid.

Rigorous, “intelligently disobedient” debates are to be actively encouraged, while divisive arguments intended to shut down meaningful discussion all together should not be tolerated.

Decentralize some of the decision-making in your unit. If you are used to making all the decisions, allow those closest to the customer the flexibility to make appropriate decisions on the spot, including the authority to bend the rules when necessary.

Don’t surround yourself with yes-men. Barry Rand of Xerox, quoted in Colin Powell’s A Leadership Primer: “…if you have a yes-man working for you, one of you is redundant.”

Help your people distinguish between fact and conjecture. If you have one data point, you don’t have data; you have an anecdote. Conjecture can be influenced by anecdotes, assumptions and other mental scripts which don’t have a bearing on reality… Encourage people to ask questions, analyze assumptions and conjectures that may or may not be accurate.

Poor Communication can Scuttle Effective BI, Your Reputation, and a Simple Bus Ride

Several years ago I flew to and from a trade show via TF Green Airport in Providence, RI instead of Boston Logan Airport as usual.  This small airport has (or at least had at the time) one large economy parking lot with shuttle buses.

Remember Ralph Kramden? The bus driver I dealt with was the Anti-Kramden.

You were supposed to give the bus driver the number of your bus stop near your car.  Running late, I rushed to catch my departing flight and didn’t make note of the number, but I knew I had parked near a certain corner of the lot.

“Excuse me,” I said to the bus driver, “but I don’t have my bus stop number. Can you just drop me off at whatever stop is nearest to the far right corner of the lot?”

“What’s the number?” grunted the bus driver.

“I don’t have the number.  But I know my car is near the far right corner of the lot from where we are right now.”

“What’s the number?” the driver again grunted, a little louder this time.

(What…?!) “I said I don’t have the number. I’m near that corner of the lot over to your right.”

“What’s the number?”

(Is this guy for real?!) “Look, can you just stop anywhere near the far corner of the lot?”

One of my colleagues from the trade show, a TF Green regular and just as annoyed with the driver as I was, shouted out a stop number he happened to know was close to my car. The bus driver, now given “The Number,” did silently agree to stop there, his eyes forward as I walked off the bus. Note that there was no language, cultural or hearing-ability issue with the driver. He was simply locked into his own way of thinking to a ridiculous degree: no stop number, no stop.

The way a person communicates is a major component of their reputation and personal brand.  And I believe the vast majority of communication problems are caused by the personal baggage we bring to the table when communicating, known in psychological terms as confirmation bias.   Continue reading

Collective Intelligence for Business Intelligence (or: Why Kids with Big Feet have Better Handwriting) – from recent Timo Elliott Interview

Is this all the proof we need this child got an A+ in penmanship?!

Is this all the proof we need this child got an A+ in Handwriting?! Photo: A Taridona  (Flickr CC)

I came across a great interview of Timo Elliott, Senior Product Director for SAP Business Objects by Ajay Ohri (DecisionStats). Timo Elliot was employee number 8 for (SAP) Business Objects!

Timo Elliott weighs in thoughtfully on a number of different BI topics and future BI challenges, but the part of the interview that stood out for me was Timo Elliott’s comments when asked about BI and social media. Social media facilitates communication between lots of people, which could be used to enable large teams to collectively interpret business intelligence results; in other words, using collective intelligence as an aid for effective business intelligence:

Conversations and collaboration are an essential part of effective business intelligence … (W)hile it’s vital to try to give everybody access to the same data, there will always be plenty of room for interpretation and discussion. BI platforms need to support this collaborative decision-making.

In particular, there are many, many studies that show up our all-too-human limitations when it comes to analyzing data. For example, did you know that children with bigger feet have better handwriting?

Continue reading

Hans Rosling, the Guitar Hero of Data Analytics & Business Intelligence!

In May 2009 I blogged about an article by BI authority and writer Dan E. Linstedt in which Dan called for a visualization breakthrough in business intelligence, on a par with the graphical breakthroughs seen in such popular games as Guitar Hero. I suggested that “most of the ‘must have’ visual features of BI are already fairly well covered” and “this same end user demand for more realistic, graphical experiences [as for gamers] doesn’t really exist for BI.” I agreed with Dan that BI should be “used” and not be “merely useful,” but innovations were better focused on such breathtaking areas as…Excel integration. Okay, Mike!

My suggestions were, in fact, already soundly debunked by global health professor and data visionary Hans Rosling in his simply amazing 2006 and 2007 TED talks, which I have just gotten around to discovering for myself. Hans Rosling’s presentations prove a “Guitar Hero-style experience” with data is not only possible but also highly beneficial. Hans Rosling turned global health and poverty data into graphically engaging, focused, insightful, compelling, even exciting stories. I am certain you will be captivated by both presentations and be as convincingly informed by them as I was. Here they are…

Continue reading

Doing the Wrong Things Right (part 2), or: Doing and Measuring the RIGHT things Right!

Photo: mrwilleeumm (Flickr)

Photo: mrwilleeumm (Flickr)

Recently I blogged on how organizations all too often “do the wrong things right” (see part 1) due to misguided, fundamentally flawed traditional management techniques of rule enforcement and incentives. I also noted that Business Intelligence/Business Performance Management (BI/BPM) software is often misused by supporting misguided rule enforcement and incentives. In other words, many companies are unwittingly “measuring the wrong things right” to help “do the wrong things right”!

I have repeatedly pointed to Barry Schwartz’ related TED talk in past postings (here and also here) for good reason: I think any successful deployment of BI/BPM solutions must begin with full awareness of Barry Schwartz’ warning that rule enforcement and incentives will often lead workers to “stop being wise” — that is, unwittingly discourage workers from independently “doing the right thing” in a given work situation. I also believe organizations can and do unwittingly misuse BI/BPM solutions to help “do the wrong things right” by “measuring the wrong things right.”

I cited a worker behavior example in my above-linked posts. Consider a much bigger, all too real example: CEOs have been compensated quite handsomely — even grotesquely — only to see the companies in their care later implode. Why? Because those CEOs were often heavily comped on one key measure, most notably increasing stock price. Unfortunately, many a CEO “earned” treasure troves of incentives for hitting that higher stock price, only to be found out later the CEO failed to do so in a sustainable, ethical or even legal manner. It’s as if the CEO stuck his or her hand over a glass thermometer, made the measured temperature go “up” and then proclaimed, “See? The room is warmer now! Where’s my bonus?!”

OK, so how can organizations “measure the right things right” and “do the right things right”? It all starts with the company’s mission statement.

Continue reading

Doing the Wrong Things Right (part 1)

“[I know of] employees who plan ahead to sit next to their CEO on a five-hour flight to bend his or her ear, thinking it will boost their career.  Don’t do it! Again, it’s an example of doing the wrong things right, which will only harm your career.” — Jeffrey Fox

The phrase “doing the wrong things right” came up in in my phone interview with best selling business author Jeffrey Fox for an article available on this blog.  And if I recall our conversation correctly, someone really did do the ‘annoy the CEO for five hours on a plane’ tactic, thinking it would help their career. It didn’t.

Right or Wrong, Well or Poorly - by EffectiveCIO.comI googled “doing the wrong things right” awhile back and found a very relevant article Right or Wrong? Well or Poorly? by Chuck Musciano. (One quibble: for Chuck’s chart (see left) accompanying his blog post, it is the “doing the wrong things right” quadrant that should be red; doing the wrong things with relentless speed and efficiency is the worst of all possible worlds!)

From Chuck Musciano’s post:

Doing the wrong things right is often known as “paving cowpaths.”  Some awful business processes are so entrenched that they cannot be rooted out.  Discretion being the better part of valor, we choose to automate bad processes, throwing good technology at a bad system.

Agree!  But how do such awful business processes come into being in the first place?  I suggest those ‘cowpaths to hell’ were paved with good but badly misguided intentions, using misguided information.

I think the key cause to “doing the wrong things right” from both a leadership and business intelligence (BI perspective is clearly traceable back to “measuring the wrong things right.”

For example, consider an an article by business performance management expert Craig Schiff, advocating for employee incentive pay based on performance:

The holy grail of business performance management (BPM) is to compensate people based on their achievement of corporate, departmental, and individual goals and objectives. While BPM is good at measuring progress against objectives, it is of little value if it doesn’t change people’s behavior, which in turn should help improve the bottom line. Incentive compensation based on what is being measured by the BPM system is a way to do that.

It is true that setting incentives will change employee behavior, but unfortunately not as often for the better as one might imagine.  I have major concerns with the concept of widespread, heavily incentive-driven compensation based on the certain data measures tracked by a BPM/BI solution…

Continue reading