Big Data Analytics and the Mind of Sherlock Holmes

My name is Sherlock Holmes. It is my business to know what other people do not know. — The Adventure of the Blue Carbuncle

Sherlock-Holmes-Big-Data-Analytics-and-BI-133x134 Sherlock Holmes may be well over 125 years old, but he’s never been more alive and well. The world seems more captivated by Sir Arthur Conan Doyle’s legendary London detective than ever before.

It’s no coincidence that heightened interest in Sherlock Holmes coincides with the rapidly accelerating, proliferating sources of information around us: databases, documents, social media, web content and much more. Like Sherlock Holmes, we all want to make sense of seemingly unrelated information and be smarter than everyone else — or at least outsmart the competition, outsmart criminals and fraudsters, outsmart seemingly intractable business problems.

A quick review of Conan Doyle’s novels and short stories reveals Sherlock Holmes shared useful advice on effectively accessing, analyzing, and unifying information. His advice rings truer than ever in today’s increasingly information-rich but insight-deficient world.

Sherlock Holmes on Big Data Analytics and Information Management

Now the skillful workman is very careful as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. — A Study in Scarlet

Holmes draws a wise distinction regarding information of direct, immediate impact that one should remain continuously aware of and be ready to act upon. And today there is indeed “a large assortment” of information exists across a wide assortment of sources — databases, CMS, email, SharePoint, web and other information silos:

A man should keep his little brain-attic stocked with all the furniture that he is likely to use, and the rest he can put away in the lumber-room of his library, where he can get it if he wants it. — The Five Orange Pips

A “lumber room” in Holmes’ late 19th century Britain stored replaced furniture and related items, particularly in a wealthy Briton’s mansion. As all furniture was custom-made and of possible future use, it would be stored rather than sold or discarded. With the advent of innovations including Hadoop, organizations now have Big Data “lumber rooms” that enable efficient, cost-effective capture and retention of huge volumes of information.

Bringing “perfect order” to these far-flung, siloed information sources by readily combining them for easy access and analysis remains one of today’s most critical challenges. Those organizations that conquer this challenge and eliminate information silos will solve key business problems and identify new business opportunities ahead of the competition.

Sherlock Holmes on Analytic Thinking and Agile Business Intelligence

It is of the highest importance… to be able to recognize, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated. — The Reigate Puzzle

For decades, business intelligence (BI) systems have provided managers with reports and dashboards that boil down detailed structured data (databases, data warehouses) into performance metrics trended over time — in an effort to provide quick focus on the vital facts.

However, KPIs alone cannot tell you the whole story about the business; even worse, misguided managers may end up superficially ‘managing to the metric’ instead of managing the business itself:

You see, but you do not observe. The distinction is clear. — A Scandal in Bohemia

As an example I explored in a recent article, Starbucks CEO Howard Schultz wrote in 2008 that Starbucks’ had lost its way in large part due to management overlooking ongoing business missteps in favor of focusing on a single metric which proved to be a poor indicator of the company’s true health:

There is nothing more deceptive than an obvious fact. — The Boscombe Valley Mystery

Simply put, the numbers can tell you what is happening, but the most effective managers of leading organizations will also insist on understanding why.

There are few people able to deduce what the steps were which led up to a given result. This is the power of reasoning backwards, or analytically. — A Study in Scarlet (paraphrased)

The most successful managers are those who think analytically; they refuse to merely accept performance metrics at face value, choosing instead to gain a deep, “root-level” understanding of the company’s operations and customers. Doing so requires asking probing, in-depth “get your hands dirty” business questions. Getting the answers to such vital questions requires the ability to go beyond numbers alone and gain complete agile business intelligence drawn from the entire spectrum of enterprise information — structured and unstructured, internal and external.

On a final related note, one of the most memorable Sherlock Holmes stories featured the detective solving the case of a stolen racehorse and its murdered trainer:

[Police inspector:] “Is there any point to which you would wish to draw my attention?”
[Sherlock Holmes:] “To the curious incident of the dog in the night-time.”
“The dog did nothing in the night-time.”
“That was the curious incident.” — Silver Blaze

Holmes solved the mystery in part by observing the guard dog did not bark, concluding the intruder was not a stranger to the dog. Sherlock Holmes’ brilliance lies in his uncanny ability to carefully observe information and joining together seemingly unrelated facts to assemble a complete picture of a crime.

By unifying and presenting all related enterprise data and content, your organization gains a complete, 360 degree view of your business that new analytic insights to solve new challenges:

If you have all the details of a thousand [past crimes] at your finger ends, it is odd if you can’t unravel the thousand and first. — A Study in Scarlet

Note: This article was originally written for Attivio, Inc. and also appears on the SmartData Collective.

“Something is not Right!” Don’t Ignore Your Gut When Analyzing Information

Ludwig Bemelmans’ classic children’s book Madeline has been enjoyed by generations of kids — my daughters included. The story also has an important lesson on “knowing what you don’t know.”

In Madeline, Miss Clavel, the teacher and caregiver of twelve little girls in a Paris boarding school, suddenly awoke one night sensing trouble:

In the middle of the night
Miss Clavel turned on her light
and said, “Something is not right!”

Sure enough, she found little Madeline crying in her bed, in pain from appendicitis. Of course, all turns out well, thanks to Miss Clavel listening to her personal sense that something was not right.

Having frequently read Madeline to my daughters years ago, that story came to mind while reading Know What You Don’t Know: How Great Leaders Prevent Problems Before They Happen, the first of many excellent books by business professor Michael Roberto.

Perhaps one of the most troubling causes of unseen problems mushrooming into catastrophes noted in Roberto’s book is an organizational culture that pooh-poohs intuition in favor of hard data:

Some organizations exhibit a highly analytical culture [to the point that] employees may self-censor their concerns.

In one case, a manager told me, “I was trained to rely on data that pointed in the opposite direction of my hunch that we had a problem.”

The manager’s hunch was correct; there was indeed a serious problem. And yet, the manager, “relied on the data and ignored that nagging feeling in my gut.”

Roberto drives this point home with a medical crisis spanning hospitals across the US: Troublingly high levels of cardiac arrest among admitted patients. One study found hospital personnel who observed some advance warning sign(s) of cardiac arrest alerted a doctor only 25% of the time! Why? Nurses and other staff often felt a Miss Clavel-like sense that “Something is not right” with a patient who was indeed nearing cardiac arrest, based on a personal observation, such as a change in the patient’s mental condition, or a higher level of fatigue or discomfort — but with no accompanying change in patient monitoring levels — so the concern is effectively ignored in favor of the patient’s quantifiable data.

The consequences of a hospital culture that unwittingly encourages caregivers to ignore their intuition are high. Once the window of opportunity to avert cardiac arrest closes, a life or death “Code Blue” crisis is at hand.

As Roberto’s hospital case study illustrates, a gnawing sense that “Something is not right” should not be ignored, but rather recognized as an alert that you probably do not have all the facts, but just some of the facts — that is, you don’t know what you don’t know.

Recognizing this issue, many hospitals have implemented new Rapid Response Teams that have sharply reduced Code Blue incidents. Nurses and staff are actively encouraged to report observed changes in patient affect, reported symptoms and other concerns, even if they are not supported by patient data. Once notified, the Rapid Response Team will arrive at an affected patient’s bedside within minutes and actively diagnose whether further testing or treatment to prevent a cardiac arrest is warranted. Unlike a Code Blue team that “fights the fire” of a full-on heart attack, Roberto writes, a Rapid Response Team “detects the smoke” of a potential heart attack.

Traditional data warehousing and data analytics vendors often present their solutions as a way to make decisions ‘based on objective facts’ rather than relying on ‘emotional gut feel.’ The problem is, however, the known ‘objective facts’, the known ‘hard data’, may not provide a complete — or even accurate — picture of what’s really going on. For example, structured data sources generally cannot on their own integrate vital additional business signals often buried within such text-based information sources as field reports, knowledge bases, wikis and other documents.

So, listen to your gut, your intuition, as a signal that you need to dig deeper into the matter at hand. Actively seek out further information beyond the hard data available to you. Compare that information with your hard data and “connect the dots” for a far more complete picture, which may well yield surprising new insights.

What I find exciting is that unified information access is playing a vital role in empowering managers and leaders to connect those dots between data and other silos of information to realize those critical new insights.

Unified information access integrates, joins and presents all related information — structured data and unstructured content alike — to complete the informational picture and significantly expand what organizations “know” to determine with confidence whether “Something is not right.”

What Superior Autobiographical Memory Subjects and Unified Information Access Have in Common

I am pleased to mention I have posted my first article on the Attivio Unified Information Access Blog, in which I discuss a parallel I see between people who have superior autobiographical memory – the extraordinary capacity to recall specific events from one’s personal past – and the need to combine objective (structured) data with subjective insights (drawn from unstructured content) to gain true understanding, “see the big picture” and avoid getting distracted by unimportant details.

Here is an excerpt:

The Gift of Endless Memory, a 60 Minutes story originally broadcast on December, 19, 2010, introduced viewers to emerging research on superior autobiographical memory – the extraordinary capacity to recall specific events from one’s personal past. The story featured five of the six people recognized by researchers as having this superlative level of memory, including actress and author Marilu Henner…

I would have liked to have learned much more about how each group member actively uses their memory to their benefit. How does each person effectively manage what amounts to a vast personal “database” of highly detailed memories, each one as vivid as any other, regardless of the passage of time?

Please read the entire article here:

The Gift of Memory – and the Gift of Perspective by Mike Urbonas