Big Data Wisdom, Courtesy of Monty Python

Monty Python and the Holy Grail

One of the best parts of the hilarious 1975 King Arthur parody, Monty Python and the Holy Grail is the “Bridge of Death” scene: If a knight answered the bridge keeper’s three questions, he could safely cross the bridge; if not, he would be catapulted into… the Gorge of Eternal Peril! 

 

Unfortunately, that’s exactly what happened to most of King Arthur’s knights…

Fortunately when King Arthur was asked, “What is the airspeed velocity of an unladen swallow?” he wisely sought further details: “What do you mean – an African or European swallow?” The stunned bridge keeper said, “Uh, I don’t know that… AAAGH!” Breaking his own rule, the bridge keeper was thrown over into the gorge, freeing King Arthur to continue his quest for the Holy Grail.

Many organizations are on Holy Grail Big Data quests of their own, looking to deliver game-changing analytics, only to find themselves in a “boil-the-ocean” Big Data project that “after 24 months of building… has no real value.” Unfortunately, many organizations have rushed into hasty Hadoop implementations, fueled by a need to ‘respond’ to Big Data and ‘not fall behind.’ (1)

The correct response, of course, is to first understand essential details behind the question as King Arthur did. Jim Kaskade, CEO of tech consultancy Infochimps, recently suggested to InformationWeek a simple yet “practical and refreshing” question to ask:

Whether it’s churn, anti-money-laundering, risk analysis, lead-generation, marketing spend optimization, cross-sell, up-sell, or supply chain analysis, ask yourself, ‘How many more data elements can you add with big data that can make your analysis more statistically accurate?’

The answer to this key question will lead to additional important questions:

  • “What variety of data sources are needed to fulfill my business case – structured data, unstructured data and/or unstructured content?”
  • “How do I correlate structured and unstructured information together?”
  • “How do I integrate data and content so our users can analyze it on demand, using our existing data visualization tools?”

There is no one-size-fits-all “Holy Grail” Big Data technology out there. In reality, a successful Big Data architecture consists of multiple components to address the unique aspects of all your disparate data sources, structured and unstructured, internal and external. Keep that in mind and show the wisdom of a king by taking pause and asking a few basic business questions to stay on the right path to Big Data business success.

 

(1) Source: InformationWeek article by Doug Henschen, Vague Goals Seed Big Data Failures.

Advertisement

Big Data Analytics and the Mind of Sherlock Holmes

My name is Sherlock Holmes. It is my business to know what other people do not know. — The Adventure of the Blue Carbuncle

Sherlock-Holmes-Big-Data-Analytics-and-BI-133x134Sherlock Holmes may be well over 125 years old, but he’s never been more alive and well. The world seems more captivated by Sir Arthur Conan Doyle’s legendary London detective than ever before.

It’s no coincidence that heightened interest in Sherlock Holmes coincides with the rapidly accelerating, proliferating sources of information around us: databases, documents, social media, web content and much more. Like Sherlock Holmes, we all want to make sense of seemingly unrelated information and be smarter than everyone else — or at least outsmart the competition, outsmart criminals and fraudsters, outsmart seemingly intractable business problems.

A quick review of Conan Doyle’s novels and short stories reveals Sherlock Holmes shared useful advice on effectively accessing, analyzing, and unifying information. His advice rings truer than ever in today’s increasingly information-rich but insight-deficient world.

Sherlock Holmes on Big Data Analytics and Information Management

Now the skillful workman is very careful as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. — A Study in Scarlet

Holmes draws a wise distinction regarding information of direct, immediate impact that one should remain continuously aware of and be ready to act upon. And today there is indeed “a large assortment”  of information exists across a wide assortment of sources — databases, CMS, email, SharePoint, web and other information silos:

A man should keep his little brain-attic stocked with all the furniture that he is likely to use, and the rest he can put away in the lumber-room of his library, where he can get it if he wants it. — The Five Orange Pips

A “lumber room” in Holmes’ late 19th century Britain stored replaced furniture and related items, particularly in a wealthy Briton’s mansion. As all furniture was custom-made and of possible future use, it would be stored rather than sold or discarded. With the advent of innovations including Hadoop, organizations now have Big Data “lumber rooms” that enable efficient, cost-effective capture and retention of huge volumes of information.

Bringing “perfect order” to these far-flung, siloed information sources by readily combining them for easy access and analysis remains one of today’s most critical challenges. Those organizations that conquer this challenge and eliminate information silos will solve key business problems and identify new business opportunities ahead of the competition.

Sherlock Holmes on Analytic Thinking and Agile Business Intelligence

It is of the highest importance… to be able to recognize, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated. — The Reigate Puzzle

For decades, business intelligence (BI) systems have provided managers with reports and dashboards that boil down detailed structured data (databases, data warehouses) into performance metrics trended over time — in an effort to provide quick focus on the vital facts.

However, KPIs alone cannot tell you the whole story about the business; even worse, misguided managers may end up superficially ‘managing to the metric’ instead of managing the business itself:

You see, but you do not observe. The distinction is clear. — A Scandal in Bohemia

As an example I explored in a recent article, Starbucks CEO Howard Schultz wrote in 2008 that Starbucks’ had lost its way in large part due to management overlooking ongoing business missteps in favor of focusing on a single metric which proved to be a poor indicator of the company’s true health:

There is nothing more deceptive than an obvious fact. — The Boscombe Valley Mystery

Simply put, the numbers can tell you what is happening, but the most effective managers of leading organizations will also insist on understanding why.

There are few people able to deduce what the steps were which led up to a given result. This is the power of reasoning backwards, or analytically. — A Study in Scarlet (paraphrased)

The most successful managers are those who think analytically; they refuse to merely accept performance metrics at face value, choosing instead to gain a deep, “root-level” understanding of the company’s operations and customers. Doing so requires asking probing, in-depth “get your hands dirty” business questions. Getting the answers to such vital questions requires the ability to go beyond numbers alone and gain complete agile business intelligence drawn from the entire spectrum of enterprise information — structured and unstructured, internal and external.

On a final related note, one of the most memorable Sherlock Holmes stories featured the detective solving the case of a stolen racehorse and its murdered trainer:

[Police inspector:] “Is there any point to which you would wish to draw my attention?”
[Sherlock Holmes:] “To the curious incident of the dog in the night-time.”
“The dog did nothing in the night-time.”
“That was the curious incident.” — Silver Blaze

Holmes solved the mystery in part by observing the guard dog did not bark, concluding the intruder was not a stranger to the dog. Sherlock Holmes’ brilliance lies in his uncanny ability to carefully observe information and joining together seemingly unrelated facts to assemble a complete picture of a crime.

By unifying and presenting all related enterprise data and content, your organization gains a complete, 360 degree view of your business that new analytic insights to solve new challenges:

If you have all the details of a thousand [past crimes] at your finger ends, it is odd if you can’t unravel the thousand and first. — A Study in Scarlet

Note: This article was originally written for Attivio, Inc. and also appears on the SmartData Collective.

Back to the Future of Business Intelligence with H.P. Luhn

When was the term “business intelligence” first coined? You might assume it was first conceived in the late 1980’s; coinciding with the initial emergence of companies offering visual analytic software, but the term was first used decades earlier by visionary IBM technology scientist Hans-Peter Luhn in his groundbreaking 1958 research paper, A Business Intelligence System.

Hans-Peter Luhn’s life work at IBM did not include quantifiable, structured data. That realm of BI would not become practical and cost effective until the advent of mainframe systems later in the 1960’s. Rather, H.P. Luhn focused on the unstructured content of his day: documents, letters, research reports and manuals. So too, the business intelligence system Luhn envisioned focused on computer-automated document processing: auto-abstraction, auto-indexing, selective dissemination of information, and information retrieval.

It’s plain to see that Luhn was well ahead of his time, envisioning critical technology components that set the stage for knowledge management and enterprise search today. And now, Luhn’s insights are more relevant to today’s business intelligence than ever before.

For example, Luhn demonstrated a keen awareness of the roles communication and collaboration play in the effective use of business information. The system would “channel a given item of information to those who need to know it” and find co-workers “whose interests or activities coincide most closely with a given situation,” using action point profiles of each person’s interests and activities. Luhn’s BI system would therefore quickly answer three vital overarching questions: what is known, who needs to know, and who knows what.

Who knows what - 3 key questions

Now over six decades later, it is remarkable how well Luhn’s insights remain a very solid, results-oriented description of BI. What is also striking to me is how early the silos separating unstructured content and structured data emerged.

Business analytics analyst and consultant Seth Grimes, who has written and presented on Luhn’s contribution to BI, sums up this issue of structured and unstructured silos well:

[For decades,] business intelligence detoured around the estimated 80 percent of enterprise information locked inaccessibly in textual form … So BI thrived crunching numerical, RDBMS-managed data… and delivered findings via tables, charts and dashboards that focus more on numbers than on knowledge.

~ Seth Grimes, BI at 50 Turns Back to the Future

Today it is widely recognized that BI based on structured data alone is not enough. Structured data and unstructured content must be integrated and harmonized together to create a complete analytic picture capable of revealing new breakthrough business insights. Again quoting Grimes, “in the last few years, BI has headed back to the future foreseen by Luhn in 1958.”

What is known? Who needs to know? Who knows what? These three timeless, mission-critical questions Hans-Peter Luhn asked in 1958 require unified information — data and content — to fully answer. And these key questions have become easier than ever to effectively answer and act upon by leveraging new digital transformation technologies capable of integrating and presenting analytic-ready data drawn from all relevant data sources, whether structured or unstructured, internal or external.