About Using Big Data: The Three Core Dependencies You Can’t Live Without

This post has already been read 3004 times!
0 Flares Twitter 0 Facebook 0 0 Flares ×

Simply put, Big-Data-driven insight will be what drives business by 2020. For every organization the core, fundamental, four benefits of using Big Data analytics for strategy-driving insight are:

  1. Customer Growth and Retention: Data-driven insight improves organizations’ ability to identify the right customers, attract them, convert them, retain them by preempting defection, and transform them into brand ambassadors that sell the organization voluntarily, All these good things can happen by allowing company teams to understand exactly what customers want, how they want it, and what they think of their experience with the company and reacting to that information on a timely basis.
  2. Profitability and Sustainability: By selling the right value to the right customer the right way-every time, helps organizations reduce the cost of returns, unwanted inventory and doing customer dis-service damage. This helps control costs, increase volume, increase margins and keep products and services fresh and in demand based on customer feedback. All of these positive effects can profoundly and sustainably impact the bottom line.
  3. Process Improvement and Change Management: Intelligent, data-driven enterprise solutions virtually eliminate the guesswork in innovation, sales and marketing and creation of the customer experience. Such solutions help to streamline those and other processes where the pathway to the anticipated outcomes is clear, concise and the outcome results are predictable. When outcomes are clear at the outset and have inspiring dividends, and team members know what pathway will deliver them, getting them to align their efforts with the goals and processes becomes easy.
  4. Product and Service Innovation and R&D: The customer is (almost) always right. While great innovators develop products and services customers want, those consumers create data that makes their needs and desires very clear, and their disappointments very tangible. This communicative customer base can virtually do away with the necessity to manage and compensate high-end innovation teams. Indeed, the data speaks for itself……and the customer.

Three Core Dependencies

The core three dependencies management must have to reap these data-driven insight benefits are:


“Data Scientists” are no longer an elite and rarified group. Now and for the future, all employees will be required to turn data into insight and intelligence. While most successful companies will have a strong IT-reliant “Center of Excellence,” the information they process through to team members will require additional analysis and action. As such, end-user analytics on multiple levels will have to be extremely sophisticated using easy-to-understand parameters and metaphors, natural language, and intensely precise visuals for quick understanding of and reaction to insights. It’s estimated that 4.4 million people trained at various levels on data analysis will be needed in the workforce by 2015 and there will likely be a shortfall of talent of at least a quarter of a million of these people. This number does not include computer science, statistics and IT trained professionals.[i]

Process Flow

Machines and people are required at every level of Big Data processing. Once a business goal is established, team members must assess what types of data are most likely to deliver the goal-supporting insights and where and how to obtain that data, format it, blend it with other data and define query parameters. In the next step, the queries filter and prioritize the data, and analysts must determine if the query and the data are, in fact, delivering information that will help the organize and devise strategies for success. If the data is delivering, they can move to the next phase, analysis and recommendation. Following that, is the important strategizing and implementing. If the data is not delivering, the first steps have to be repeated in part or in full.

When planning is generated by the insights from a successful query and analysis, a people-based activity for the most part, systems need to be built in for collecting the data resulting from plan implementation that will allow for a feedback loop and measurement of strategic impact. The resulting data goes through the entire process to deliver intelligence that allows for adjustment and improvement of strategy and improved outcomes.


The process flow requires a complex architecture of machines and machine managers that store the data, ensure it is safe and uncorrupt or dirty, sort and filter it, download it in appropriate formats at high speed, run queries, deliver insightful and highly visual analytics and much more. This complex event processing requires at least the following systems:

  • Virtualized Data Access: Simplified, this is a real-time service level that delivers data layers such as information fabric and information-as-a-service in virtual views that hide the complexity of the data itself and allow users to identify desired data. Once accomplished, it is then a matter of acquiring it through a middleware that structures the delivered data through query languages such as XQuery.
  • Extract, Transform and Load (ETL): Typically done by a specialized server into which the data chosen during Virtualized Data Access is loaded. This function cleans, formats, aggregates large amounts of data into data that is structured as much as desired.
  • Business Event Processing: In this step, an organization can ask the data a question like “what’s happening in my business that is important and I can access information about?” Business event processing relies on technologies that allow patterns and values come to light during streaming, complex event processing and business activity monitoring.
  • Natural Language Processing (NLP) Platform: Also referred to as text processing, this technology level derives meaning by segmenting and tagging words and wording and following algorithms and rules to extract text patterns, entities, files, manners of speech and the relationships they represent. This allows the information in the data to be coded, ranked and mapped, often into specific lexicons. Semantically annotated information results, often with clearly defined sentiment included.
  • Data Quality Software: These technology tools merges old data with current data and makes sure what used to be of interest is reported in real-time, changes in information are made known, and quality of data is maintained as fresh, relevant and accurate. Essentially, the data is profiled by this technology and aggregated to deliver comprehensive, insightful and often actionable information.
  • Master Data Management (MDM): At the reference data stage, the magic really begins to happen as MDM employs the parameters that make it possible for an organization to trust its data and use it for strategy and process decision making. MDM solutions look at the data and data relationships in virtual or physical depositories to create the map of what to do, when to do it, who is involved/impacted and much, much more.
  • Data-governance Tools: One of the big issues of the exponentially growing mass of Big Data is the policy/policies by which data is going to be gathered, processed, used, updated, etc. It is essential that all Big Data users have this governance process in place as, without it, chaos reigns and business intelligence gathered is hit or miss (not unlike the pre-Big Data system of educated guessing).

The vast majority of organizations are not presently capability-rich enough to handle all three dependencies in house and are in the position of evaluating whether they should, or if using third party providers will deliver bigger dividends. If your organization hasn’t begun this evaluative process, start now or the ones that have, with whom you compete, will be way out in front, and you may find it impossible to catch up.

[i] CNNMoney, The big data employment boom, 2013, http://tech.fortune.cnn.com/2013/09/04/big-data-employment-boom/

[ii] Forrester Research 2012, Craft Your Future State BI Reference Architecture


Additional Reading

MetaOps Intelligent Enterprise System Delivers The Top 10 Big Data-to-Insight Opportunities for Operational Excellence

Big Data and the Customer Loyalty: Use it or lose it 

If you liked this article, we'll be happy to send you one email a month to let you know the newest edition of the MetaOps/MetaExperts MegEzine has been published. Just fill the form below.