Problems with Big Data

This post has already been read 2042 times!
0 Flares Twitter 0 Facebook 0 0 Flares ×

Problems with Big DataA recent study done by CapGemini, only 27% of the executives surveyed described their big data initiatives as successful. This seems to indicate a huge gap between the theoretical knowledge of big data and actually putting this theory into practice.

So what’s the problem?

According to the study, there are 5 main problems with Big Data

  1. Too much noise in the data

Data Scientist and author of the book “Social Network Analysis for Startups”, Maksim Tsvetovat said that “There has to be a discernible signal in the noise that you can detect, and sometimes there just isn’t one. Once we’ve done our intelligence on the data, sometimes we have to come back and say we just didn’t measure this right or measured the wrong variables because there’s nothing we can detect here.” He went on to say that in its raw form, Big Data looks like a hairball and scientific approach to the data is necessary.

  1. Data Silos

Too often all of that wonderful data you’ve captured in separate, disparate units, that have nothing to do with one another and therefore provide few insights gathered from this data because it simply isn’t integrated on the back end.

The way to eliminate data silos? Integrate your data.

  1. Inaccurate Data

According to a recent report from Experian Data Quality, 75% of businesses believe their customer contact information is incorrect. If you’ve got a database full of inaccurate customer data, you might as well have no data at all. The best way to combat inaccurate data?

By eliminating data silos by integrating your data.

  1. Technology Moves too Fast

The CapGemini report is that stalwarts like telcos and utilities “…are noticing high levels of disruption from new competitors moving in from other sectors. This issue was mentioned by over 35% of respondents in each of these industries, compared with an overall average of under 25%.” In essence, traditional players are slower to move on technological advances and are finding themselves faced with serious competition from smaller companies because of this.
Paul Maritz, Pivotal Chief Executive Officer of the EMC Federation, wrote in a recent CapGemini Report that, “If you can obtain all the relevant data, analyze it quickly, surface actionable insights, and drive them back into operational systems, then you can affect events as they’re still unfolding. The ability to catch people or things “in the act”, and affect the outcome, can be extraordinarily important, valuable and disruptive.”
The ability to make snap decisions and quickly move on Big Data insights is the advantage SMEs have over large corporations.

  1. Lack of Skilled Workers

CapGemini’s report found that 37% of companies have trouble finding skilled data analysists to make use of their data. The best bet is to form one common data analyst team for the company, either through re-skilling your current workers or recruiting new workers specialized in big data.

You need to find employees that not only understand data from a scientific perspective, but who also understand the business and its customers, and how their data findings apply directly to them.

Data Integration is Key

The CapGemini report goes on to point out that data integration – or to be technical, data harmonization – is absolutely essential for getting the full advantage out of your Big Data. Data integration addresses the backend need for getting data silos to work together so you can obtain deeper insight from Big Data.

The first step to integrating your data is to ensure you’ve got clean data. Big Data Consultant Ted Clark, from the data consultancy company Adventag, said that “80% of the work Data Scientists do is cleaning up the data before they can even look at it. They’re data custodians rather than analysts. Anything you’ve done more than three times, you should automate – it might take longer the first time but the other times you will save time and focus on an analysis.”
Vanessa Rombaut, the Digital Communications Marketer at PieSync provides the following ways to clean your data

How to Clean and Maintain your Data

  1. Remove Duplicates

If you’re using multiple channels to capture data, such as through your website, customer care center, and marketing leads, you’re running the risk of collecting duplicate information. There are tools to help you remove duplicate data. For instance, if you work with Google Contacts you can merge your contacts.

  1. Verify New Data

Set company-wide standards on verifying all new, captured data before it enters the central database. Put in checks to see if the customer isn’t already in the system, or that they’re not in the system under a different name or under their email address.

  1. Update Data

Keep your data updated. You can do this by using parsing tools, which scans all incoming emails and updates contact information as it comes to hand.

  1. Implement Consistent Data Entry

Ensure that all employees are aware of company-wide data entry standards. For instance, each customer record has to have first and last names.

Originally published at


Additional Reading

Big Data Analytics for Inclusive Growth

Big Data and the Customer Loyalty: Use it or lose it


If you liked this article, we'll be happy to send you one email a month to let you know the newest edition of the MetaOps/MetaExperts MegEzine has been published. Just fill the form below.