Have you heard about Big Data? Most likely yes! You were told that Big Data is going to provide all kinds of answers. It was going to allow organizations to make smart data-driven decisions, increase their efficiency and give them a sharp competitive edge. It was going to enable health diagnosis and improve medical treatment. To some degree, that is possible, but it’s far from the current reality.
Big Data is the combination of technology, processes and people (technical and business). The most complex part among these three pillars is people, where human skills are needed to understand the data and make smart decisions.
Data is at the base of the knowledge pyramid. It is the raw material for understanding, but out of context, Data is not evidence. When we put data into a context and ask the right questions, we get useful Information. When we interpret that information and get answers, we gain Knowledge. When we use that knowledge in different situations and synthesize the outcomes, we reach Wisdom.
Currently, when we talk about Big Data, we refer mostly to technologies that collect and process the data. There is a key missing link between Big Data and Big Wisdom. If we don’t find a way to curb this trend, we will continue to gather more confusing data and gain less significant insights.
GAP BETWEEN BIG DATA & BIG WISDOM
It is naive to think that getting value from Big Data is as simple as collecting data, running a program and getting insights automagically. In reality, it usually is much more complicated than that and might require human intelligence. People can help by developing powerful algorithms, but they also have to figure out how to make the best use of what the data is telling them. Sometimes it’s straightforward, but often it’s not.
There are cases where data can give a direct answer to our questions without human intelligence. For instance, predictive analytics can accurately monitor network status and users behaviour. The network operations can be automatically optimized and marketing campaigns can be targeted to the right customers.
There are also several cases when you cannot get a definitive answer, especially when it comes to human behaviour or security. For instance, soft skills such as communication skills, passion, commitment, are hard to measure. Medical decisions can also be risky to be taken only based on data.
To fill the gap between the raw data and the useful insights, Data Scientists become indispensable. Their role is to process data, conduct analysis and find answers to make smart and fast decisions.
The Moneyball movie is a great example of this problem. Peter Brand (Jonah Hill) built metrics that provided valuable information that helped the coach to identify a good baseball player to fill a particular role. Even though the stats that were collected had accurate insights about the performance of the players, they couldn’t accurately predict the performance of the player in the future. It was simply impossible to take into account some human factors that are difficult to measure. For instance, factors like how comfortable a player plays under pressure, how well he practiced the week before the game, or how well he gets along with teammates. All of these factors matter in the decision process and are much harder to quantify.
The data scientist can develop sophisticated algorithms and make their results available for those who require answers. However, data scientists can quickly become a bottleneck when it comes to analyzing data for complex problems. This problem is amplified when decision makers need answers quickly that only data experts can provide. This is where machines can shine and provide insights that would have taken years for a Data Scientist to figure out on their own. Machine Learning will continue to progress to reach higher levels and make more decisions for us. In the meantime, we need to find a balance between the use of human intelligence and technologies.
The other issue that affects Big Data is the quality of the data collected. Given the shear enormity of the data being collected, many organizations are not aware that the data they are collecting is meaningless or worse yet, is of bad quality. This is another area where the right human touch is essential. If you compliment your Big Data with senior Data Scientists who can work with key quality assurance staff to ensure that the data being collected not only is meaningful but is of the highest quality, then you are ensuring that your Big Data will be providing you with useful insights in your quest for Big Wisdom and not garbage.
We are still in the early stage of the digital era. With the rise of the Internet Of Things (cf. Figure 1), we will have tremendous amounts of data. If we don’t focus on understanding the data, we will be left with a big pile of confusing data with few meaningful insights.
Big Data technology has gotten a lot of attention lately, and is currently moving faster than our ability to wisely understand the data. Big Data hasn’t yet translated into Big Wisdom, and we still have a huge gap between expectations and results. If we want to build smarter organizations and societies, we need to switch the conversation from Big Data to Big Wisdom.