Big Data: Big Threat to Market Research?

Posted on Jul 29 2013 - 3:03pm by Fayaz

Before answering the question “Big Data: Big Threat to Market Research?” we really need to look at the definition of both Market Research and Big Data. This would help us understand the realm and objective of both.

Market Research (MR) is the systematic and objective identification, collection, analysis, dissemination and use of information for the purpose of improving decision making related to the identification and solution of the problems and opportunities in marketing.

“Big data” refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze. There has been a growing consensus in the industry to define big data, by three V’s-

  • Volume – the magnitude of data has to be large, in petabytes not just gigabytes
  • Velocity – the data has to be frequent, daily or even real-time
  • Variety – the data is typically (but not always) unstructured (like videos, tweets, chats)


Market Research

Image Source:

The definitions suggest big data is indeed complementary to MR. If we dig deeper we notice some points of parity and some points of disparity.

Points of parity:

  • Big Data also helps improve our decision making ability.
  • Both use data to improve our understanding of the problem at hand.

Points of disparity:

  • Data is extremely large in case of Big Data (in petabytes or more).
  • Big Data is mostly unstructured while MR has mostly structured data.
  • The decision making time in case of Big Data is very short. Following example from Forbes illustrates the point:”When you walk through the airport and they take pictures of everybody in the security line to match every face through facial recognition, they have to do that almost in real-time. That becomes a big data problem. If I am a bank and looking at a vast number of credit scores and histories, and I don’t need to provide an answer in five seconds but can do it next day, then that is not a big data problem.”

The beauty of big data and machine learning techniques is that we can learn pattern and this helps us solve our marketing problems and increases our knowledge base. There is a lot of data which could be gathered by social media, telephonic/electronic feedback. These datapoints can be data mined to obtain meaningful patterns. If we see this point in context of our discussion it means big data replacing use of some MR techniques such as panels and focus groups. Example: A car company wants to ascertain when people buy cars, let’s say, during festive season or appraisal time. It can check tweets and facebook posts tagging the company to analyse context and time of the year. Its CRM data could be used for cross verification.

We now agree big data is helpful in (or even a threat to) MR, but don’t expect it to be a silver bullet. MR deals with data collection, qualitative and quantitative analysis. Big Data could certainly help in some of these areas, but not all, especially replacing techniques like “Experimentation” and “Observation” seems difficult. Big data doesn’t help you with a problem where context or information is relatively less. Example: A cola giant wants to introduce a new flavour and wants to try out before launching it full-fledged. Here also, we cannot apply big Data. Some might say, “We could come to a conclusion that people may like vanilla cola by analysing their social data on foursquare, zomato, and facebook etc.”but nonetheless it’s not sufficient for our case. We would definitely like to experiment with the flavour, launch in a test market to know consumer reaction.

We can thus say there are occasions where we can embrace big data and leave out MR, but nonetheless we cannot abandon MR completely. The relation between big data and MR has some common intersection, but for other only one of them suffices.

Big Data

Following example illustrates cases where only one of them seems appropriate:

Example 1: A SME wants to ascertain reasons for its falling sales.

Here using big data is not possible as we have limited datapoints, and also dealing with the complexity of the problem it would be like killing an ant with a nuclear weapon. In this case, MR helps us understand and solve the problem.

Example 2: Can you predict which state might be hit by cold next? If possible this would help chemists and stockist to prepare stores for appropriate medicines beforehand.

This is actually being done by Drug maker SSP Co. in Japan by analysing tweets. This problem is outside realm of MR, it needs dynamic datasets and powerful and fast computing, i.e. Big Data problem.

Example 3: Target group used their data on shopping patterns to predict if a teen was pregnant and all this with help of big data.

Considering above points we can say that big data is not a substitute for MR, but a helping hand. At best it could be said they have their own areas of problem solving with some overlap. Big data lets you uncover “what” part of the problem, but some times it’s more important to know “why”. Primary market research helps you answer “why” behind the “what”. The two approaches are scientific in nature and at best complimentary, but not a threat to each other.

The article is authored by Manish Arora from IIT Roorkee.

If you wish to write for us then kindly check this

About the Author

Fayaz is the founder of Marketing Bloggers. Apart from the jibber-jabber he is Human by birth, Engineer by mistake, Marketer by Choice.. That says it all :)

Leave A Response

You must be logged in to post a comment.