How can Retailers Overcome the Big Challenges of Big Data?

Looking back at the past few years, retail technology trends have shown that retailers who are able to make sense of their data have been able to overcome the challenge of evolving customer expectations.  Retailers now have unprecedented access to massive amounts of operational and consumer data, yet face significant challenges putting this data into action.  To remain competitive, retailers will need to overcome these challenges to make decisions based on data at every level of their business – from increasing sales, boosting operational efficiency, to optimizing promotions and loyalty programs.

What Exactly Is Big Data?

Nearly 20 years ago (in 2001), Gartner coined the term Big Data and defined it as, “Data that contains greater variety arriving in increasing volumes and with ever-higher velocity.”  This is referred to as the three Vs of Big Data.  However, two additional Vs have been added over the years: veracity and value.  Together, these five Vs outline the five distinct dimensions of big data.  Put simply, big data refers to large, complex data sets, often from new data sources that can be interpreted to address or uncover business problems or opportunities for improvement. 

Retailers stream a massive volume of data across their supply chain and through a collection of diverse customer touch points across omnichannel operations.  And yet, few retail enterprises have strategically reaped the benefits of big data.  Instead, the majority of leading companies face challenges in collecting valuable data, making sense of their data, and using it to increase overall business value.

The Four Biggest Big Data Challenges for Retailers

Dirty Data

When data is filled with mistakes, errors, duplications, and/or incomplete values, it is known as dirty data.  According to The Data Warehouse Institute (TDWI), dirty data costs US companies around $600 billion every year.  If data is not accurate, up to date, well organized, and easily understood, both the veracity and value of big data drops drastically.  Moreover, it only takes one or two people saying, “I don’t trust the data,” to invalidate a report and tank a whole project.  In many ways, a database full of inaccurate data is worse than none at all.  High quality insights from data analysis require clean, high quality data.  Retailers must adopt a strong validation process that focuses on enabling access to all the data needed to make informed decisions without sacrificing data integrity. 

Data Silos

Retailers have long struggled with data silos or repositories of fixed data, each belonging to a department or business function that is isolated from the rest of the organization.  These silos have made it difficult to manage, analyze, and activate data which hampers the identification of valuable opportunities for enterprise-wide improvement.  These silos are the reason you need to manually crunch numbers from multiple sources to produce any sort of comprehensive reporting.  Often, the task of breaking down these data silos falls to a data champion, whose goal is to advocate for the importance of collecting and leveraging data, supporting the use of data analytics in decision-making, and actively promoting data analytics best practices throughout the organization.

Untangling the Big Data Hairball

The big data hairball refers to the massive amounts of raw, unstructured data generated every day.  Unstructured data is more or less useless until it can be transformed into an existing data model like structured or even semi-structured data can.  Untangling and making sense of this data is a difficult but necessary step in becoming a data-driven organization.  To add to the hairball, the speed that data is being generated is increasing exponentially.  Currently, there are 2.5 quintillion bytes of data created every day, with over 90 percent of the data in the world generated in the last two years alone.  However, it’s not the amount of data that makes big data a big deal, it’s the ability to separate the signal from the great volume of data noise to make real use of it.

The Data Science Talent Gap

Since nearly every major company is actively looking for data science talent, the demand has rapidly outpaced the supply of people with the required skills.  Data science and analytics jobs can take 53 days to fill,  13 days longer than the US market average, and 78% of US organizations report problems filling data-analysis positions.  The skills gap and long hiring times can cause project delays and increase costs, hindering enterprise data analytics efforts.  But a number of recent technology trends, including the rise of the citizen data scientist, have changed how companies acquire and apply data science capabilities, providing some options for alleviating the big data talent bottleneck.

Overcoming the challenges associated with Big Data means delivering insights to the right user, at the right time, in a way that they can understand and act on. The right data analytics and/or business intelligence tools can be an incredible help in creating a data-driven organization and overcoming common big data challenges.  To learn more about how to embrace enterprise-wide data analytics to improve decision-making, read our whitepaper, "Vertical-Focused vs. Generic Business Intelligence."