Big Data Is Getting Bigger: Are Retail Companies Ready?

One of the interesting transitions I have witnessed in Retail has been the significant change in all dimensions of data. Not only more granular but also more categories of data have become more accessible to planning functions.

Retail companies are the closest entities to the end-customer of supply chains: consumer. This setup requires collecting tremendous amounts of very detailed transaction data, a must in retail business. Consider the following situation:

• Customer: For this very random reason, I’ll rightfully return this product that I don’t remember from which store I bought.

• Retailer: Ooops, we cannot see how much we charged you for this product, so we’ll credit the ticket price back to you.

Hence, my observation of big data getting even bigger is due to this proximity to the consumer factor. Yet, what I argue is that this transition is parallel to what is happening to our society as more computational power is now available. Smart phones, social networks, image processing, cloud computing are now few keywords that are frequently pronounced in our everyday lives. Thus, what companies do is to enjoy what Moore’s Law predicts.

The big data transition is a great opportunity for a manager who knows how to utilize it. The possibility of localized assortment planning with reliable store-SKU forecasts, localized pricing, demand planning with social networks and web-based trends are now very real. The limitation is neither the analytical capability nor the existence of reliable data, but legacy planning systems (which will eventually be retired).

It could also be a curse if all you know about data is limited to spreadsheet programs. My humble observation is that there is now an ever-expanding gap: the rate of increase in the data very much exceeds the spreadsheet capabilities of individual users. Getting the big picture could now be a difficult task. Not knowing what to do with so much data, the risk is to either get stuck in an outlier case or draw conclusions based on a limited sample.

Given the increasing gap, the traditional way of building IT solutions based on ‘business requirements documents’ and restricted interaction is no longer viable. What big data requires is cross-functional and data-capable analytical teams that operate as intermediary functions between business and IT organizations. This is not a team simply put together from ex-business and ex-IT folks, but data scientists, optimization experts, experienced data and business analytics consultants that could unleash the capabilities of SQL and spreadsheets together. Such teams not only facilitate discussions between the two organizations but also make the tool design process more interactive with rapid experimentation of ideas. In the end, no one really knows what assumptions and models about data would consistently work.

Most companies are organized as silos, so are their datasets (though IT might be storing these tables in the same server). Thus, in addition to enhanced IT-business interaction, using data as the common ground, these analytical teams could work with different business functions that do not communicate (at least not enough to impact business). For example, the inventory management teams would use sales and inventory data, but they may not know or may choose to ignore store traffic trends that consumer insights teams usually track. Yet these two sets could well be related: a decreasing sales trend in a well-inventoried store could be due to an assortment problem, even if there is sufficient traffic. Nevertheless, looking into silos of data will not help break the cyclic nature of retail planning: sales drop → buy less inventory → sales drop.

Usually, IT organizations are not seen as centers of innovation. However, big data presents a certain technical challenge to business units. Cloud based solutions may help individual business functions maintain their own small data warehouses, yet a holistic approach demands IT’s greater involvement. My proposed solution of intermediary analytical teams would be best destined for success if they can easily access IT’s computational capabilities.

In summary, big data is now more accessible, and companies continuously explore new ways of using these datasets to increase profitability. In particular for retail, these opportunities are far greater than any other industry due to their proximity to consumers and availability of structured data (social networks, CRM, POS, inventory, traffic, e-commerce). Yet, integrating all this valuable information into a predictive planning process is a difficult task, which requires an interaction and engagement much closer than the traditional IT and business relationship. Analytical consulting teams with enhanced data-capabilities who could facilitate and guide this interaction are now more important than ever in achieving this goal.

Özgür

Originally posted: February, 2014