Beer, diapers and the big data bug

Dubai, UAE – 22 September 2013 - Individuals who have studied marketing will know about the connection between beer and diapers*. It's all about pattern recognition: how raw information can become useful when it's interpreted and applied in the right way.  According to Dimension Data, that's where big data fits into the picture. The world, it seems, has been bitten by the big data bug.

Many commentators say that big data has the potential to help organisations spot useful trends, for example, in customer purchasing behaviour. This can go a long way toward driving more targeted sales. But much of big data's value has been hyped to levels approaching the unbelievable.

Stephen Green, General Manager for Data Centre Solutions at Dimension Data, agrees. “Big data has become over-hyped by commentators without understanding its full implication. It's like ‘cloud' – and similar to cloud, its underlying principles are fundamentally changing the way businesses will be able to react to, or anticipate, business opportunities.”

But, warns Green, despite being a fashionable buzzword, the impact and value of big data should not be underestimated. “Imagine what's possible with all the new data generated from web browsing, online transactions, even tracking movements within shopping malls via mobile devices. All these are forms of big data. Yes, it's a buzzword in the ICT world – but it has the potential to fundamentally change the way organisations react to, or anticipate, business opportunities.”

In simple terms, big data refers to data sets that can't be handled easily through traditional methods, such as dedicated servers driving a traditional database or data warehouse structure (like Oracle or Teradata), and the associated analytics toolsets like Cognos that would drive interrogation and analytics. It's also called unstructured data, which can't be structured into columns and rows in a SQL or other form of database.  There are three attributes which further define a big data environment. These include:

• volume: the massive amount of data generated and collected by organisations
• variety:  the array of different types of collected data, from text, to audio, video, web logs, social media and more; and
• velocity: the speed at which data is collected, analysed and some even say ‘anticipated'.

“Big data's use lies in the ability to identify patterns from raw information - also called data mining.  The beer and diapers principle accounts for businesses that use data mining techniques to spot patterns

*The classic beer-and-diaper example is often used to illustrate the principle of associated buying patterns. By analysing cash slips, we can discover an unexpected correlation between the sales of beer and diapers. This could be because fathers on an errand to buy diapers conveniently purchase beer at the same time. This newly discovered information can be used to motivate a change in sales strategy that could drive higher sales, such as positioning the products closer together on the shop floor.

in other forms of customer behaviour.  For example, in the financial services sector, banks and insurance organisations use big data to identify fraud. This involves spotting patterns that would indicate the likelihood of fraudulent transactions,” says Green.

Green sites another example within the world of telecommunications. “A large US mobile phone operator − let's call it X Telecoms − was suffering significant customer churn across its mobile customer base. By using traditional data analytics tools and processes, the organisation was able to quantify the amount of churn quite accurately, but not the reasons for it.

“X Telecoms turned to a group of data scientists - another new buzzword - to identify the underlying cause of the churn. Using the volumes of unstructured data that X Telecoms captured every day and by writing an advanced set of algorithms, it was able to provide some interesting insights.

“Each time one person switched a mobile plan to a competing provider, five friends would closely follow, which then meant that each of those five friends would have another five friends leaving the network … in other words, a snowball effect.

“This behaviour was driven by a bundling offer from mobile companies, offering free phone calls and texts to ‘five friends'. X Telecoms quickly took action by introducing a counter campaign: every time one of its customers switched providers, it immediately sent an offer to their five selected friends, providing them with a compelling offer to renew their plans with  X Telecoms. Through that one action, it reduced churn in its base by more than 65%.”

While unstructured data can't be easily converted into actionable intelligence by traditional databases, these examples confirm that the tools for gleaning knowledge and insights from it are developing fast. Says Green: ‘At the forefront are rapidly advancing techniques of artificial intelligence, such as natural-language processing, pattern recognition and machine learning. These artificial-intelligence technologies can be applied in many fields. For example, Google's search and advertisement business and its experimental robot cars – which have navigated thousands of miles of California roads – use a bundle of artificial-intelligence tools that analyse vast quantities of data and enable instant decision-making.

These developments are ushering in massive opportunities for businesses. In turn, CIOs are coming under increasing pressure to provide the necessary tools and processes to enable a big data strategy for their businesses in order to capture market opportunities and/or prevent reputational damage.

Green points out that despite all the hype around big data, organisations don't need large investments in infrastructure and resources to start with. “Organisations can start by installing a low-cost, simple platform to gather the data, and from there, begin to identify useful patterns that would almost immediately drive returns, if followed up with proactive activity. A small investment in such a platform can be funded from the benefits gained by its use. This is possible across all business sectors where a broader range of patterns may become relevant. These could include quality control patterns in manufacturing, patient re-admittance patterns in hospitals, bookings versus cancellations patterns in travel, and many more.  Even small entry points are showing business returns that fund your business growth and allow IT to build the skills needed to take this to the next level,” he concludes.