Thursday, September 5, 2013

Big Data: The Race Is On


One of the buzzwords in the IT world apart from cloud computing technology is big data. For a lot of people, (the concept) is hard to comprehend because no matter the size of the data, data has been around us all the time. Why now in a sudden, big data becomes significant and gathers so much attention? And how big is one’s data to be considered as BIG data? 

Big Data
Big data is claimed to be valuable in improving service and user experience, and sometimes, even to refine prediction. So, as a not-so-big company, what is the size of data we have? If it can be considered as big data, how can we make use of it? I pondered a little and told the cloud surveillance development team on how we can tap on big data to improve our service. 

Currently, we are using a customer's last page view as default for their next login page. This is hoped to reduce the unnecessary clicking and to bring the customer straight to the "intended" page. We went ahead with this decision only based on the assumption that customers would always want to visit their last logout page the next time they login.  However, this built-in mechanism may not be necessarily right even when we observed a high hit rate on the relevant pages. 

Hence, I suggested that they crunch individual customer’s data for a period of one month based on two parameters, frequency of page log and total time spent on a page, and strike a balance between the two.  Therefore, if a customer’s behavior changes, the intended first view page would also change accordingly to improve user experience. The method is considered a deployment of the big data concept on a smaller scale.

Big Data
According to Viktor Mayer-Schönberger and Kenneth Cukier in their book, Big Data, “Data was no longer regarded as static or stale, whose usefulness was finished once the purpose for which it was collected was achieved, such as after the plane landed (or in Google’s case, once a search query had been processed). Rather, data became a raw material of business, a vital economic input, used to create a new form of economic value. In fact, with the right mindset, data can be cleverly reused to become a fountain of innovation and new services. The data can reveal secrets to those with humility, the willingness, and the tools to listen.”

In the past we depended on samplings when dealing with large populations and numbers. This was due to the constraints of tools to collect, organize, store and analyze data, which often times did not provide enough confidence to the participants and led to many cases of self-censorship. But with the era of big data, where N=all, using all the available data lets us see details we never could when we were limited to smaller quantity of data.

When we can make use of data effectively, some market surveys become irrelevant because now you realize that the answers to your questions are somehow buried deep down somewhere in the pool of your big data, something that you possess all these while. If you know how to excavate the dormant value, it’d give you much accurate findings than those sophisticated pricey market surveys.   

Big Data


For example, when you dine in a restaurant, a waitress approached you with the sweetest smile she could muster and passed you a survey form to fill. If the service column were filled with “excellent”, she would have a chance for a pay hike. In such a case, you would oblige hoping that your response helps her to get a raise. 


In a market survey however, we usually tend to ask why in our effort to understand the preference of our customers. But when we’re dealing with big dataset, it’s more towards what, rather than why. The big data tells you outright about what is happening, and what you should do next. 

That’s how the Internet giants, Google, Amazon and Facebook become humongous and invincible. For them to improve their services in all aspects; they dig into their own data, find the logic, extract the hidden value, and act according to pattern of the data. With astronomical quantity of data, the numbers speak for themselves, without any need of conceptual model and hypothesis.

Big data marks the beginning of a major transformation, from quantitative to qualitative change. Since we are providing two cloud services, TimeTec as a cloud based workforce management system and IPCamera as cloud surveillance system; we will start to collect data in a larger scale. How can we differentiate ourselves among the competitors? This relies on how data holders open up their big-data mindset, continuously improve the algorithms, innovate the services and enhance user experience too.

When words, locations and interactions are considered data; when one can dataficate almost everything; the big data race is on. The earlier a web firm can capture and reuse its data, the accumulative of data exhaust would turn into a huge competitive advantage for the company, and raises the barrier of entry against its rivals.