Internet of Things and Big Data

Imagine a world where everything you use could capture your interactions and transmit that information to servers somewhere on the web. Imagine that toasters, washers & dryers, cars, refrigerators, phones, digital watches, TV sets, blenders, coffee makers, game consoles, smart meters, etc, recorded your usage, actions and preferences and fed that information to their home servers. This is not an implausible idea as there are already more than a few devices (Cable and Satellite box sets, game consoles) that already behave just like that. But in our not too distanced future all of our powered devices will follow that model. Internet of Things (IoT) is about physical objects reaching the Internet on their own. By using technologies like RFIDs, sensor networks, short-range wireless communications and LANs physical objects become smart devices that would make periodic calls to their data centers either to report on their status or transmit the latest batch of locally captured data. This model will gradually change the way we view and interact with the physical things around us and will offer great new opportunities to manufacturers, distributors, service providers, retailers and users alike.

For example, it is not hard to image that in the future appliances will schedule their own service calls, preempt real-time measures to remediate an issue or relay users’ inputs and usage data. Equally property-casualty insurance industry could use the information compiled by a person’s automobile’s IoT to more accurately assess and price a particular policy and prevent fraudulent claims by looking at the data provided by policy holders’ driving habits (Progressive Insurance has introduced Snapshot device which plugs into a car’s diagnostics port and keeps track of a person’s driving). Using information gleaned from the Internet of Things data, product design could be helped by examining user interaction patterns, supply-management and distribution logistics could be optimized by reviewing adoption rates - how quickly a device goes from manufacturer to distributor, retailer and finally getting connected to a home server from end users’ domiciles – and health care delivery could be made more efficient and effective by making common portable devices, like a cell phones or digital watches, health monitoring tools that could transmit bearer’s vital signs information. Welcome to the Internet of Things world.

Data Challenge

It is expected that by 2020 about 50 to 100 billion things will be connected to the Internet. These smart objects will be using common connectivity devices to link to the Internet and exchange messages with arrays of dedicated servers, potentially generating a staggering 35 ZB/year of data. If you were wondering, a zettabyte (ZB) of storage capacity is 10 to the power of 21 bytes. Just to get a better perspective, as of 2009 the entire world wide web was estimated to contain close to 500 exabytes of data, which is half of a zettabyt. IoT will produce lots of data. IoT data won’t be at all different from that of Big Data which is heterogeneous, dissimilar, unstructured and noisy. But more remarkable will be the growth rate of IoT data. Currently the amount of data generated by social media, transactions, public and corporate entities are scaling faster than computer resources allow.

Add to that challenge the volume of data expected to be generated by IoT and it becomes clear that traditional solutions of data storage and processing could hardly be applied to ingest, validate and analyze these volumes of data. IoT data will be granular in nature and will have information about locations, temperatures, patterns and behaviors. In the IoT world the challenge will be to finding ways to analyze and capitalize on this information quickly and in near real-time. It should come as no surprise that organizations that are able to make business decisions using this data will have a strategic advantage over their competition. But as mentioned before to do so requires a robust computing infrastructure and that will not be cheap. IoT data, like that of Big Data, is unstructured and here lies one of its more significant challenges.

To address this issue effectively it is important that manufacturers, distributers, service providers and retailers agree on a simple, generic and textual format to build and describe IoT data, similar to XML markup language. This investment in standardization will affect the entire IoT/Big Data processing pipeline – data acquisition, extraction & cleaning, integration and aggregation and finally analysis – as some existing tools could be used to clean and transform this data more quickly and cheaply to a format best suited for analytics applications.

Technology challenge because of the exponential growth rate of IoT data, the need for a computing infrastructure that can balance performance, energy efficiency and cost becomes important. To scale successfully for the data growth patterns envisioned by IoT IT departments must prepare for hyperscale computing environments with thousands of computer clusters that can support scalable and predictable frameworks which process large data sets. This new computing environment is best achieved in the cloud where sharing of very large expensive clusters has become economical. Another benefit of cloud computing is its modular architecture where horizontal scaling can be achieved quickly and easily. A central challenge in IoT data processing is the limitations inherent in conventional computing resources. Largely due to power constraints, processor clock speeds have stalled and instead processors are being built with higher number of cores. As a result application developers must now be concerned with parallelism within a node, as well as across nodes. Because this architecture is very different – more shared processor caches and memory across cores - techniques for inter-node processing algorithms do not work for intra-node parallelism. As such application developers must reevaluate how they design, build and field data processing applications.

Another technology challenge is the traditional I/O systems which for decades were designed and optimized for sequential I/O performance rather than random access. But with advent of solid state drives this performance limitation is disappearing and hard disk drives are being replaced by the newer generation of I/O systems which in turn necessitate IT departments to rethink how they design and implement database systems for large data set processing.

Since an IoT/Big Data computing infrastructure requires large investments it is even more important that IT departments better manage their operations and resources. A user driven application optimization will fail to meet performance goals of process intensive jobs cost-effectively. Such architecture requires a holistic optimization approach. Remember that as jobs get bigger system failures become more frequent.

Privacy Challenge

Should an appliance’s usage patterns, say how and when you use your dishwasher or your choice of washing liquid, be available to the appliance manufacturer? Is it legal for a gaming console to relay your favorite game’s information to the console maker? And should that information be shared with the game publisher? And if you passed a red light and your vehicle recorded your mishap is it ethical for your insurance provider to use that information to adjust your premium and reassess your policy? Should the insurance company pass that information to the local law enforcement agency?

Data privacy and security are not new topics and certainly are not exclusive to data generated by IoT. What is different here is the question of privacy “contract” between machine and human. We have come to accept that some of our smart devices, such as computers or phones, do capture our interactions with them – even the ownership of this data is a matter of conjecture - but we’ve never had to consider an appliance or a vehicle with that kind of potential.

So the question becomes who owns the data generated by IoT and what is the acceptable or legal usage of this data. For some types of information there are already legislation to limit their usage and distribution, such as a person’s medical or financial information. But there are no legal remedies for controlling access to data generated by IoT. Eventually legislation will catch-up with technology but for now we can only assume IoT information won’t be used to the disadvantage of consumers.