Emerging Trends and Technologies

Over the years, since the introduction of Big Data, the amount of data that has been created grew exponentially. Referencing from the graph above, it is evident how exponential the growth of data was; with roughly a 6000% increase in data growth within a span of ten years (2005-2015). It was from 2005 to 2010 however, where the greatest surge of data can be seen - an 843% increase. Over time, the increase in percentage decreases but that does not mean data does not continue to grow exponentially. From 2015 to 2020, the data we generate will increase by 405%.

Several identified causes for the growth of data are the following: There is a significant rise in human engagement with social network and Internet of Things (IoT). There is an explosion of machine-generated data. Because of the enormous amount of data we have generated, we enter the Big Data phenomenon. What is Big Data? Big Data has been described differently by many articles. Oracle has described it as a holistic information management that includes and integrates many new types of data and data management alongside traditional data. McKinsey, on the other hand, refers to it as datasets whose size are beyond the ability of typical database software tools to capture, store, manage and analyze. While these two reputed companies offer a variated definition, they are actually not very far from each other. To summarize, Big Data is basically the data sets which are mostly unstructured and are so large and/or complex in both volume and variety that traditional data processing application software might not be enough to deal with them. Main Characteristics of Big Data It is often defined by the 3 ‘V’s namely volume, velocity, and variety.

Volume refers to data at rest. It is how much data we have - from online channels, social media platforms, online transactions etc. We usually look at the terabytes to petabytes of data. In Big Data, megabytes and gigabytes are deemed small. Velocity refers to data at motion. It is the speed at which data is generated, collected and analyzed as it is generated. The current Big Data technology we have today gives the ability to instantly analyze data. Variety is the different types of data that we use. It also describes one of the biggest challenges of data as it can come unstructured - not categorized - and it can include so many different formats such as XML, MP4 etc. Equally important characteristics to consider Besides the 3Vs, there are 3 more Vs that is worth considering when dealing with Big Data. Veracity is all about making sure the data is accurate and how trustworthy it is. This is where you would require processes to keep bad data from accumulating in your systems. It is important to set metrics around the type of data you collect and from which sources it comes from.

Variability is when your data constantly changes. It may be similar to variety but it differs in some aspect. To explain in a layman's term, imagine going to a coffee shop that offers five different blends of coffee and you get the same blend every day but, it tastes different every day. In this case, the different blends refer to variety and when you get a specific blend every day yet it tastes different, that is variability. It is important to note this ‘V’ as it can have a huge impact on your data homogenization. The last ‘V’ to take note of is value. It refers to the worth of the data being extracted. It is the end game because after addressing other characteristics, you must make sure you are getting a value from the data that is being analyzed. It is critical to fully understand that costs and benefits of collecting and analyzing data. No matter how big the volume of data you have, if it is not a high-value data, essentially it is useless. The Most Important Characteristic It is arguable. But for a company or an organization, variety is more important. It is because it is not the ability to process and manage large data volumes that are driving successful Big Data outcomes. Rather, it is the ability to integrate more sources of data than ever before - from old to new, small to big, unstructured to structured and so on. Market Drivers Big Data continues to thrive and prosper because of the following: Exponential Data Growth There is a ceaseless adoption of mobile devices.

The prevalence of mobile Internet sees consumers being increasingly connected - using social media networks as their communication platform as well as their source of information. This then leads to the enormous amount of unstructured data created that needs to be analyzed. Increased Investments in Big Data Technologies The competitive pressure on organizations has increased to the point where most traditional strategies are offering only marginal benefits. It has been proven and recognized that Big Data has the potential to provide new forms of competitive advantage for organizations. Thus, it has become crucial for them to make better decisions. This then translates into future demand for better Big Data technologies hence the investment to enhance grows too. Innovations and Developments in Software for Unstructured Data Big Data technologies have to be in enforced to aid Research and Development efforts because they have to support the growth of digital content and enable more efficient analysis outputs. Strong Open Source Initiatives The Hadoop Framework together with software components like R language and NoSQL tools i.e. Cassandra and Apache HBase is the core of many Big Data discussions. Because of its popularity and viability, other vendors launch their own versions. One example is Oracle’s NoSQL Database. IoT evolution In relation to the first point, there are billions of devices connected to the web and it will continue to increase over the years. Gartner predicted that there will be more than a hundred billion devices connected to the internet over the next years. These send out huge amounts of data that is needed to be stored and analyzed. Companies that deploy sensor networks will have to adopt relevant Big Data technologies to process a large amount of data sent by these networks. Big Data’s influence at a glance At a business level, it enables innovative and new business models as well as provide insights, thus driving competitive advantage over its competitors.

At a technical level, as data continues to grow and comes in so many formats, traditional methods alone cannot suffice. Big data relieves that burden and offers a new solution for efficient data analysis. At a financial level, data system costs continue to grow and it is much more advantageous to opt for commodity hardware and open source software which you would find in big data technologies. Why is Big Data important? Point of reference With such a massive amount of information, data is able to be shaped or test in any way that an organization sees fit. Improving businesses In business, sharing insights is a must-have ability. Big data is allowing for more effective business outcomes because collecting masses of data and finding a trend allows them to move more quickly, smoothly, and efficiently. Furthermore, it allows them to meet strategic objectives.

Problem areas elimination Such areas are reduced into a more comprehensible form or are removed before issues pull an organization’s profits or reputation. Amplifies other technology innovations Technologies to combine and interrogate Big Data have matured to a point where deployments are practical. Additionally, the underlying cost of the infrastructure to power analysis has fallen dramatically, making it economic to mine information. In turn, the data that we gather and analyze has brought us to hundreds of innovative startups bought by companies such a Google, Facebook and Amazon to name a few. AI startups grew by 40% between 2013-2016 according to McKinsey.