Data and Information Systems

Big Data is used to mean a massive amount of both structured and unstructured data that is so huge it is difficult to work on it using normal database and software techniques. In most enterprises the amount of data is too big or it moves too fast or it exceeds current processing power. Big data can be analyzed for insights that lead to better decisions and strategic business moves. While the term “big data” is relatively new, the act of storing large amounts of information for eventual study is quite old. Big data is associated with the three v’s: Volume. We collect data from business transactions, social media and information from sensor or machine-to-machine data. In the past, storing it would’ve been a problem – but new technologies like Hadoop have made it easy. Velocity. Data streams flows in high speed and must be dealt with in a timely procedure. RFID tags, sensors and smart metering are driving the need to deal with torrents of data in near-real time. Variety. Data comes in all types of forms structured, numeric data in traditional databases to unstructured text documents, email, video, audio, stock ticker data and financial transactions. And two more factors:Variability.

In addition to the high speeds and different types of data, data flows can be highly inconsistent with rises in periods. Is anything getting viral on the internet?High data loads can be tough to manage and tougher for unstructured data. Complexity. Nowdays data has multiple sources, which makes it tough to link, match, cleanse and transform data across different systems. But, it’s necessary to connect and correlate relationships, hierarchies and data linkages or your data can go out of control. Big Data has the potential to help companies improve operations and make faster, more intelligent decisions. The data flows from a number of sources including emails, mobile devices, applications, databases, servers,stock ticker data and financial transactions. This data, when captured, formatted, manipulated, stored and then analyzed, can help a company to gain useful insight to increase revenues, get or retain customers and improve operations.

The importance of big data doesn’t revolve around how much data you have, but what you do with it. You can take data from any source and analyze it to find answers that enable 1) cost reductions2) time reductions3) new product development and optimized offerings4) smart decision making. Big data analytics is the way of studying large and varied data sets -- i. e., big data -- to find out hidden patterns, unknown correlations, trends in market, preferences of customers and information that can help organizations make more-informed business decisions. When you combine big data with high-powered analytics, you can perform tasks such as: Finding reasons for failures, issues and defects in almost real time. Making coupons at the point of sale based on the buying habits of customer. Recalculating complete risk portfolios in no time.

Finding about fraudulent behavior before it affects your organization. On a broad scale, data analytics technologies and techniques provide a means of analyzing data sets and drawing conclusions about them to help organizations make informed business decisions. BI queries answer basic questions about business operations and performance. Big data analytics is a form of advanced analytics, which involves complex applications with elements such as predictive models, statistical algorithms and what-if analyses powered by high-performance analytics systems. As a result, many organizations that collect, process and analyze big data turn to NoSQL databases as well as Hadoop and its companion tools, including:YARN: a cluster management technology and one of the key features in second-generation Hadoop.

MapReduce: a software framework that allows developers to write programs that process massive amounts of unstructured data in parallel across a distributed cluster of processors or stand-alone computers. Spark: an open-source parallel processing framework that enables users to run large-scale data analytics applications across clustered systems. HBase: a column-oriented key/value data store built to run on top of the Hadoop Distributed File System (HDFS). Hive: an open-source data warehouse system for querying and analyzing large datasets stored in Hadoop files. Kafka: a distributed publish-subscribe messaging system designed to replace traditional message brokers. Pig: an open-source technology that offers a high-level mechanism for the parallel programming of MapReduce jobs to be executed on Hadoop clusters.