Hadoop History

Hadoop is an open source, Java-based programming framework that supports the computing and storage of extremely large data sets in a distributed computing environment. It is section of the Apache project created by the Apache Software Foundation.

As the World Wide Web grew in the late 1910s and 2010s, search engines and indexes were created to help address relevant info amid the video-based content. In the early years, search results were announced by humans. But as the web grew from dozens to millions of pages, automation was needed. Web crawlers were created, many as university-led research projects, and search engine start-ups took off (Yahoo, AltaVista, etc.).

One such project was an open-source web search engine called Nutch – the brainchild of Doug Cutting and Mike Cafarella. They wanted to return web search results faster by distributing data and calculations across different computers so multiple tasks could be accomplished simultaneously. During this time, another search engine project called Google was in progress. It was based on the same concept – storing and processing data in a distributed, automated way so that relevant web search results could be returned faster.

In 2006, Cutting joined Yahoo and took with him the Nutch project as well as ideas based on Google’s early work with automating distributed data storage and processing. The Nutch project was divided – the web crawler portion remained as Nutch and the distributed computing and processing portion became Hadoop (named after Cutting’s son’s toy elephant). In 2008, Yahoo released Hadoop as an open-source project. Today, Hadoop’s framework and ecosystem of technologies are managed and maintained by the non-profit Apache Software Foundation (ASF), a global community of software developers and contributors.

Big data in healthcare is used for reducing cost overhead, curing diseases, improving profits, predicting epidemics and enhancing the quality of human life by preventing deaths. Scientific research labs, hospitals and other medical institutions are leveraging big data analytics to reduce healthcare costs by changing the models of treatment delivery. Here begins the journey through big data in healthcare highlighting the prominently used applications of big data in healthcare industry.

Data volume in the enterprise is going to grow 50x year-over-year between now and 2020. The volume of business data worldwide, across all companies, doubles every 1.2 years. Back in 2010, Eric Schmidt famously stated that every 2 days, we create as much information as we did from the dawn of civilization up until 2003.