Hadoop History

Introduction to Hadoop

Hadoop is an open-source, Java-based programming framework that supports the computing and storage of extremely large data sets in a distributed computing environment. It is part of the Apache project created by the Apache Software Foundation. This framework allows for the processing of big data across clusters of computers using simple programming models, making it a crucial component in the world of data analytics today.

Hadoop History

As the World Wide Web grew in the late 1990s and 2000s, search engines and indexes were created to help address relevant information amid the rapidly expanding content. In the early years, search results were announced by humans. However, as the web expanded from dozens to millions of pages, automation became necessary. Web crawlers were developed, many as university-led research projects, and search engine start-ups began to flourish (Yahoo, AltaVista, etc.).

One such project was an open-source web search engine called Nutch – the brainchild of Doug Cutting and Mike Cafarella. They aimed to improve the speed of web search results by distributing data and calculations across different computers, enabling multiple tasks to be accomplished simultaneously. During this time, another search engine project called Google was in progress. It was based on the same concept – storing and processing data in a distributed, automated way to return relevant web search results faster.

In 2006, Cutting joined Yahoo and brought with him the Nutch project, as well as ideas based on Google’s early work with automating distributed data storage and processing. The Nutch project was divided – the web crawler portion remained as Nutch, and the distributed computing and processing portion became Hadoop (named after Cutting’s son’s toy elephant). In 2008, Yahoo released Hadoop as an open-source project. Today, Hadoop’s framework and ecosystem of technologies are managed and maintained by the non-profit Apache Software Foundation (ASF), a global community of software developers and contributors (White, 2015; Taylor, 2010).

Big Data in Healthcare

Big data in healthcare is utilized for reducing cost overhead, curing diseases, improving profits, predicting epidemics, and enhancing the quality of human life by preventing deaths. Scientific research labs, hospitals, and other medical institutions are leveraging big data analytics to reduce healthcare costs by changing the models of treatment delivery. The integration of big data in healthcare has led to significant improvements in patient care and operational efficiencies (Raghupathi & Raghupathi, 2014). Here begins the journey through big data in healthcare, highlighting the prominently used applications of big data in the healthcare industry.

Data Volume Growth

Data volume in the enterprise is projected to grow 50x year-over-year between now and 2020. The volume of business data worldwide, across all companies, doubles every 1.2 years. Back in 2010, Eric Schmidt famously stated that every two days, we create as much information as we did from the dawn of civilization up until 2003 (Schmidt, 2010). This exponential growth in data volume underscores the critical need for robust frameworks like Hadoop that can efficiently process and store vast amounts of information.

References

Raghupathi, W., & Raghupathi, V. (2014). Big data analytics in healthcare: promise and potential. Health Information Science and Systems, 2(1), 3.

Schmidt, E. (2010). Eric Schmidt at Techonomy: Mobile First. Techonomy. Retrieved from https://techonomy.com/2010/08/eric-schmidt-at-techonomy-mobile-first/

Taylor, K. (2010). Hadoop: The Definitive Guide. O'Reilly Media.

White, T. (2015). Hadoop: The Definitive Guide. O'Reilly Media.