07-04-2017, 10:06 AM
History of Hadoop
As the World Wide Web grew in the late 1900s and early 2000s, search engines and indexes were created to help locate relevant information in the midst of text-based content. In the early years, the search results were returned by humans. But as the web grew from dozens to millions of pages, automation was needed. Web crawlers, many as university-run research projects, were set up and search companies (Yahoo, AltaVista, etc.) started.
The Hadoop open source framework is based on Google's MapReduce software and can process large datasets at a granular level. It offers low-cost, high-speed analysis that some analysts say can not be achieved any other way. The Hadoop Distributed File System (HDFS), which enables parallel processing by expanding data across different nodes in a single cluster and providing fault tolerance, is essential for the effectiveness of Hadoop.
However, HDFS is the source of one of the major problems that users see with Hadoop technology: increased capacity requirements due to Hadoop storing three copies of each piece of data in the event of a DataNode failure or being taken out of line. Such a failover configuration is necessary because each NameNode that controls the data copying and distribution process is a single point of failure. Other complaints point to the complicated technology derived from the Hadoop Java framework.
Despite obstacles with Hadoop technology, analysts and users say the benefits are worth it. To help you determine for yourself, this guide will guide you through the basics of what Hadoop technology can achieve, describe the main concerns about the technology, and describe how it works with storage and the cloud.
