Apache Hadoop is an open-source software framework for storage and large scale processing of data-sets on clusters of commodity hardware. Hadoop, an Apache top-level project is built and used by a global community of contributors and users. Rather than relying on hardware to deliver high-availability, the library is designed to detect and handle failures at the application layer itself. It delivers a highly-available service on top of a cluster of computers, each of which may be prone to failures.
A small Hadoop cluster has a single master and multiple worker nodes. The master node consists of a JobTracker, TaskTracker, NameNode and DataNode. A slave or worker node acts as both a DataNode and TaskTracker, though it is possible to have data-only worker nodes and compute-only worker nodes. These are normally used only in nonstandard applications. Hadoop requires Java Runtime Environment (JRE) 1.6 or higher. The standard start-up and shutdown scripts require ssh to be set up between nodes in the cluster.
The Apache Hadoop framework is composed of the modules Hadoop Common which contains libraries and utilities for other Hadoop modules, Hadoop MapReduce is a programming model for large scale data processing, Hadoop Distributed File System (HDFS) is a distributed file-system which stores data that provides very high aggregate bandwidth across the cluster and Hadoop YARN, a resource-management platform that manages computer resources in clusters and uses them for scheduling of user applications.
The Hadoop distributed file system is a distributed, scalable, and portable file-system written in Java for the Hadoop framework. Each node in a Hadoop instance has a single namenode; a cluster of datanodes form the HDFS cluster as shown in Figu...
... middle of paper ...
....
[12]
More information on the Apache
Pig
can be found at http://hortonworks.com/hadoop/pig/ on the Web.
[13] More information on the XML and Security: introduction to XACML-
Access
Control
Policies
in
XML
can be found https://community.emc.com/docs/DOC-7314 at and http://dimacs.rutgers.edu/Workshops/Commerce/slides/crampton.pdf on the
Web.
[14]More information on the XACML
Policy
language can be found at http://wso2.com/library/articles/2011/10/understanding-xacml-policy-language-xacml- extended-assertion-markup-langue-part-1/ on the Web.
[15] More information on the Authorization and Authentication In Hadoop can be found at http://blog.cloudera.com/blog/2012/03/authorization-and-authentication-in-hadoop/ on the Web.
[16]
More information on the A
HDFS
Architecture can be found http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#NameNode+and+DataNodes on the Web.
Kerberos provides a secure authentication scheme. Authentication is needed to restrict the intruders and malicious users. The major security issues discussed are privacy of the data, integrity of data and authentication mechanism which is not there in Hadoop. Hadoop supports Kerberos for authentication and many security features can be configured with the Hadoop to restrict the accessibility of the data. The data can be associated with the user names or group names in which data can be accessed. Kerberos is a conventional authentication system, improved authentication systems can be used which are more secure and efficient than
Amdocs Data Hub is a scalable, Hadoop-based data management platform which enables service providers to seamlessly extract, integrate and visualize their diverse and rapidly growing data sources. This leads to data-driven decision making and analytics (Amdocs Data Hub).
In a larger cluster,an extra NameNode called secondary NameNode is configured to avoid single point of failure .HDFS is managed with dedicated NameNode to host the file system index, and secondary NameNode can generate snapshots of namenode's memory structures.In this way ,it preventing file-system errors or corruption and reducing data loss. Similarly, job scheduling can manage through a standalone JobTracker server. In clusters,the Hadoop MapReduce engine deployed against an alternate file system, the NameNode, DataNode,secondary NameNode.
Gomoloski, Barb. Going global: Some lessons from eToys and Yahoo that might help You. InfoWorld. February 2001.
Hardware, software, support and maintenance costs grow each year with multiple systems in each local region running different types of software and hardware. The application and hardware support teams are larger than could be possible with one integrated solution.
Apache Hadoop is one of the solutions; it is an open-source software framework for storage and large scale processing of data-sets on clusters of commodity hardware [3]. Also, Apache Hadoop is a scalable fault-tolerant distributed system for data storage and processing. The core of Hadoop has ...
Google File System (GFS) was developed at Google to meet the high data processing needs. Hadoop’s Distributed File System (HDFS) was originally developed by Yahoo.Inc but it is maintained as an open source by Apache Software Foundation. HDFS was built based on Google’s GFS and Map Reduce. As the internet data was rapidly increasing there was a need to store the large data coming so Google developed a distributed file system called GFS and HDFS was developed to meet the different client needs. These are built on commodity hardware so the systems often fail. To make the systems reliable the data is replicated among multiple nodes. By default minimum number of replicas is 3. Millions of files and large files are common with these types of file systems. Data is more often read than writing. Large streaming needs and small random needs are supported.
It was integrated with one platform( three products with various interfaces) and it is beneficial for the admin
Hadoop is a free, java based programming framework that usually supports processing of large data sets that are in diverse computing environment. Hadoop is a cluster computation framework. Apache Hadoop is a programming framework that provides support to data intensive distributed applications with a free license. The program was inspired by Google file and Google’s MapReduce system (Eadline, 2013). According to Eadline (2013), the Hadoop technology was designed to solve various problems such as to provide a fast and reliable analysis for both complex and clustered data. Consequently, different organizational enterprises deployed Hadoop with the existing IT systems, thereby, allowing them to combine old and new data in a strong framework. The major industrial players who used Hadoop technology include IBM, Yahoo, and Google (Lavalle, Lesser, Shockley, Hopkins & Kruschwitz, 2011).
Section 1.4 is basically the author telling us about two important network architectures, the OSI reference model and the TCP/IP reference model. The OSI (Open Systems Interconnection) deals with connecting opens systems, which are systems that talk with other systems. This model has seven layers: the physical layer, the data link layer, the network layer, the transport layer, the session layer, the presentation layer, and the application layer. From this information, we discover that the OSI model is used only for its model (the 7 layers) and not as a network architecture since it doesn’t specify the exact services and protocols to be used in each layer.
It simplifies the storage and processing of large amounts of data, eases the deployment and operation of large-scale global products and services, and automates much of the administration of large-scale clusters of computers.
The protocol stack is built up with a few layers and planes, namely application layer, transport layer, network layer, data link layer, physical layer. And the planes are task management plane, mobility management plane and power management plane. The block diagram of a protocol stack is shown below
The Internet is a global network connecting millions of personal, institutional and company computers. The number of computers used by the internet is growing rapidly. The United States is connected with over 100 countries worldwide and linked together to exchange of data, news and opinions. The Internet is decentralized design. This means that there isn't just one computer that stores all of the information from the Internet. There are many independent host servers located throughout the US and the world that store the information made available to the global Internet community.
...tecture for scalability and availability as the public cloud but is restricted to a single organization.