hadoop-mapreduce-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From acmur...@apache.org
Subject svn commit: r1195764 - in /hadoop/common/trunk/hadoop-mapreduce-project: ./ hadoop-yarn/hadoop-yarn-site/src/site/ hadoop-yarn/hadoop-yarn-site/src/site/apt/ hadoop-yarn/hadoop-yarn-site/src/site/resources/
Date Tue, 01 Nov 2011 02:00:59 GMT
Author: acmurthy
Date: Tue Nov  1 02:00:58 2011
New Revision: 1195764

URL: http://svn.apache.org/viewvc?rev=1195764&view=rev
Log:
MAPREDUCE-3322. Added a better index.html and an brief overview of YARN architecture.  

Added:
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/yarn_architecture.gif
  (with props)
Modified:
    hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
    hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml

Modified: hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt?rev=1195764&r1=1195763&r2=1195764&view=diff
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt (original)
+++ hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Tue Nov  1 02:00:58 2011
@@ -455,6 +455,9 @@ Release 0.23.0 - Unreleased
     MAPREDUCE-3275. Added documentation for AM WebApp Proxy. (Robert Evans via
     acmurthy)
 
+    MAPREDUCE-3322. Added a better index.html and an brief overview of YARN
+    architecture. (acmurthy) 
+
   OPTIMIZATIONS
 
     MAPREDUCE-2026. Make JobTracker.getJobCounters() and

Modified: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm?rev=1195764&r1=1195763&r2=1195764&view=diff
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
(original)
+++ hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
Tue Nov  1 02:00:58 2011
@@ -22,7 +22,6 @@ Hadoop MapReduce Next Generation - Capac
 
 %{toc|section=1|fromDepth=0}
 
-
 * {Purpose} 
 
   This document describes the <<<CapacityScheduler>>>, a pluggable scheduler


Modified: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm?rev=1195764&r1=1195763&r2=1195764&view=diff
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm
(original)
+++ hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm
Tue Nov  1 02:00:58 2011
@@ -20,6 +20,8 @@ Hadoop MapReduce Next Generation - Setti
 
   \[ {{{./index.html}Go Back}} \]
 
+%{toc|section=1|fromDepth=0}
+
 * Mapreduce Tarball
 
   You should be able to obtain the MapReduce tarball from the release.

Modified: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm?rev=1195764&r1=1195763&r2=1195764&view=diff
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
(original)
+++ hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
Tue Nov  1 02:00:58 2011
@@ -21,7 +21,7 @@ Hadoop MapReduce Next Generation - Writi
 
   \[ {{{./index.html}Go Back}} \]
 
-%{toc|section=1|fromDepth=1}
+%{toc|section=1|fromDepth=0}
 
 * Purpose 
 

Added: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm?rev=1195764&view=auto
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm
(added)
+++ hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm
Tue Nov  1 02:00:58 2011
@@ -0,0 +1,77 @@
+~~ Licensed under the Apache License, Version 2.0 (the "License");
+~~ you may not use this file except in compliance with the License.
+~~ You may obtain a copy of the License at
+~~
+~~   http://www.apache.org/licenses/LICENSE-2.0
+~~
+~~ Unless required by applicable law or agreed to in writing, software
+~~ distributed under the License is distributed on an "AS IS" BASIS,
+~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+~~ See the License for the specific language governing permissions and
+~~ limitations under the License. See accompanying LICENSE file.
+
+  ---
+  YARN
+  ---
+  ---
+  ${maven.build.timestamp}
+
+Apache Hadoop NextGen MapReduce (YARN)
+
+  MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, 
+  what we call, MapReduce 2.0 (MRv2) or YARN.
+
+  The fundamental idea of MRv2 is to split up the two major functionalities of 
+  the JobTracker, resource management and job scheduling/monitoring, into 
+  separate daemons. The idea is to have a global ResourceManager (<RM>) and 
+  per-application ApplicationMaster (<AM>).  An application is either a single 
+  job in the classical sense of Map-Reduce jobs or a DAG of jobs. 
+
+  The ResourceManager and per-node slave, the NodeManager (<NM>), form the 
+  data-computation framework. The ResourceManager is the ultimate authority that 
+  arbitrates resources among all the applications in the system. 
+
+  The per-application ApplicationMaster is, in effect, a framework specific 
+  library and is tasked with negotiating resources from the ResourceManager and 
+  working with the NodeManager(s) to execute and monitor the tasks.
+
+[./yarn_architecture.gif] MapReduce NextGen Architecture
+
+  The ResourceManager has two main components: Scheduler and 
+  ApplicationsManager.
+
+  The Scheduler is responsible for allocating resources to the various running 
+  applications subject to familiar constraints of capacities, queues etc. The 
+  Scheduler is pure scheduler in the sense that it performs no monitoring or 
+  tracking of status for the application. Also, it offers no guarantees about 
+  restarting failed tasks either due to application failure or hardware 
+  failures. The Scheduler performs its scheduling function based the resource 
+  requirements of the applications; it does so based on the abstract notion of 
+  a resource <Container> which incorporates elements such as memory, cpu, disk, 
+  network etc. In the first version, only <<<memory>>> is supported. 
+
+  The Scheduler has a pluggable policy plug-in, which is responsible for 
+  partitioning the cluster resources among the various queues, applications etc. 
+  The current Map-Reduce schedulers such as the CapacityScheduler and the 
+  FairScheduler would be some examples of the plug-in. 
+
+  The CapacityScheduler supports <<<hierarchical queues>>> to allow for
more 
+  predictable sharing of cluster resources
+
+  The ApplicationsManager is responsible for accepting job-submissions, 
+  negotiating the first container for executing the application specific 
+  ApplicationMaster and provides the service for restarting the
+  ApplicationMaster container on failure.
+
+  The NodeManager is the per-machine framework agent who is responsible for 
+  containers, monitoring their resource usage (cpu, memory, disk, network) and 
+  reporting the same to the ResourceManager/Scheduler.
+
+  The per-application ApplicationMaster has the responsibility of negotiating 
+  appropriate resource containers from the Scheduler, tracking their status and 
+  monitoring for progress.
+
+  MRV2 maintains <<API compatibility>> with previous stable release 
+  (hadoop-0.20.205).  This means that all Map-Reduce jobs should still run 
+  unchanged on top of MRv2 with just a recompile.
+

Modified: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm?rev=1195764&r1=1195763&r2=1195764&view=diff
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
(original)
+++ hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
Tue Nov  1 02:00:58 2011
@@ -11,14 +11,32 @@
 ~~ limitations under the License. See accompanying LICENSE file.
 
   ---
-  Hadoop MapReduce Next Generation  ${project.version}
+  Apache Hadoop 0.23
   ---
   ---
   ${maven.build.timestamp}
 
-Hadoop MapReduce Next Generation
+Apache Hadoop 0.23
 
-* Architecture
+  Apache Hadoop 0.23 consists of significant improvements over the previous 
+  stable release (hadoop-0.20.205).
+
+  Here is a short overview of the improvments to both HDFS and MapReduce.
+
+* {HDFS Federation}
+
+  In order to scale the name service horizontally, <federation> uses multiple 
+  independent Namenodes/Namespaces. The Namenodes are federated, that is, the 
+  Namenodes are independent and don't require coordination with each other. 
+  The datanodes are used as common storage for blocks by all the Namenodes. 
+  Each datanode registers with all the Namenodes in the cluster. Datanodes 
+  send periodic heartbeats and block reports and handles commands from the 
+  Namenodes.
+
+  More details are available in the {{{./Federation.html}HDFS Federation}}
+  document.
+  
+* {MapReduce NextGen aka YARN aka MRv2}
 
   The new architecture introduced in hadoop-0.23, divides the two major 
   functions of the JobTracker: resource management and job life-cycle management 
@@ -32,22 +50,32 @@ Hadoop MapReduce Next Generation
   or a DAG of such jobs. 
   
   The ResourceManager and per-machine NodeManager daemon, which manages the 
-  user processes on that machine, form the computation fabric. The 
-  per-application ApplicationMaster is, in effect, a framework specific library 
-  and is tasked with negotiating resources from the ResourceManager and working 
-  with the NodeManager(s) to execute and monitor the tasks.
+  user processes on that machine, form the computation fabric. 
+  
+  The per-application ApplicationMaster is, in effect, a framework specific 
+  library and is tasked with negotiating resources from the ResourceManager and 
+  working with the NodeManager(s) to execute and monitor the tasks.
 
-* User Documentation
+  More details are available in the {{{./YARN.html}YARN}}
+  document.
+
+* Release Documentation
 
   * {{{./SingleCluster.html}Setting up a Single Node Cluster}}
 
   * {{{./ClusterSetup.html}Setting up a full-fledged Hadoop Cluster}}
   
-  * {{{./apidocs/index.html}JavaDocs}}
+  * {{{./CapacityScheduler.html}Capacity Scheduler}}
+
+  * {{{./Federation.html}HDFS Federation feature description, configuration and 
+    management}}
+
+  * {{{./YARN.html}NextGen MapReduce}}
+
+* User Documentation
 
   * {{{./WritingYarnApplications.html}Writing Yarn Applications}}
 
-  * {{{./CapacityScheduler.html}Capacity Scheduler}}
+  * {{{./apidocs/index.html}JavaDocs}}
 
-  * {{{./Federation.html}HDFS Federation feature description, configuration and management}}
 

Added: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/yarn_architecture.gif
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/yarn_architecture.gif?rev=1195764&view=auto
==============================================================================
Binary file - no diff available.

Propchange: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/yarn_architecture.gif
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Modified: hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml
URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml?rev=1195764&r1=1195763&r2=1195764&view=diff
==============================================================================
--- hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml
(original)
+++ hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml
Tue Nov  1 02:00:58 2011
@@ -11,7 +11,7 @@
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
 -->
-<project name="Hadoop MapReduce Next Gen">
+<project name="Apache Hadoop 0.23">
 
   <skin>
     <groupId>org.apache.maven.skins</groupId>



Mime
View raw message