tez-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ss...@apache.org
Subject git commit: TEZ-1471. Additional supplement for TEZ local mode document. Contributed by Chen He.
Date Mon, 25 Aug 2014 21:51:46 GMT
Repository: tez
Updated Branches:
  refs/heads/branch-0.5 d520f732c -> df152969d


TEZ-1471. Additional supplement for TEZ local mode document. Contributed
by Chen He.


Project: http://git-wip-us.apache.org/repos/asf/tez/repo
Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/df152969
Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/df152969
Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/df152969

Branch: refs/heads/branch-0.5
Commit: df152969d6a3362ae34fd8b8eba976c1f1300e13
Parents: d520f73
Author: Siddharth Seth <sseth@apache.org>
Authored: Mon Aug 25 14:49:29 2014 -0700
Committer: Siddharth Seth <sseth@apache.org>
Committed: Mon Aug 25 14:51:33 2014 -0700

----------------------------------------------------------------------
 docs/src/site/apt/localmode.apt | 29 ++++++++++++++++++-----------
 docs/src/site/site.xml          |  2 +-
 2 files changed, 19 insertions(+), 12 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/tez/blob/df152969/docs/src/site/apt/localmode.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/localmode.apt b/docs/src/site/apt/localmode.apt
index b38bf77..25fe13a 100644
--- a/docs/src/site/apt/localmode.apt
+++ b/docs/src/site/apt/localmode.apt
@@ -17,7 +17,7 @@
   Tez Local Mode
 -----
 
-  Tez LocalMode is a development tool to test Tez jobs without needing to bring up a Hadoop
cluster. LocalMode runs the Tez compoenents - AppMaster, TaskRunner that are used when executing
a job on a cluster. From a developer tool perspective, it offers several advantages.
+  Tez Local Mode is a development tool to test Tez jobs without needing to bring up a Hadoop
cluster. Local Mode runs the Tez compoenents - AppMaster, TaskRunner that are used when executing
a job on a cluster. From a developer tool perspective, it offers several advantages.
 
     * Fast prototyping - Hadoop setup, launch cost etc not involved.
 
@@ -28,23 +28,23 @@
 
   While majority of the components are re-used in Local Mode, there are some bits which are
not
 
-    * Scheduling and Container Re-Use differs
+    * Scheduling and Container Re-Use differs.
 
-    * Handling of YARN Local Resources. LocalMode expects necessary jars to be loaded with
the Client when executing.
+    * Handling of YARN Local Resources. Local Mode expects necessary jars to be loaded with
the Client when executing.
 
     * Contains some performance improvements - like skipping RPC invocations since everything
runs within the same JVM.
 
 
-  Running a DAG in LocalMode
+  Running a DAG in Local Mode
 
-    * "tez.local.mode" should be set to true in the confgiuration instance used to create
the TezClient
+    * "tez.local.mode" should be set to true in the confgiuration instance used to create
the TezClient.
 
     * The FileSystem must be configured to the local file system ("fs.defaultFS" must be
set to "file:///"). This is required to be setup in all Configuration instances used to create
a DAG. Typically, when using Tez for testing and prototyping without a Hadoop cluster, this
is not a problem. It becomes a problem when Hadoop Configuration files are in the classpath,
with a different default filesystem configured.
 
     * Setup the fetchers to make use of local reads instead of fetching from remote nodes.
("tez.runtime.optimize.local.fetch must be set to true)
 
-    * Beyond this, no other changes are required, to make use of LocalMode instead of running
a job on a cluster.
-
+    * Beyond this, no other changes are required, to make use of Local Mode instead of running
a job on a cluster.
+  
     * If using this in code, the following changes should be made to configuration, after
which this configuration instance becomes the base for all other Configuration instances.
 
 +------------------
@@ -62,7 +62,7 @@ conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH,
true);
   <value>file:///</value>
 </property>
 <property>
-  <name>tez.local.mode<name>
+  <name>tez.local.mode</name>
   <value>true</value>
 </property>
 <property>
@@ -70,13 +70,20 @@ conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH,
true);
   <value>true</value>
 </property>
 +------------------
+  
+  Things to watch out for
+
+    * In current Local Mode, large amount of input data may lead to JVM out of memory since
all TEZ components are running in single JVM. The input data size should be kept small.
+
+    * TezConfiguration.TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS(tez.am.inline.task.execution.max-tasks)
should not be changed (defaults to 1).
 
+    * "tez.history.logging.service.class" should be the default value: "org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService".
It means ATS is disabbled in current Local Mode.
 
-  Potential pitfalls when moving from LocalMode to a real cluster
+  Potential pitfalls when moving from Local Mode to a real cluster
 
     * Resource requirements (CPU, Memory, etc) which would otherwise have been specified
for a YARN Cluster will now start taking affect, and should be considered.
 
-    * The Java Options and Environment variables which may have been setup for the DAG do
not take affect in LocalMode, and could be a source of migration problems.
+    * The Java Options and Environment variables which may have been setup for the DAG do
not take affect in Local Mode, and could be a source of migration problems.
 
-    * The ObjectRegistry will work within a single task, when running in LocalMode. The behaviour
would be different on a real cluster, where it would work across tasks which share the same
container.
+    * The ObjectRegistry will work within a single task, when running in Local Mode. The
behaviour would be different on a real cluster, where it would work across tasks which share
the same container.
 

http://git-wip-us.apache.org/repos/asf/tez/blob/df152969/docs/src/site/site.xml
----------------------------------------------------------------------
diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml
index 98f7207..1a915db 100644
--- a/docs/src/site/site.xml
+++ b/docs/src/site/site.xml
@@ -99,7 +99,7 @@
 
     <menu name="Documentation">
       <item name="Install Guide" href="install.html"/>
-      <item name="LocalMode" href="localmode.html"/>
+      <item name="Local Mode" href="localmode.html"/>
     </menu>
 
     <menu name="Community">


Mime
View raw message