tez-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeag...@apache.org
Subject [08/50] [abbrv] tez git commit: TEZ-3360. Tez Custom Shuffle Handler Documentation (jeagles)
Date Wed, 24 May 2017 21:07:37 GMT
TEZ-3360. Tez Custom Shuffle Handler Documentation (jeagles)


Project: http://git-wip-us.apache.org/repos/asf/tez/repo
Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/613f1e07
Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/613f1e07
Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/613f1e07

Branch: refs/heads/master
Commit: 613f1e0743999c3bf558761c60165bcd2dcd05c7
Parents: 78eee17
Author: Jonathan Eagles <jeagles@yahoo-inc.com>
Authored: Mon Aug 15 10:05:26 2016 -0500
Committer: Jonathan Eagles <jeagles@yahoo-inc.com>
Committed: Mon Aug 15 10:05:26 2016 -0500

----------------------------------------------------------------------
 TEZ-3334-CHANGES.txt                      |  1 +
 docs/src/site/markdown/shuffle-handler.md | 72 ++++++++++++++++++++++++++
 docs/src/site/markdown/user_guides.md     |  1 +
 docs/src/site/site.xml                    |  1 +
 4 files changed, 75 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/TEZ-3334-CHANGES.txt
----------------------------------------------------------------------
diff --git a/TEZ-3334-CHANGES.txt b/TEZ-3334-CHANGES.txt
index 493614f..f93ec6e 100644
--- a/TEZ-3334-CHANGES.txt
+++ b/TEZ-3334-CHANGES.txt
@@ -4,6 +4,7 @@ Apache Tez Change Log
 INCOMPATIBLE CHANGES:
 
 ALL CHANGES:
+  TEZ-3360. Tez Custom Shuffle Handler Documentation
   TEZ-3411. TestShuffleHandler#testSendMapCount should not used hard coded ShuffleHandler
port
   TEZ-3412. Modify ShuffleHandler to use Constants.DAG_PREFIX and fix AttemptPathIdentifier#toString()
   TEZ-3410. ShuffleHandler should use Path.SEPARATOR instead "/"

http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/markdown/shuffle-handler.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/shuffle-handler.md b/docs/src/site/markdown/shuffle-handler.md
new file mode 100644
index 0000000..5dc6289
--- /dev/null
+++ b/docs/src/site/markdown/shuffle-handler.md
@@ -0,0 +1,72 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<head><title>Shuffle Handler</title></head>
+
+## Tez Shuffle Handler Overview
+
+A Tez specific shuffle handler allows Tez DAGs to shuffle data in a way that takes advantage
of the new features in Tez.
+In particular, the Tez shuffle handler allows DAGs to shuffle data more efficiently for Tez’s
new data movements types
+and runtime optimizations, such as auto-reduce parallelism. Long running Tez sessions will
be able to clean up
+intermediate data for completed queries and Tez applications can decide to clean up completed
intermediate data for
+running applications.
+
+## Setup for the Tez Shuffle Handler
+---------
+
+_Requires_: **Apache Tez 0.9.0 or above**
+
+Configuration in the client specify the Tez shuffle handler
+
+```
+tez-site.xml
+-------------
+...
+<property>
+  <name>tez.am.shuffle.auxiliary-service.id</name>
+  <value>tez_shuffle</value>
+</property>
+...
+
+```
+
+## Deploying the Tez Shuffle Handler
+
+The Tez Shuffle Handler jar artifact org.apache.org:tez-aux-services needs to be placed into
the Node Manager classpath and restarted
+
+## Setup for Node Manager
+
+_Requires_: **Apache Hadoop 2.6.0 or above**
+
+The following configuration needs to be setup in the Node Manager yarn-site.xml to enable
the Tez Shuffle Handler
+
+```
+yarn-site.xml
+-------------
+...
+<property>
+  <name>yarn.nodemanager.aux-services</name>
+  <value>tez_shuffle</value>
+</property>
+
+<property>
+  <name>yarn.nodemanager.aux-services.tez_shuffle.class</name>
+  <value>org.apache.tez.auxservices.ShuffleHandler</value>
+</property>
+...
+
+```

http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/markdown/user_guides.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/user_guides.md b/docs/src/site/markdown/user_guides.md
index 890974e..453df16 100644
--- a/docs/src/site/markdown/user_guides.md
+++ b/docs/src/site/markdown/user_guides.md
@@ -23,4 +23,5 @@
    - [Access Control in Tez](./tez_acls.html)
    - [Embedding Application Specific Data into Tez UI](./tez_ui_user_data.html)
    - [Tez UI - Overview and installation](./tez-ui.html)
+   - [Tez Shuffle Handler - Overview and installation](./shuffle-handler.html)
 

http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/site.xml
----------------------------------------------------------------------
diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml
index 3317bac..5674951 100644
--- a/docs/src/site/site.xml
+++ b/docs/src/site/site.xml
@@ -107,6 +107,7 @@
       <item name="Install Guide" href="install.html"/>
       <item name="Local Mode" href="localmode.html"/>
       <item name="Tez UI" href="tez-ui.html"/>
+      <item name="Tez Shuffle Handler" href="shuffle-handler.html"/>
       <item name="User Guides" href="user_guides.html"/>
     </menu>
 


Mime
View raw message