tez-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hit...@apache.org
Subject tez git commit: TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN. (hitesh)
Date Mon, 27 Jul 2015 20:59:35 GMT
Repository: tez
Updated Branches:
  refs/heads/master b9d5c2056 -> c4dbeeab3


TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN. (hitesh)


Project: http://git-wip-us.apache.org/repos/asf/tez/repo
Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/c4dbeeab
Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/c4dbeeab
Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/c4dbeeab

Branch: refs/heads/master
Commit: c4dbeeab3fc20ccd75367013f2bbffc2fec7f3a3
Parents: b9d5c20
Author: Hitesh Shah <hitesh@apache.org>
Authored: Mon Jul 27 13:59:21 2015 -0700
Committer: Hitesh Shah <hitesh@apache.org>
Committed: Mon Jul 27 13:59:21 2015 -0700

----------------------------------------------------------------------
 CHANGES.txt                                |   1 +
 docs/src/site/markdown/tez-ui.md           | 136 ++++++++++++------------
 docs/src/site/markdown/tez_acls.md         |   5 +
 docs/src/site/markdown/tez_ui_user_data.md |   5 +-
 4 files changed, 80 insertions(+), 67 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index bd6aa25..dbf0833 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -11,6 +11,7 @@ INCOMPATIBLE CHANGES
   TEZ-2468. Change the minimum Java version to Java 7.
 
 ALL CHANGES:
+  TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN.
   TEZ-2602. Throwing EOFException when launching MR job.
   TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes
from the source.
   TEZ-2616. Fix build warning by undefined version of maven-findbugs-plugin.

http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/docs/src/site/markdown/tez-ui.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/tez-ui.md b/docs/src/site/markdown/tez-ui.md
index 52332ce..3855fbb 100644
--- a/docs/src/site/markdown/tez-ui.md
+++ b/docs/src/site/markdown/tez-ui.md
@@ -25,12 +25,74 @@ application data generated during the lifetime of a YARN application.
 Tez provides its own UI that interfaces with the _Application Timeline Server_ and displays
both a
 live and historical view of the Tez application inside a Tez UI web application.
 
-## Application Timeline Server Setup
+__For information on how application-specific data ends up being displayed in the Tez UI,
please refer to [this User Guide](tez_ui_user_data.html)__
 
-_Requires_: **Hadoop 2.6.0 or above**
+## Setting up Tez for the Tez UI
+---------
+
+_Requires_: **Tez 0.6.0 or above**
+
+For Tez to have a functional UI and for the History links to work correctly from the YARN
ResourceManager UI, the following settings need to be enabled in tez-site.xml:
+
+```
+tez-site.xml
+-------------
+...
+<property>
+  <description>Enable Tez to use the Timeline Server for History Logging</description>
+  <name>tez.history.logging.service.class</name>
+  <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
+</property>
 
-A minimal _yarn-site.xml_ configuration snippet is provided below. Replace localhost with
the actual
-hostname if running in a distributed setup.
+<property>
+  <description>URL for where the Tez UI is hosted</description>
+  <name>tez.tez-ui.history-url.base</name>
+  <value>http://<webserver-host:9999/tez-ui/</value>
+</property>
+...
+
+```
+
+## Deploying the Tez UI
+
+The tez-ui.war can be obtained in 3 ways:
+
+  - Obtain the tez-ui.war from the binary artifacts [for a given release](./releases/). Binary
artifacts are not available for all releases and in such cases, please use any of the other
options listed below.
+  - Obtain the tez-ui.war from the [Maven repo](https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-ui/)
for a given release.
+  - Build the tez-ui.war from the source itself. Refer to the README.txt in the tez-ui module
for more details.
+
+Once the tez-ui.war is available, you can extract the contents of the war and host them on
any webserver. The Tez UI consists only of html, js and css files
+and does not require any complex server-side hosting logic.
+
+### Configuring the Timeline Server URL and Resource Manager UI URL.
+By default, the Tez UI attempts to connect to Timeline Server using the same host as the
Tez UI.  For example, if the UI is hosted on localhost, the Timeline Server URL is assumed
to be http(s)://localhost:8188 and the Resource manager web url is assumed to be http(s)://localhost:8088.
+
+If the Timeline Server and/or Resource manager is hosted on a different host, the Tez UI
needs corresponding changes to be configured in scripts/configs.js ( within the extracted
tez-ui.war). Uncomment the following lines and set the hostname and port appropriately. "timelineBaseUrl"
maps to YARN Timeline Server and "RMWebUrl" maps to YARN ResourceManager.
+
+```
+    // timelineBaseUrl: 'http://localhost:8188',
+    // RMWebUrl: 'http://localhost:8088',
+
+```
+
+### Hosting the UI in Tomcat.
+
+1. Remove any old deployments in $TOMCAT_HOME/webapps
+2. Extract the war into $TOMCAT_HOME/webapps/tez-ui/
+3. Modify scripts/config.js as needed.
+4. Restart tomcat and the UI should be available under the tez-ui/ path.
+
+### Hosting the UI using any standalone webserver
+1. Untar the war file
+2. Modify scripts/config.js as needed.
+3. Copy the resulting directory to the document root of the web server.
+4. Restart the webserver
+
+## Additional Setup steps for the Application Timeline Server
+
+_Requires_: **Apache Hadoop 2.6.0 or above**
+
+The following configurations need to be setup in yarn-site.xml for correct functional behavior
of the Timeline Server with respect to the Tez UI. Replace localhost with the actual hostname
if running in a distributed setup.
 
 ```
 yarn-site.xml
@@ -57,75 +119,17 @@ yarn-site.xml
   <name>yarn.timeline-service.http-cross-origin.enabled</name>
   <value>true</value>
 </property>
-...
-```
-
-[Detailed Application Timeline Server Configuration](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServer.html)
-
-```
-# Starting the Application Timeline Server
-$ $HADOOP_PREFIX/sbin/yarn-daemon.sh start timelineserver
-
-```
 
-## Tez Setup
----------
-
-_Requires_: **Tez 0.6.0 or above**
-
-A minimal _tez-site.xml_ configuration snippet is provided below
-
-```
-tez-site.xml
--------------
-...
-<property>
-  <description>Log history using the Timeline Server</description>
-  <name>tez.history.logging.service.class</name>
-  <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
-</property>
 <property>
-  <description>Publish configuration information to Timeline server.</description>
-  <name>tez.runtime.convert.user-payload.to.history-text</name>
+  <description>Publish YARN information to Timeline Server</description>
+  <name> yarn.resourcemanager.system-metrics-publisher.enabled</name>
   <value>true</value>
 </property>
 ...
-
 ```
 
-[Detailed Tez - Timeline Server Configuration](tez_yarn_timeline.html)
-
-[Embedding Application Specific Data into Tez UI](tez_ui_user_data.html)
+__For more detailed information (setup, configuration, deployment), please refer to the [Apache
Hadoop Documentation on the Application Timeline Server](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServer.html)__
 
-## Tez UI Install
+__For general information on the compatibility matrix of the Tez UI with YARN TimelineServer,
please refer to the [Tez - Timeline Server Guide](tez_yarn_timeline.html)__
 
-### Building Tez UI from sources
-For instructions on building Tez UI from sources or setting up a development environment
see the
-README.txt in tez-ui subfolder. The build generates a war file of the distribution
-in the tez-ui/target. Currently the Tez-UI can be hosted using one of the following options.
-
-### hosting using the war file and tomcat.
-1. Remove any old deployments in $TOMCAT_HOME/webapps
-2. Copy the war to $TOMCAT_HOME/webapps
-3. Restart tomcat and the war will get deployed. The content of the war would be available
in
-$TOMCAT_HOME/webapps/tez-ui-x.x.x.
 
-### hosting using extracted war file and a standalone webserver
-1. untar the war file
-2. copy the resulting directory to the document root of the web server.
-3. reload/restart the webserver
-
-### configuring the Timeline Server URL and Resource manager web url.
-By default, the Tez UI attempts to connect to Timeline Server using the same host as the
Tez UI.
-For example, if the UI is hosted on localhost, the Timeline Server URL is assumed to be
-http(s)://localhost:8188 and the Resource manager web url is assumed to be http(s)://localhost:8088.
-
-If the Timeline Server and/or Resource manager is hosted on a different host, the Tez UI
needs
-corresponding changes to be configured in scripts/configs.js. Uncomment the following lines
and
-set the hostname and port appropriately.
-
-```
-    // timelineBaseUrl: 'http://localhost:8188',
-    // RMWebUrl: 'http://localhost:8088',
-
-```

http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/docs/src/site/markdown/tez_acls.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/tez_acls.md b/docs/src/site/markdown/tez_acls.md
index 2ac6830..7264c58 100644
--- a/docs/src/site/markdown/tez_acls.md
+++ b/docs/src/site/markdown/tez_acls.md
@@ -37,6 +37,11 @@ From the above, you can see that All users/groups that have access to do
operati
 
 Support for ACLs was introduced in Tez 0.5.0. Integration of these ACLs with YARN Timeline
is only available from Tez 0.6.0 onwards.
 
+## ACLs and the Tez UI
+
+For [versions of YARN Timeline Server](./tez_yarn_timeline.html) that support ACLs, the UI
will respect the ACLs for such secure clusters. For example, when a particular
+authenticated user is viewing the "All DAGs" page on the UI, the user will only be shown
DAGs that the user has access to view.
+
 ## How to setup the ACLs
 
 By default, ACLs are always enabled in Tez. To disable ACLs, set the following configuration
property:

http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/docs/src/site/markdown/tez_ui_user_data.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/tez_ui_user_data.md b/docs/src/site/markdown/tez_ui_user_data.md
index 71ffde3..7dd6abd 100644
--- a/docs/src/site/markdown/tez_ui_user_data.md
+++ b/docs/src/site/markdown/tez_ui_user_data.md
@@ -26,6 +26,7 @@ The Tez UI is built mainly off data stored in [YARN Timeline]. The Tez API,
curr
 To set DAG level information, the API to use is DAG::setDAGInfo.  ( Please refer to the Javadocs
for more detailed and up-to-date information )
 
 The DAG::setDAGInfo() API expects a to be String passed to it. This string is recommended
to be a json-encoded value with the following keys recognized keys by the UI:
+
   - "context": The application context in which this DAG is being used. For example, this
could be set to "Hive" or "Pig" if this is being run as part of a Hive or Pig script.
   - "description": General description on what this DAG is going to do. In the case of Hive,
this could be the SQL query text.
 
@@ -34,6 +35,7 @@ The DAG::setDAGInfo() API expects a to be String passed to it. This string
is re
 Each Input/Output/Processor specified in the DAG Plan is specified via a TezEntityDescriptor.
Applications specify a user payload that is used to initialize/configure the instance as needed.
From a Tez UI point of view, users are usually keen to understand what "work" the particular
Input/Output/Processor is doing in addition to any additional configuration information on
how the object was initialized/configured. Keeping that in mind, each TezEntityDescriptor
supports an api for application developers to specify this information when creating the DAG
plan. The API to use for this is setHistoryText(). 
 
 The setHistoryText() API expects a String to be passed to it. This string is recommended
to be a json-encoded value with the following keys recognized keys by the UI:
+
   - "desc" : A simple string describing for the object in question. For example, for a particular
Hive Processor, this could be a description of what that particular processor is doing.
   - "config" : A map of key-value pairs representing the configuration/payload used to initialize
the object in question.
 
@@ -44,9 +46,10 @@ By default, the Inputs/Outputs/Processors that are part of the tez-runtime-libra
 > &nbsp;&nbsp;&nbsp;&lt;value&gt;true&lt;/value&gt;<br/>
 > &lt;/property&gt;<br/>
 
+Please note that this is a data-heavy operation and adds a lot of load on the TimelineServer
(due to the additional amount of configuration data generated) and therefore should be disabled
for most jobs unless there is a real debugging need.
+
 ## Use of DAG Info and History Text in the Tez UI
 
 If the data setup in the DAG Info and History Text conforms to the format expected by the
UI, it will be displayed in the Tez UI in an easy to consume manner. In cases where this is
not possible, the UI may fall back to either not displaying the data at all or displaying
the string as is in a safe manner. 
 
 [YARN Timeline]:./tez_yarn_timeline.html
-


Mime
View raw message