tez-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bi...@apache.org
Subject [2/2] git commit: TEZ-1440. Post-release tasks (bikas)
Date Fri, 05 Sep 2014 03:41:45 GMT
TEZ-1440. Post-release tasks (bikas)


Project: http://git-wip-us.apache.org/repos/asf/tez/repo
Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/eed2d9ab
Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/eed2d9ab
Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/eed2d9ab

Branch: refs/heads/master
Commit: eed2d9ab0c90d82b2c3cf1302b33a0c1acd9fa03
Parents: b4cd972
Author: Bikas Saha <bikas@apache.org>
Authored: Thu Sep 4 20:41:38 2014 -0700
Committer: Bikas Saha <bikas@apache.org>
Committed: Thu Sep 4 20:41:38 2014 -0700

----------------------------------------------------------------------
 docs/src/site/markdown/index_0_5_0.md       |  29 ++++++
 docs/src/site/markdown/install_pre_0_5_0.md | 109 +++++++++++++++++++++++
 2 files changed, 138 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/tez/blob/eed2d9ab/docs/src/site/markdown/index_0_5_0.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/index_0_5_0.md b/docs/src/site/markdown/index_0_5_0.md
new file mode 100644
index 0000000..5295ca3
--- /dev/null
+++ b/docs/src/site/markdown/index_0_5_0.md
@@ -0,0 +1,29 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<head><title>Apache Tez 0.5.0</title></head>
+
+Apache Tez 0.5.0
+----------------
+
+- [Release Artifacts](http://www.apache.org/dyn/closer.cgi/tez)
+- [Release Notes](releases/0.5.0/release-notes.txt)
+- Documentation
+    - [API Javadocs](releases/0.5.0/tez-api-javadocs/index.html) : Documentation for the
Tez APIs
+    - [Runtime Library Javadocs](releases/0.5.0/tez-runtime-library-javadocs/index.html)
: Documentation for built-in implementations of useful Inputs, Outputs, Processors etc. written
based on the Tez APIs 
+    - [Tez Mapreduce Javadocs](releases/0.5.0/tez-mapreduce-javadocs/index.html) : Documentation
for built-in implementations of Mapreduce compatible Inputs, Outputs, Processors etc. written
based on the Tez APIs 
+

http://git-wip-us.apache.org/repos/asf/tez/blob/eed2d9ab/docs/src/site/markdown/install_pre_0_5_0.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/install_pre_0_5_0.md b/docs/src/site/markdown/install_pre_0_5_0.md
new file mode 100644
index 0000000..494ff54
--- /dev/null
+++ b/docs/src/site/markdown/install_pre_0_5_0.md
@@ -0,0 +1,109 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<head><title>Install and Deployment Instructions</title></head>
+
+[Install instructions for Tez (post 0.5.0)](./install.html)
+-----------------------------------------------------------------------------------
+
+Install/Deploy Instructions for Tez release pre-0.5.0 E.g. [(Tez-0.4.1)](http://archive.apache.org/dist/incubator/tez/tez-0.4.1-incubating/)
+--------------------------------------------------------------------------------------------------------------------------------------------------
+
+1.  Deploy Apache Hadoop using either the 2.2.0 release or a compatible
+    2.x version.
+    -   One thing to note though when compiling Tez is that you will
+        need to change the value of the hadoop.version property in the
+        toplevel pom.xml to match the version of the hadoop branch being
+        used.
+2.  Build tez using `mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true`
+    -   This assumes that you have already installed JDK6 or later,
+        Maven 3 or later and Protocol Buffers (protoc compiler) 2.5 or
+        later
+    -   If you prefer to run the unit tests, remove skipTests from the
+        command above.
+    -   If you would like to create a tarball of the release, use `mvn
+        clean package -Dtar -DskipTests=true -Dmaven.javadoc.skip=true`
+    -   If you use Eclipse IDE, you can import the projects using
+        "Import/Maven/Existing Maven Projects". Eclipse does not
+        automatically generate Java sources or include the generated
+        sources into the projects. Please build using maven as described
+        above and then use Project Properties to include
+        "target/generated-sources/java" as a source directory into the
+        "Java Build Path" for these projects: tez-api, tez-mapreduce,
+        tez-runtime-internals and tez-runtime-library. This needs to be done
+        just once after importing the project.
+3.  Copy the tez jars and their dependencies into HDFS.
+    -   The tez jars and dependencies will be found in
+        tez-dist/target/tez-0.4.1-incubating/tez-0.4.1-incubating if you run
+        the intial command mentioned in step 2.
+    -   Assuming that the tez jars are put in /apps/ on HDFS, the
+        command would be `hadoop dfs -put
+        tez-dist/target/tez-0.4.1-incubating/tez-0.4.1-incubating /apps/`
+    -   Please do not upload the tarball to HDFS, upload only the jars.
+4.  Configure tez-site.xml to set tez.lib.uris to point to the paths in
+    HDFS containing the jars. Please note that the paths are not
+    searched recursively so for *basedir* and *basedir*/lib/, you will
+    need to configure the 2 paths as a comma-separated list. * Assuming
+    you followed step 3, the value would be:
+    "${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name}/apps/tez-0.4.1-incubating/lib/"
+5.  Modify mapred-site.xml to change _mapreduce.framework.name_ property
+    from its default value of *yarn* to *yarn-tez*
+6.  Set HADOOP_CLASSPATH to have the following paths in it:
+    -   TEZ_CONF_DIR - location of tez-site.xml
+    -   TEZ_JARS and TEZ_JARS/libs - location of the tez jars and
+        dependencies.
+    -   The command to set up the classpath should be something like:
+        `export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*`
+        Please note the "*" which is an important requirement when
+        setting up classpaths for directories containing jar files.
+7.  Submit a MR job as you normally would using something like:
+
+    ```
+    $HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar
sleep -mt 1 -rt 1 -m 1 -r 1
+    ```
+
+    This will use the TEZ DAG ApplicationMaster to run the MR job. This
+    can be verified by looking at the AM’s logs from the YARN
+    ResourceManager UI.
+8.  There is a basic example of using an MRR job in the
+    tez-mapreduce-examples.jar. Refer to OrderedWordCount.java in the
+    source code. To run this example:
+
+    ``` 
+    $HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount <input>
<output>
+    ```
+
+    This will use the TEZ DAG ApplicationMaster to run the ordered word
+    count job. This job is similar to the word count example except that
+    it also orders all words based on the frequency of occurrence.
+
+    There are multiple variations to run orderedwordcount. You can use
+    it to run multiple DAGs serially on different inputs/outputs. These
+    DAGs could be run separately as different applications or serially
+    within a single TEZ session.
+
+    ```
+    $HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount <input1>
<output1> <input2> <output2> <input3> <output3> ...
+    ```
+
+    The above will run multiple DAGs for each input-output pair.
+
+    To use TEZ sessions, set -DUSE_TEZ_SESSION=true
+
+    ```
+    $HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount -DUSE_TEZ_SESSION=true
<input1> <output1> <input2> <output2>
+    ```


Mime
View raw message