spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject [2/3] git commit: more docs
Date Sat, 07 Dec 2013 01:29:21 GMT
more docs


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/e2c2914f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/e2c2914f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/e2c2914f

Branch: refs/heads/master
Commit: e2c2914faaf3d8ab849ad0477152f64df7adf4c3
Parents: f2fb4b4
Author: Ali Ghodsi <alig@cs.berkeley.edu>
Authored: Fri Dec 6 16:54:06 2013 -0800
Committer: Ali Ghodsi <alig@cs.berkeley.edu>
Committed: Fri Dec 6 16:54:06 2013 -0800

----------------------------------------------------------------------
 docs/cluster-overview.md | 2 +-
 docs/index.md            | 4 +++-
 docs/running-on-yarn.md  | 2 +-
 3 files changed, 5 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/e2c2914f/docs/cluster-overview.md
----------------------------------------------------------------------
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md
index 5927f73..e167032 100644
--- a/docs/cluster-overview.md
+++ b/docs/cluster-overview.md
@@ -45,7 +45,7 @@ The system currently supports three cluster managers:
   easy to set up a cluster.
 * [Apache Mesos](running-on-mesos.html) -- a general cluster manager that can also run Hadoop
MapReduce
   and service applications.
-* [Hadoop YARN](running-on-yarn.html) -- the resource manager in Hadoop 2.0.
+* [Hadoop YARN](running-on-yarn.html) -- the resource manager in Hadoop 2.
 
 In addition, Spark's [EC2 launch scripts](ec2-scripts.html) make it easy to launch a standalone
 cluster on Amazon EC2.

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/e2c2914f/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index 56e1142..bbb2733 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -63,7 +63,9 @@ In addition, if you wish to run Spark on [YARN](running-on-yarn.md), set
 
     SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
 
-(Note that on Windows, you need to set the environment variables on separate lines, e.g.,
`set SPARK_HADOOP_VERSION=1.2.1`.)
+Note that on Windows, you need to set the environment variables on separate lines, e.g.,
`set SPARK_HADOOP_VERSION=1.2.1`.
+
+For this version of Spark (0.8.1) Hadoop 2.2.x (or newer) users will have to build Spark
and publish it locally. See [Launching Spark on YARN](running-on-yarn.md). This is needed
because Hadoop 2.2 has non backwards compatible API changes.
 
 # Where to Go from Here
 

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/e2c2914f/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 3ec656c..ae65127 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -114,7 +114,7 @@ For example:
 
 # Building Spark for Hadoop/YARN 2.2.x
 
-Hadoop 2.2.x users must build Spark and publish it locally. The SBT build process handles
Hadoop 2.2.x as a special case. This version of Hadoop has new YARN API changes and depends
on a Protobuf version (2.5) that is not compatible with the Akka version (2.0.5) that Spark
uses. Therefore, if the Hadoop version (e.g. set through ```SPARK_HADOOP_VERSION```) starts
with 2.2.0 or higher then the build process will depend on Akka artifacts distributed by the
Spark project compatible with Protobuf 2.5. Furthermore, the build process then uses the directory
```new-yarn``` (stead of ```yarn```), which supports the new YARN API. The build process should
seamlessly work out of the box. 
+Hadoop 2.2.x users must build Spark and publish it locally. The SBT build process handles
Hadoop 2.2.x as a special case. This version of Hadoop has new YARN API changes and depends
on a Protobuf version (2.5) that is not compatible with the Akka version (2.0.5) that Spark
uses. Therefore, if the Hadoop version (e.g. set through ```SPARK_HADOOP_VERSION```) starts
with 2.2.0 or higher then the build process will depend on Akka artifacts distributed by the
Spark project compatible with Protobuf 2.5. Furthermore, the build process then uses the directory
```new-yarn``` (instead of ```yarn```), which supports the new YARN API. The build process
should seamlessly work out of the box. 
 
 See [Building Spark with Maven](building-with-maven.md) for instructions on how to build
Spark using the Maven process.
 


Mime
View raw message