spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: [SPARK-15821][DOCS] Include parallel build info
Date Tue, 14 Jun 2016 12:59:09 GMT
Repository: spark
Updated Branches:
  refs/heads/master 96c3500c6 -> a431e3f1f

[SPARK-15821][DOCS] Include parallel build info

## What changes were proposed in this pull request?

We should mention that users can build Spark using multiple threads to decrease build times;
either here or in "Building Spark"

## How was this patch tested?

Built on machines with between one core to 192 cores using mvn -T 1C and observed faster build
times with no loss in stability

In response to the question here I think
we should suggest this option as we know it works for Spark and can result in faster builds

Author: Adam Roberts <>

Closes #13562 from a-roberts/patch-3.


Branch: refs/heads/master
Commit: a431e3f1f8575e2498650ac767e69fbc903e9929
Parents: 96c3500
Author: Adam Roberts <>
Authored: Tue Jun 14 13:59:01 2016 +0100
Committer: Sean Owen <>
Committed: Tue Jun 14 13:59:01 2016 +0100

----------------------------------------------------------------------                | 2 ++
 dev/ | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/ b/
index d5804d1..c77c429 100644
--- a/
+++ b/
@@ -25,6 +25,8 @@ To build Spark and its example programs, run:
     build/mvn -DskipTests clean package
 (You do not need to do this if you downloaded a pre-built package.)
+You can build Spark using more than one thread by using the -T option with Maven, see ["Parallel
builds in Maven 3"](
 More detailed documentation is available from the project site, at
 ["Building Spark"](
 For developing Spark using an IDE, see [Eclipse](
diff --git a/dev/ b/dev/
index 4f7544f..9be4fdf 100755
--- a/dev/
+++ b/dev/
@@ -53,7 +53,7 @@ while (( "$#" )); do
       echo "Error: '--hadoop' is no longer supported:"
       echo "Error: use Maven profiles and options -Dhadoop.version and -Dyarn.version instead."
-      echo "Error: Related profiles include hadoop-2.2, hadoop-2.3 and hadoop-2.4."
+      echo "Error: Related profiles include hadoop-2.2, hadoop-2.3, hadoop-2.4, hadoop-2.6
and hadoop-2.7."
@@ -150,7 +150,7 @@ export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCac
 # Store the command as an array because $MVN variable might have spaces in it.
 # Normal quoting tricks don't work.
 # See:
-BUILD_COMMAND=("$MVN" clean package -DskipTests $@)
+BUILD_COMMAND=("$MVN" -T 1C clean package -DskipTests $@)
 # Actually build the jar
 echo -e "\nBuilding with..."

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message