Return-Path: 
 <commits-return-3597-apmail-spark-commits-archive=spark.apache.org@spark.apache.org>
X-Original-To: apmail-spark-commits-archive@minotaur.apache.org
Delivered-To: apmail-spark-commits-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC0C1118BD
	for <apmail-spark-commits-archive@minotaur.apache.org>;
 Wed, 16 Apr 2014 16:14:51 +0000 (UTC)
Received: (qmail 37203 invoked by uid 500); 16 Apr 2014 16:14:48 -0000
Delivered-To: apmail-spark-commits-archive@spark.apache.org
Received: (qmail 37079 invoked by uid 500); 16 Apr 2014 16:14:43 -0000
Mailing-List: contact commits-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:commits-help@spark.apache.org>
List-Unsubscribe: <mailto:commits-unsubscribe@spark.apache.org>
List-Post: <mailto:commits@spark.apache.org>
List-Id: <commits.spark.apache.org>
Reply-To: dev@spark.apache.org
Delivered-To: mailing list commits@spark.apache.org
Received: (qmail 36431 invoked by uid 99); 16 Apr 2014 16:14:27 -0000
Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org)
 (140.211.11.114)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2014 16:14:27 +0000
Received: by tyr.zones.apache.org (Postfix, from userid 65534)
	id CAFF1931ECA; Wed, 16 Apr 2014 16:14:26 +0000 (UTC)
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: pwendell@apache.org
To: commits@spark.apache.org
Message-Id: <5eb3ac478ef845afa2c81f35d3ff550b@git.apache.org>
X-Mailer: ASF-Git Admin Mailer
Subject: git commit: update spark.default.parallelism
Date: Wed, 16 Apr 2014 16:14:26 +0000 (UTC)

Repository: spark
Updated Branches:
  refs/heads/master fec462c15 -> 9edd88782


update spark.default.parallelism

actually, the value 8 is only valid in mesos fine-grained mode :
<code>
  override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
   conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
  }
</code>

Author: Chen Chao <crazyjvm@gmail.com>

Closes #389 from CrazyJvm/patch-2 and squashes the following commits:

84a7fe4 [Chen Chao] miss </li> at the end of every single line
04a9796 [Chen Chao] change format
ee0fae0 [Chen Chao] update spark.default.parallelism


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9edd8878
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9edd8878
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9edd8878

Branch: refs/heads/master
Commit: 9edd88782e0268439c5ab57400d6a7ab432fc269
Parents: fec462c
Author: Chen Chao <crazyjvm@gmail.com>
Authored: Wed Apr 16 09:14:18 2014 -0700
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Wed Apr 16 09:14:18 2014 -0700

----------------------------------------------------------------------
 docs/configuration.md | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/9edd8878/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index f3bfd03..a302983 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -96,7 +96,13 @@ Apart from these, the following properties are also available, and may be useful
 <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
 <tr>
   <td>spark.default.parallelism</td>
-  <td>8</td>
+  <td>
+    <ul>
+      <li>Mesos fine grained mode: 8</li>
+      <li>Local mode: core number of the local machine</li>
+      <li>Others: total core number of all executor nodes or 2, whichever is larger</li>
+    </ul>
+  </td>
   <td>
     Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
     <code>reduceByKey</code>, etc) when not set by user.