Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 117EA104E1 for ; Sat, 19 Oct 2013 03:32:15 +0000 (UTC) Received: (qmail 82750 invoked by uid 500); 19 Oct 2013 03:31:47 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 82618 invoked by uid 500); 19 Oct 2013 03:31:40 -0000 Mailing-List: contact commits-help@spark.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.incubator.apache.org Delivered-To: mailing list commits@spark.incubator.apache.org Received: (qmail 82525 invoked by uid 99); 19 Oct 2013 03:31:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Oct 2013 03:31:39 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 19 Oct 2013 03:31:34 +0000 Received: (qmail 80995 invoked by uid 99); 19 Oct 2013 03:31:09 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Oct 2013 03:31:09 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 412C944126; Sat, 19 Oct 2013 03:31:08 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: matei@apache.org To: commits@spark.incubator.apache.org Date: Sat, 19 Oct 2013 03:31:13 -0000 Message-Id: In-Reply-To: <64395da14e284d4b999b86b72d561ec9@git.apache.org> References: <64395da14e284d4b999b86b72d561ec9@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [07/13] git commit: Code styling. Updated doc. X-Virus-Checked: Checked by ClamAV on apache.org Code styling. Updated doc. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/35b2415f Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/35b2415f Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/35b2415f Branch: refs/heads/master Commit: 35b2415fb32cea506c60d6f1f824ac194f4604f8 Parents: e663750 Author: Mosharaf Chowdhury Authored: Thu Oct 17 13:14:12 2013 -0700 Committer: Mosharaf Chowdhury Committed: Thu Oct 17 13:14:12 2013 -0700 ---------------------------------------------------------------------- .../scala/org/apache/spark/broadcast/TorrentBroadcast.scala | 8 ++++---- docs/configuration.md | 8 ++++++++ 2 files changed, 12 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/35b2415f/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala b/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala index 8c23584..073a0a5 100644 --- a/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala +++ b/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala @@ -84,7 +84,7 @@ extends Broadcast[T](id) with Logging with Serializable { val start = System.nanoTime logInfo("Started reading broadcast variable " + id) - // Master might send invalid values + // Initialize @transient variables that will receive garbage values from the master. resetWorkerVariables() if (receiveBroadcast(id)) { @@ -135,8 +135,9 @@ extends Broadcast[T](id) with Logging with Serializable { } attemptId -= 1 } - if (totalBlocks == -1) + if (totalBlocks == -1) { return false + } // Receive actual blocks val recvOrder = new Random().shuffle(Array.iterate(0, totalBlocks)(_ + 1).toList) @@ -151,8 +152,7 @@ extends Broadcast[T](id) with Logging with Serializable { pieceId, arrayOfBlocks(pid), StorageLevel.MEMORY_AND_DISK, true) case None => - throw new SparkException( - "Failed to get " + pieceId + " of " + broadcastId) + throw new SparkException("Failed to get " + pieceId + " of " + broadcastId) } } } http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/35b2415f/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 7940d41..c5900d0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -319,6 +319,14 @@ Apart from these, the following properties are also available, and may be useful Should be greater than or equal to 1. Number of allowed retries = this value - 1. + + spark.broadcast.blockSize + 4096 + + Size of each piece of a block in kilobytes for TorrentBroadcastFactory. + Too large a value decreases parallelism during broadcast (makes it slower); however, if it is too small, BlockManager might take a performance hit. + +