Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5EBB3173B9 for ; Sun, 11 Jan 2015 21:50:57 +0000 (UTC) Received: (qmail 28790 invoked by uid 500); 11 Jan 2015 21:50:58 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 28760 invoked by uid 500); 11 Jan 2015 21:50:58 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@spark.apache.org Received: (qmail 28751 invoked by uid 99); 11 Jan 2015 21:50:58 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Jan 2015 21:50:58 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 860999A9DB1; Sun, 11 Jan 2015 21:50:58 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: adav@apache.org To: commits@spark.apache.org Message-Id: <2536ee5238ed4848bfbd948d2e16ef81@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: spark git commit: [SPARK-5073] spark.storage.memoryMapThreshold have two default value Date: Sun, 11 Jan 2015 21:50:58 +0000 (UTC) Repository: spark Updated Branches: refs/heads/master 331326090 -> 1656aae2b [SPARK-5073] spark.storage.memoryMapThreshold have two default value Because major OS page sizes is about 4KB, the default value of spark.storage.memoryMapThreshold is integrated to 2 * 4096 Author: lewuathe Closes #3900 from Lewuathe/integrate-memoryMapThreshold and squashes the following commits: e417acd [lewuathe] [SPARK-5073] Update docs/configuration 834aba4 [lewuathe] [SPARK-5073] Fix style adcea33 [lewuathe] [SPARK-5073] Integrate memory map threshold to 2MB fcce2e5 [lewuathe] [SPARK-5073] spark.storage.memoryMapThreshold have two default value Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1656aae2 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1656aae2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1656aae2 Branch: refs/heads/master Commit: 1656aae2b4e8b026f8cfe782519f72d32ed2b291 Parents: 3313260 Author: lewuathe Authored: Sun Jan 11 13:50:42 2015 -0800 Committer: Aaron Davidson Committed: Sun Jan 11 13:50:42 2015 -0800 ---------------------------------------------------------------------- core/src/main/scala/org/apache/spark/storage/DiskStore.scala | 3 ++- docs/configuration.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/1656aae2/core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/storage/DiskStore.scala b/core/src/main/scala/org/apache/spark/storage/DiskStore.scala index 8dadf67..61ef5ff 100644 --- a/core/src/main/scala/org/apache/spark/storage/DiskStore.scala +++ b/core/src/main/scala/org/apache/spark/storage/DiskStore.scala @@ -31,7 +31,8 @@ import org.apache.spark.util.Utils private[spark] class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManager) extends BlockStore(blockManager) with Logging { - val minMemoryMapBytes = blockManager.conf.getLong("spark.storage.memoryMapThreshold", 2 * 4096L) + val minMemoryMapBytes = blockManager.conf.getLong( + "spark.storage.memoryMapThreshold", 2 * 1024L * 1024L) override def getSize(blockId: BlockId): Long = { diskManager.getFile(blockId.name).length http://git-wip-us.apache.org/repos/asf/spark/blob/1656aae2/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 2add485..f292bfb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -678,7 +678,7 @@ Apart from these, the following properties are also available, and may be useful spark.storage.memoryMapThreshold - 8192 + 2097152 Size of a block, in bytes, above which Spark memory maps when reading a block from disk. This prevents Spark from memory mapping very small blocks. In general, memory --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org For additional commands, e-mail: commits-help@spark.apache.org