spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-5112] Expose SizeEstimator as a developer api
Date Tue, 05 May 2015 11:38:58 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.4 93af96a2f -> 0327ca2b2


[SPARK-5112] Expose SizeEstimator as a developer api

"The best way to size the amount of memory consumption your dataset will require is to create
an RDD, put it into cache, and look at the SparkContext logs on your driver program. The logs
will tell you how much memory each partition is consuming, which you can aggregate to get
the total size of the RDD."
-the Tuning Spark page

This is a pain. It would be much nicer to expose simply functionality for understanding the
memory footprint of a Java object.

Author: Sandy Ryza <sandy@cloudera.com>

Closes #3913 from sryza/sandy-spark-5112 and squashes the following commits:

8d9e082 [Sandy Ryza] Add SizeEstimator in org.apache.spark
2e1a906 [Sandy Ryza] Revert "Move SizeEstimator out of util"
93f4cd0 [Sandy Ryza] Move SizeEstimator out of util
e21c1f4 [Sandy Ryza] Remove unused import
798ab88 [Sandy Ryza] Update documentation and add to SparkContext
34c523c [Sandy Ryza] SPARK-5112. Expose SizeEstimator as a developer api

(cherry picked from commit 4222da68dc5360b7a2a8b8bdce231e887ac2f044)
Signed-off-by: Sean Owen <sowen@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0327ca2b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0327ca2b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0327ca2b

Branch: refs/heads/branch-1.4
Commit: 0327ca2b2c6ffb4dbd2770861a3b3ba806d55e58
Parents: 93af96a
Author: Sandy Ryza <sandy@cloudera.com>
Authored: Tue May 5 12:38:46 2015 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Tue May 5 12:38:54 2015 +0100

----------------------------------------------------------------------
 .../scala/org/apache/spark/SizeEstimator.scala  | 44 ++++++++++++++++++++
 docs/tuning.md                                  | 10 +++--
 2 files changed, 50 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/0327ca2b/core/src/main/scala/org/apache/spark/SizeEstimator.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SizeEstimator.scala b/core/src/main/scala/org/apache/spark/SizeEstimator.scala
new file mode 100644
index 0000000..54fc3a8
--- /dev/null
+++ b/core/src/main/scala/org/apache/spark/SizeEstimator.scala
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in
+ * memory-aware caches.
+ *
+ * Based on the following JavaWorld article:
+ * http://www.javaworld.com/javaworld/javaqa/2003-12/02-qa-1226-sizeof.html
+ */
+@DeveloperApi
+object SizeEstimator {
+  /**
+   * :: DeveloperApi ::
+   * Estimate the number of bytes that the given object takes up on the JVM heap. The estimate
+   * includes space taken up by objects referenced by the given object, their references,
and so on
+   * and so forth.
+   *
+   * This is useful for determining the amount of heap space a broadcast variable will occupy
on
+   * each executor or the amount of space each object will take when caching objects in
+   * deserialized form. This is not the same as the serialized size of the object, which
will
+   * typically be much smaller.
+   */
+  @DeveloperApi
+  def estimate(obj: AnyRef): Long = org.apache.spark.util.SizeEstimator.estimate(obj)
+}

http://git-wip-us.apache.org/repos/asf/spark/blob/0327ca2b/docs/tuning.md
----------------------------------------------------------------------
diff --git a/docs/tuning.md b/docs/tuning.md
index 1cb223e..572c727 100644
--- a/docs/tuning.md
+++ b/docs/tuning.md
@@ -94,11 +94,13 @@ We will then cover tuning Spark's cache size and the Java garbage collector.
 
 ## Determining Memory Consumption
 
-The best way to size the amount of memory consumption your dataset will require is to create
an RDD, put it into cache, and look at the SparkContext logs on your driver program. The logs
will tell you how much memory each partition is consuming, which you can aggregate to get
the total size of the RDD. You will see messages like this:
+The best way to size the amount of memory consumption a dataset will require is to create
an RDD, put it
+into cache, and look at the "Storage" page in the web UI. The page will tell you how much
memory the RDD
+is occupying.
 
-    INFO BlockManagerMasterActor: Added rdd_0_1 in memory on mbk.local:50311 (size: 717.5
KB, free: 332.3 MB)
-
-This means that partition 1 of RDD 0 consumed 717.5 KB.
+To estimate the memory consumption of a particular object, use `SizeEstimator`'s `estimate`
method
+This is useful for experimenting with different data layouts to trim memory usage, as well
as
+determining the amount of space a broadcast variable will occupy on each executor heap.
 
 ## Tuning Data Structures
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message