spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: [SPARK-13227] Risky apply() in OpenHashMap
Date Tue, 19 Apr 2016 01:51:43 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.6 58dfba66e -> cd812143d

[SPARK-13227] Risky apply() in OpenHashMap

It might confuse the future developers when they use OpenHashMap.apply() with a numeric value

null.asInstance[Int], null.asInstance[Long], null.asInstace[Float] and null.asInstance[Double]
will return 0/0.0/0L, which might confuse the developer if the value set contains 0/0.0/0L
with an existing key

The current patch only adds the comments describing the issue, with the respect to apply the
minimum changes to the code base

The more direct, yet more aggressive, approach is use Option as the return type

andrewor14  JoshRosen  any thoughts about how to avoid the potential issue?

Author: CodingCat <>

Closes #11107 from CodingCat/SPARK-13227.

(cherry picked from commit 4b3d1294aeecc0001a7fa48c92796e6075d34540)
Signed-off-by: Reynold Xin <>


Branch: refs/heads/branch-1.6
Commit: cd812143d350fd059e5d9db5d6065fdf6b0c88a7
Parents: 58dfba6
Author: CodingCat <>
Authored: Mon Apr 18 18:51:23 2016 -0700
Committer: Reynold Xin <>
Committed: Mon Apr 18 18:51:40 2016 -0700

 .../main/scala/org/apache/spark/util/collection/OpenHashMap.scala | 3 +++
 1 file changed, 3 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala b/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
index efc2482..3bc6a32 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
@@ -28,6 +28,9 @@ import org.apache.spark.annotation.DeveloperApi
  * space overhead.
  * Under the hood, it uses our OpenHashSet implementation.
+ *
+ * NOTE: when using numeric type as the value type, the user of this class should be careful
+ * distinguish between the 0/0.0/0L and non-exist value

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message