spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@apache.org
Subject spark git commit: [SPARK-16699][SQL] Fix performance bug in hash aggregate on long string keys
Date Mon, 25 Jul 2016 04:54:47 GMT
Repository: spark
Updated Branches:
  refs/heads/master daace6014 -> 468a3c3ac


[SPARK-16699][SQL] Fix performance bug in hash aggregate on long string keys

In the following code in `VectorizedHashMapGenerator.scala`:
```
    def hashBytes(b: String): String = {
      val hash = ctx.freshName("hash")
      s"""
         |int $result = 0;
         |for (int i = 0; i < $b.length; i++) {
         |  ${genComputeHash(ctx, s"$b[i]", ByteType, hash)}
         |  $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) + ($result >>>
2);
         |}
       """.stripMargin
    }

```
when b=input.getBytes(), the current 2.0 code results in getBytes() being called n times,
n being length of input. getBytes() involves memory copy is thus expensive and causes a performance
degradation.
Fix is to evaluate getBytes() before the for loop.

Performance bug, no additional test added.

Author: Qifan Pu <qifan.pu@gmail.com>

Closes #14337 from ooq/SPARK-16699.

(cherry picked from commit d226dce12babcd9f30db033417b2b9ce79f44312)
Signed-off-by: Reynold Xin <rxin@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/468a3c3a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/468a3c3a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/468a3c3a

Branch: refs/heads/master
Commit: 468a3c3ac5d039f21613f9237c7bdef9b92f5fea
Parents: daace60
Author: Qifan Pu <qifan.pu@gmail.com>
Authored: Sun Jul 24 21:53:21 2016 -0700
Committer: Reynold Xin <rxin@databricks.com>
Committed: Sun Jul 24 21:54:42 2016 -0700

----------------------------------------------------------------------
 .../sql/execution/aggregate/VectorizedHashMapGenerator.scala   | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/468a3c3a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
index 8a3f466..b4a9059 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
@@ -313,10 +313,12 @@ class VectorizedHashMapGenerator(
     def hashLong(l: String): String = s"long $result = $l;"
     def hashBytes(b: String): String = {
       val hash = ctx.freshName("hash")
+      val bytes = ctx.freshName("bytes")
       s"""
          |int $result = 0;
-         |for (int i = 0; i < $b.length; i++) {
-         |  ${genComputeHash(ctx, s"$b[i]", ByteType, hash)}
+         |byte[] $bytes = $b;
+         |for (int i = 0; i < $bytes.length; i++) {
+         |  ${genComputeHash(ctx, s"$bytes[i]", ByteType, hash)}
          |  $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) + ($result
>>> 2);
          |}
        """.stripMargin


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message