hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7856) Enable parallelism in Reduce Side Join [Spark Branch]
Date Fri, 22 Aug 2014 21:52:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107577#comment-14107577
] 

Szehon Ho commented on HIVE-7856:
---------------------------------

Some preliminary research on how this would work, we would use the SPARK-2978 group+sort in
the shuffle phase, and we can use the HashPartitioner to decide the partition.  Today in MapReduce
we are using the default partitioner (hash partitioner), as the HiveKey has a pre-computed
hash-code field, that directs it to the right HiveReduce function. 

One issue is that HiveBaseFunctionResultList clones the HiveKey as BytesWritable during mapper-output
generation, losing the hashcode.  We need to make a change there to preserve the hashcode,
for this to work.

> Enable parallelism in Reduce Side Join [Spark Branch]
> -----------------------------------------------------
>
>                 Key: HIVE-7856
>                 URL: https://issues.apache.org/jira/browse/HIVE-7856
>             Project: Hive
>          Issue Type: New Feature
>          Components: Spark
>            Reporter: Szehon Ho
>
> This is dependent on new transformation to be provided by SPARK-2978, see parent JIRA
for details.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message