hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <>
Subject [jira] [Commented] (HIVE-7856) Enable parallelism in Reduce Side Join [Spark Branch]
Date Fri, 22 Aug 2014 21:52:12 GMT


Szehon Ho commented on HIVE-7856:

Some preliminary research on how this would work, we would use the SPARK-2978 group+sort in
the shuffle phase, and we can use the HashPartitioner to decide the partition.  Today in MapReduce
we are using the default partitioner (hash partitioner), as the HiveKey has a pre-computed
hash-code field, that directs it to the right HiveReduce function. 

One issue is that HiveBaseFunctionResultList clones the HiveKey as BytesWritable during mapper-output
generation, losing the hashcode.  We need to make a change there to preserve the hashcode,
for this to work.

> Enable parallelism in Reduce Side Join [Spark Branch]
> -----------------------------------------------------
>                 Key: HIVE-7856
>                 URL:
>             Project: Hive
>          Issue Type: New Feature
>          Components: Spark
>            Reporter: Szehon Ho
> This is dependent on new transformation to be provided by SPARK-2978, see parent JIRA
for details.

This message was sent by Atlassian JIRA

View raw message