hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerry Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-3934) Put tag in value for join with map reduce
Date Thu, 24 Jan 2013 02:47:19 GMT
Jerry Chen created HIVE-3934:
--------------------------------

             Summary: Put tag in value for join with map reduce
                 Key: HIVE-3934
                 URL: https://issues.apache.org/jira/browse/HIVE-3934
             Project: Hive
          Issue Type: Improvement
          Components: Query Processor, Serializers/Deserializers
    Affects Versions: 0.11.0
            Reporter: Jerry Chen


While trying to facilitate hash-based map reduce, I found that for join with map reduce in
hive, the tag is appended to the key writable. This is quite a hinder for facilitating other
runtime map reduce implementation of map reduce computation model such as hash-based map reduce.
For example, whent the tag was in the key, there are some special things must be cared, such
as,

1. HiveKey must handles specially for the hash code for properly partition the keys between
the reduce.
2. The key in map reduce's view is actually key + tag and which makes map reduce sort a compulsory
to satisfy the need of hive to group the key in reduce side. This disables or hinders hash-based
map reduce because group by key + tag will make no sense to hive. 
3. ExecReducer must check the real key boundary by stripping out the tag for startGroup and
endGroup calls to the operator. While without the tag, each reduce call is a natural key boundary.

Considering append the tag as the last byte to the value writable which can avoid all the
above things and fit naturually to map reduce computation model.

I see the code in JoinOperator which will generate join results ealier which assumes on the
fact that the tag is sorted. This only useful when there are many many rows with the same
key in both join tables which is not a compulsory for most cases.

Let's disucss the possibiblity of "tag in value" approach.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message