hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangmeng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-964) handle skewed keys for a join in a separate job
Date Tue, 22 Jul 2014 08:25:39 GMT

    [ https://issues.apache.org/jira/browse/HIVE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069997#comment-14069997
] 

wangmeng commented on HIVE-964:
-------------------------------

if the two join tables  have the same big skew key on one value (for example ,select *  from
 table A join B  on  A.id=b.id,  both table A  and B  have  a lot of  keys on id=1,  in  this
 case ,map join  will OOM),  how  to fix this  case?  Will  it  rollback  to common  join
? 

> handle skewed keys for a join in a separate job
> -----------------------------------------------
>
>                 Key: HIVE-964
>                 URL: https://issues.apache.org/jira/browse/HIVE-964
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: He Yongqiang
>             Fix For: 0.6.0
>
>         Attachments: hive-964-2009-12-17.txt, hive-964-2009-12-28-2.patch, hive-964-2009-12-29-4.patch,
hive-964-2010-01-08.patch, hive-964-2010-01-13-2.patch, hive-964-2010-01-14-3.patch, hive-964-2010-01-15-4.patch
>
>
> The skewed keys can be written to a temporary table or file, and a followup conditional
task can be used to perform the join on those keys.
> As a first step, JDBM can be used for those keys



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message