hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ransom.hezhiqiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-1065) When data is large,a reducer's memory can not hold the big data by using list to store the data, and get an error "out of memory"
Date Wed, 16 May 2012 12:21:02 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276691#comment-13276691
] 

ransom.hezhiqiang commented on HIVE-1065:
-----------------------------------------

MapJoin,bucket mapJoin,sorted bucket MapJoin.
and also can used Stream table to steam the big table. 
if two table's data is also very large .
try skew join. change the tasknum.
                
> When data is large,a reducer's memory can not hold the big data by using list to store
the data, and get an error "out of memory"
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-1065
>                 URL: https://issues.apache.org/jira/browse/HIVE-1065
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.3.0
>         Environment: redhat enterprise 4.0
>            Reporter: fuyangchang
>            Priority: Critical
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When data is large,I do a "join" operation ,a reducer's memory can not hold the big data
by using list to store the data, and get an error "out of memory"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message