hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-963) no out of memory errors for skewed join
Date Fri, 04 Dec 2009 07:23:20 GMT

     [ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ning Zhang updated HIVE-963:
----------------------------

    Attachment: HIVE-963.patch

HIVE-963.patch is attached. This patch 
 1) introduced a new file HashMapWrapper.java 
 2) changed JoinOperator, CommonJoinOperator, and MapJoinOperator to use HashMapWrapper.
 3) modified JDBM files to add support to disable cache and accepting File as the input parameter
to record manager. 

> no out of memory errors for skewed join
> ---------------------------------------
>
>                 Key: HIVE-963
>                 URL: https://issues.apache.org/jira/browse/HIVE-963
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Ning Zhang
>         Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for
a given key exceed a given number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message