hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1307) More generic and efficient merge method
Date Fri, 20 Aug 2010 21:33:17 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900866#action_12900866
] 

Namit Jain commented on HIVE-1307:
----------------------------------

TestParse is failing on both 17 and 20.

On 17, the following tests are failing in 17:

bucketmapjoin1.q
bucketmapjoin2.q
bucketmapjoin3.q


All of them are log file updates - can you fix the log files and submit a new patch ?

> More generic and efficient merge method
> ---------------------------------------
>
>                 Key: HIVE-1307
>                 URL: https://issues.apache.org/jira/browse/HIVE-1307
>             Project: Hadoop Hive
>          Issue Type: New Feature
>    Affects Versions: 0.6.0
>            Reporter: Ning Zhang
>            Assignee: Ning Zhang
>             Fix For: 0.7.0
>
>         Attachments: HIVE-1307.0.patch, HIVE-1307.2.patch, HIVE-1307.3.patch, HIVE-1307.3_java.patch,
HIVE-1307.4.patch, HIVE-1307.5.patch, HIVE-1307.6.patch, HIVE-1307.patch, HIVE-1307_java_only.patch
>
>
> Currently if hive.merge.mapfiles/mapredfiles=true, a new mapreduce job is create to read
the input files and output to one reducer for merging. This MR job is created at compile time
and one MR job for one partition. In the case of dynamic partition case, multiple partitions
could be created at execution time and generating merging MR job at compile time is impossible.

> We should generalize the merge framework to allow multiple partitions and most of the
time a map-only job should be sufficient if we use CombineHiveInputFormat. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message