hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <>
Subject [jira] [Commented] (HIVE-3733) Improve Hive's logic for conditional merge
Date Fri, 07 Dec 2012 05:55:20 GMT


Namit Jain commented on HIVE-3733:

[~pkamath], can you load the complete patch ?

Ideally, the sub-query should not matter. By the time you are looking at the FileSinkDesc
for insert, the union should have been processed.
You should not look at the complete stack, but the first operator which would break the tree,
which is union in this case.

Thinking more about it, this approach is fairly difficult to get right. The more I think about
it, the more I like the earlier idea, of moving
the merge to a physical optimizer. The tasks have already been broken up, and the stack would
be well defined. We dont have to hack this up.
[~kevinwilfong], what do you think ? 
> Improve Hive's logic for conditional merge
> ------------------------------------------
>                 Key: HIVE-3733
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: HIVE-3733.1.patch.txt, HIVE-3733.3.patch.txt, HIVE-3733.4.patch.txt
> If the config hive.merge.mapfiles is set to true and hive.merge.mapredfiles is set to
false then when hive encounters a FileSinkOperator when generating map reduce tasks, it will
look at the entire job to see if it has a reducer, if it does it will not merge. Instead it
should be check if the FileSinkOperator is a child of the reducer. This means that outputs
generated in the mapper will be merged, and outputs generated in the reducer will not be,
the intended effect of setting those configs.
> Simple repro:
> set hive.merge.mapfiles=true;
> set hive.merge.mapredfiles=false;
> FROM <input_table>
> INSERT OVERWRITE TABLE <output_table1> SELECT key, COUNT(*) group by key
> The output should contain a Conditional Operator, Mapred Stages, and Move tasks

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message