hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <>
Subject [jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job
Date Tue, 26 Feb 2013 19:48:12 GMT


Vinod Kumar Vavilapalli updated HIVE-3952:

    Attachment: HIVE-3952-20130226.txt

Had a patch for a while but took a bit of time to clean it up. Attached. Here's what it does:
 - The changes are in CommonJoinResolver, which does the collapse of multi-way joins into
a single task doing all map-joins.
 - Every time a join is converted to map-join, I also inspect the child task to see if it
is a MR job and then merge M with MR.
 - Added a test to multiJoin1.q to test that a M-MR collapses into a single MR job.
 - The memory model after this patch is very complicated, it all depends on what operations
are performed in the second MR job. AFAIU, We also don't have a clear memory model for HIVE-3952
multi-way map-join too. So for now, I just added a config "hive.optimize.mapjoin.mapreduce"
to control this. I think we need a bigger JIRA to figure out memory restrictions when we have
these multiple optimizations in play.

Please review. Thanks!
> merge map-job followed by map-reduce job
> ----------------------------------------
>                 Key: HIVE-3952
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: HIVE-3952-20130226.txt
> Consider the query like:
> select count(*) FROM
>         ( select idOne, idTwo, value FROM
>               bigTable                                       
>               JOIN                                                                
>               smallTableOne on (bigTable.idOne = smallTableOne.idOne)               
>           ) firstjoin                                                             
>             JOIN                                                                  
>             smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
> where smallTableOne and smallTableTwo are smaller than
> is set to true.
> The joins are collapsed into mapjoins, and it leads to a map-only job
> (for the map-joins) followed by a map-reduce job (for the group by).
> Ideally, the map-only job should be merged with the following map-reduce job.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message