hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-2566) reduce the number map-reduce jobs for union all
Date Wed, 09 Nov 2011 18:17:51 GMT
reduce the number map-reduce jobs for union all
-----------------------------------------------

                 Key: HIVE-2566
                 URL: https://issues.apache.org/jira/browse/HIVE-2566
             Project: Hive
          Issue Type: Improvement
            Reporter: Namit Jain
            Assignee: Namit Jain


A query like:

select s.key, s.value from (
  select key, value from src2  where key < 10
  union all 
  select key, value from src3  where key < 10
  union all 
  select key, value from src4  where key < 10
  union all 
  select key, count(1) as value from src5 group by key
)s;

should run the last sub-query 
'select key, count(1) as value from src5 group by key'
as a map-reduce job.

And then the union should be a map-only job reading from the first 3 map-only subqueries
and the output of the last map-reduce job.

The current plan is very inefficient.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message