Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Fri, 23 Aug 2013 16:18:53 +0000 (UTC)
From: "Ashutosh Chauhan (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12611120.1349845960621.23653.1377274733880@arcas>
In-Reply-To: <JIRA.12611120.1349845960621@arcas>
References: <JIRA.12611120.1349845960621@arcas>
Subject: [jira] [Commented] (HIVE-3562) Some limit can be pushed down to map
 stage
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748676#comment-13748676 ] 

Ashutosh Chauhan commented on HIVE-3562:
----------------------------------------

[~navis] It occurred to me that this optimization will become very powerful in combination with HIVE-4002 Imagine a case where there is a limit which can be pushed up in front of last RS. Than mappers will output very little data and with HIVE-4002 we can eliminate reducer altogether. Though these two optimizations cannot occur simultaneously in their current form since RSHash is implemented inside RS. We need to reimplement RSHash in FS. Alternative approach could be to implement RSHash as an operator which can than be put in front of either RS or FS. What do you think?
                
> Some limit can be pushed down to map stage
> ------------------------------------------
>
>                 Key: HIVE-3562
>                 URL: https://issues.apache.org/jira/browse/HIVE-3562
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Navis
>            Assignee: Navis
>            Priority: Trivial
>         Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch, HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch, HIVE-3562.D5967.6.patch
>
>
> Queries with limit clause (with reasonable number), for example
> {noformat}
> select * from src order by key limit 10;
> {noformat}
> makes operator tree, 
> TS-SEL-RS-EXT-LIMIT-FS
> But LIMIT can be partially calculated in RS, reducing size of shuffling.
> TS-SEL-RS(TOP-N)-EXT-LIMIT-FS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira