hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <>
Subject [jira] [Commented] (HIVE-3562) Some limit can be pushed down to map stage
Date Tue, 08 Jan 2013 08:42:14 GMT


Phabricator commented on HIVE-3562:

njain has commented on the revision "HIVE-3562 [jira] Some limit can be pushed down to map

  I thought about it, even with group bys, my question is still valid.
  I think, there is a bug.

  Do you think it would be simpler to allocate a heap with (upto) topN entries instead - throw
  the memory threshold out. If limit < threshold, use this optimization, otherwise just
ignore this

  ql/src/java/org/apache/hadoop/hive/ql/exec/ Isn't there a bug
here ?

  You are using keyValues last entry to figure out whether it needs to be expanded or not.
  It may have an issue at the boundary - say entry 40th when a legit. entry is found.
  It might be simpler to pass the fact whether the entry was found or not.
  ql/src/java/org/apache/hadoop/hive/ql/exec/ This is not true
if an entry is being inserted in between.
  I mean, if topN is 100, and we already have 100 entries.

  If we are inserting 50th entry, we should not be increasing usage


To: JIRA, tarball, navis
Cc: njain

> Some limit can be pushed down to map stage
> ------------------------------------------
>                 Key: HIVE-3562
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Navis
>            Assignee: Navis
>            Priority: Trivial
>         Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch, HIVE-3562.D5967.3.patch
> Queries with limit clause (with reasonable number), for example
> {noformat}
> select * from src order by key limit 10;
> {noformat}
> makes operator tree, 
> But LIMIT can be partially calculated in RS, reducing size of shuffling.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message