hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <>
Subject [jira] [Commented] (HIVE-2340) optimize orderby followed by a groupby
Date Wed, 06 Feb 2013 08:59:14 GMT


Phabricator commented on HIVE-2340:

navis has commented on the revision "HIVE-2340 [jira] optimize orderby followed by a groupby".

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ ok.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ I wish
I could but CommonJoinResolver is a physical optimizer, which means there is no RS-RS operator
tree which could me merged on that stage.

  I'm thinking of disabling this optimization if user configured
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ I'll add
more explanations on hive-default.xml.template
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ For rules
with same cost, DefaultRuleDispatcher selects last one, something like this,
  if ((cost >= 0) && (cost <= minCost)) {
      minCost = cost;
      rule = r;
  So R2 will be selected.
  conf/hive-default.xml.template:1034 It's commented on

  This optimization merges two RSs by moving key/parts/num-reducers of child-RS to parent-RS,
which means if num-reducer of child-RS is fixed (order by or forced bucketing) and small,
it can resulted to very slow, single MR. For preventing this, the configuration makes min
threshold for applying this optimization. It's not good enough, but I cannot think of better


To: JIRA, navis
Cc: hagleitn, njain

> optimize orderby followed by a groupby
> --------------------------------------
>                 Key: HIVE-2340
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Query Processor
>            Reporter: Navis
>            Assignee: Navis
>            Priority: Minor
>              Labels: perfomance
>         Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.5.patch, HIVE-2340.1.patch.txt, HIVE-2340.D1209.10.patch,
HIVE-2340.D1209.6.patch, HIVE-2340.D1209.7.patch, HIVE-2340.D1209.8.patch, HIVE-2340.D1209.9.patch,
> Before implementing optimizer for JOIN-GBY, try to implement RS-GBY optimizer(cluster-by
following group-by).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message