hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4041) Support multiple partitionings in a single Query
Date Fri, 15 Mar 2013 19:34:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603738#comment-13603738
] 

Phabricator commented on HIVE-4041:
-----------------------------------

hbutani has commented on the revision "HIVE-4041 [jira] Support multiple partitionings in
a single Query".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java:212 Yes the
translation code could do with your review.
  We were not paying much attention to optimization at the time we wrote it.
  So the TableFuncDef holds onto ShapeDetails(input, output..)
  The Shape class has Serde props that we use to reconstruct the OIs during runtime.
  This happens in PTFTranslator. Read the translate for WindowingSpec method (line 138) in
PTFTranslator.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkDeDuplication.java:415 Just added
a Jira 4186 for this.
  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java:482 yes I agree. I just don't
want to make these changes in this Jira. Want to only add multi partition support here.
  ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingComponentizer.java:38 I will add more
comments. So there is:
  - 1 PTFOp
  - It can contain one or more PTF invocations.
  - When the PTF is WindowTableFunc, it can contain 1 or more UDAFs; they have the same partitioning.
  - During translation we create a WindowingSpec for each destination with Windowing(over
clauses).
  - Here we then componentize the single WindowingSpec into multiple WindowingSpecs based
on the partitioning.
  ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingComponentizer.java:85 Yes the having
is to be removed. Haven't gotten around to it. Again didn't want to make this change in this
Jira. It is on my todo.
  ql/src/test/queries/clientpositive/windowing_multipartitioning.q:21 I added all the multipartition
tests from the spreadsheet except the ones that have no order. Those I will add once we resolve
how to handle no order.

REVISION DETAIL
  https://reviews.facebook.net/D9381

To: JIRA, ashutoshc, hbutani

                
> Support multiple partitionings in a single Query
> ------------------------------------------------
>
>                 Key: HIVE-4041
>                 URL: https://issues.apache.org/jira/browse/HIVE-4041
>             Project: Hive
>          Issue Type: Bug
>          Components: PTF-Windowing
>            Reporter: Harish Butani
>            Assignee: Harish Butani
>         Attachments: HIVE-4041.D9381.1.patch, WindowingComponentization.pdf
>
>
> Currently we disallow queries if the partition specifications of all Wdw fns are not
the same. We can relax this by generating multiple PTFOps based on the unique partitionings
in a Query. For partitionings that only differ in sort, we can introduce a sort step in between
PTFOps, which can happen in the same Reduce task.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message