hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lefty Leverenz <leftylever...@gmail.com>
Subject Re: doc on predicate pushdown in joins
Date Mon, 16 Dec 2013 11:42:01 GMT
>
> This is old code, hard to attribute to  jiras
>

Okay, HIVE-2337 was way back in release 0.8.0 so I think we can safely omit
version information.  If anyone objects, I can link the wikidoc to that
ticket.  (And I've marked HIVE-2337 as being related to HIVE-279 in the
JIRA.)

Thanks for all the help.

-- Lefty


On Wed, Dec 11, 2013 at 7:03 PM, Harish Butani <hbutani@hortonworks.com>wrote:

> I see.  Let's leave it in.
>
> This is old code, hard to attribute to  jiras:
> - The PPD code comes from: HIVE-279, HIVE-2337
> - I cannot tell when the join condition parsing code was added.
>
> regards,
> Harish.
>
> On Dec 11, 2013, at 5:17 PM, Lefty Leverenz <leftyleverenz@gmail.com>
> wrote:
>
> > Maybe we should remove the section on Hive Implementation here.
> > It is in the Design doc; this information only concerns developers.
>
> But this is the Design doc (unless there's another one somewhere -- maybe
> attached to a JIRA ticket?) and it's in the Resources for Contributors part
> of the wiki, so it seems appropriate to me.  I'll delete the implementation
> section if that's your preference.
>
> Here are the links again, with fixes:
>
>    - Design Docs<https://cwiki.apache.org/confluence/display/Hive/DesignDocs> (bottom
>    of list)
>    - Predicate Pushdown Rules<https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior#OuterJoinBehavior-PredicatePushdownRules>
>
>
> Speaking of JIRA tickets, is there one for this and should I add any
> version information?
>
> -- Lefty
>
>
> On Wed, Dec 11, 2013 at 7:59 AM, Harish Butani <hbutani@hortonworks.com>wrote:
>
>> getQualifiedAliases is a private method in JoinPPD.
>>
>> Maybe we should remove the section on Hive Implementation here. It is in
>> the Design doc; this information only concerns developers.
>>
>> regards,
>> Harish.
>>
>>
>> On Dec 11, 2013, at 3:05 AM, Lefty Leverenz <leftyleverenz@gmail.com>
>> wrote:
>>
>> Happy to fix the sentence and the link.  I pointed out the name change
>> just so you would review it, so please don't apologize!
>>
>> One more question:  why am I not finding getQualifiedAliases() in the
>> SemanticAnalyzer class?  It turns up in OpProcFactory.java with javadoc
>> comments, but I can't find it anywhere in the API docs -- not even in the
>> index (Hive 0.12.0 API <http://hive.apache.org/docs/r0.12.0/api/>):
>>
>> *getQMap()*<http://hive.apache.org/docs/r0.12.0/api/org/apache/hadoop/hive/ql/QTestUtil.html#getQMap()>
-
>> Method in class org.apache.hadoop.hive.ql.QTestUtil<http://hive.apache.org/docs/r0.12.0/api/org/apache/hadoop/hive/ql/QTestUtil.html>
>>  *getQualifiedName()*<http://hive.apache.org/docs/r0.12.0/api/org/apache/hadoop/hive/serde2/typeinfo/TypeInfo.html#getQualifiedName()>
-
>> Method in class org.apache.hadoop.hive.serde2.typeinfo.TypeInfo<http://hive.apache.org/docs/r0.12.0/api/org/apache/hadoop/hive/serde2/typeinfo/TypeInfo.html>
String
>> representing the qualified type name.*getQualifiers()*<http://hive.apache.org/docs/r0.12.0/api/org/apache/hive/service/cli/thrift/TTypeQualifiers.html#getQualifiers()>
-
>> Method in class org.apache.hive.service.cli.thrift.TTypeQualifiers<http://hive.apache.org/docs/r0.12.0/api/org/apache/hive/service/cli/thrift/TTypeQualifiers.html>
>>  *getQualifiersSize()*<http://hive.apache.org/docs/r0.12.0/api/org/apache/hive/service/cli/thrift/TTypeQualifiers.html#getQualifiersSize()>
-
>> Method in class org.apache.hive.service.cli.thrift.TTypeQualifiers<http://hive.apache.org/docs/r0.12.0/api/org/apache/hive/service/cli/thrift/TTypeQualifiers.html>
>> Most mysterious.
>>
>> -- Lefty
>>
>>
>> On Tue, Dec 10, 2013 at 2:35 PM, Harish Butani <hbutani@hortonworks.com>wrote:
>>
>>> I can see why you would rename.
>>>
>>> But this sentence is not correct:
>>> 'Hive enforces the predicate pushdown rules by these methods in the
>>> SemanticAnalyzer and JoinPPD classes:'
>>>
>>> It should be:
>>> Hive enforces the rules by these methods in the SemanticAnalyzer and
>>> JoinPPD classes:
>>>
>>> (The implementation involves both predicate pushdown and analyzing join
>>> conditions)
>>> Sorry about this.
>>>
>>> So the link should say 'Hive Outer Join Behavior'
>>>
>>> regards,
>>> Harish.
>>>
>>>
>>> On Dec 10, 2013, at 2:01 PM, Lefty Leverenz <leftyleverenz@gmail.com>
>>> wrote:
>>>
>>> How's this?  Hive Implementation<https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior#OuterJoinBehavior-HiveImplementation>
>>>
>>> Also, I moved the link on the Design Docs page<https://cwiki.apache.org/confluence/display/Hive/DesignDocs>from
>>> *Proposed* to *Other*.  (It's called SQL Outer Join Predicate Pushdown
>>> Rules<https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior>
which
>>> doesn't match the title, but seems okay because it's more descriptive.)
>>>
>>> -- Lefty
>>>
>>>
>>> On Tue, Dec 10, 2013 at 7:27 AM, Harish Butani <hbutani@hortonworks.com>wrote:
>>>
>>>> You are correct, it is plural.
>>>>
>>>> regards,
>>>> Harish.
>>>>
>>>> On Dec 10, 2013, at 4:03 AM, Lefty Leverenz <leftyleverenz@gmail.com>
>>>> wrote:
>>>>
>>>> Okay, then monospace with "()" after the method name is a good way to
>>>> show them:  parseJoinCondition() and getQualifiedAlias() ... but I
>>>> only found the latter pluralized, instead of singular, so should it be
>>>> getQualifiedAliases() or am I missing something?
>>>>
>>>> trunk> *grep -nr 'getQualifiedAlias' ./ql/src/java/* | grep -v 'svn'*
>>>>>
>>>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:221:
>>>>>> * the comments for getQualifiedAliases function.
>>>>>
>>>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:230:
>>>>>>    Set<String> aliases = getQualifiedAliases((JoinOperator)
nd, owi
>>>>>
>>>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:242:
>>>>>>      // be pushed down per getQualifiedAliases
>>>>>
>>>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:471:
>>>>>>  private Set<String> getQualifiedAliases(JoinOperator op,
>>>>>> RowResolver rr) {
>>>>>
>>>>>
>>>>>
>>>> -- Lefty
>>>>
>>>>
>>>> On Mon, Dec 9, 2013 at 2:12 PM, Harish Butani <hbutani@hortonworks.com>wrote:
>>>>
>>>>> Looks good.  Thanks for doing this.
>>>>>
>>>>> Minor point:
>>>>>
>>>>> *Rule 1:* During *QBJoinTree* construction in Plan Gen, the parse
>>>>> Join Condition logic applies this rule.
>>>>> *Rule 2:* During *JoinPPD* (Join Predicate Pushdown) the get
>>>>> Qualified Alias logic applies this rule.
>>>>>
>>>>> FYI 'parseJoinCondition' and 'getQualifiedAlias' are methods in the
>>>>> SemanticAnalyzer and JoinPPD classes respectively.
>>>>> Writing these as separate words maybe confusing. You are better judge
>>>>> of how to represent this(quoted/bold etc.)
>>>>>
>>>>> regards,
>>>>> Harish.
>>>>>
>>>>>
>>>>> On Dec 9, 2013, at 1:52 AM, Lefty Leverenz <leftyleverenz@gmail.com>
>>>>> wrote:
>>>>>
>>>>> The Outer Join Behavior
>>>>> wikidoc<
>>>>> https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior>is
>>>>>
>>>>> done, with links from the Design
>>>>> Docs <https://cwiki.apache.org/confluence/display/Hive/DesignDocs>
>>>>> page and
>>>>> the Joins doc<
>>>>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Joins#LanguageManualJoins-JoinOptimization
>>>>> >
>>>>> .
>>>>>
>>>>> Harish (or anyone else) would you please review the changes I made to
>>>>> the definition
>>>>> for "Null Supplying
>>>>> table"<
>>>>> https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior#OuterJoinBehavior-Definitions
>>>>> >
>>>>>
>>>>> ?
>>>>>
>>>>> -- Lefty
>>>>>
>>>>>
>>>>> On Mon, Dec 2, 2013 at 6:46 PM, Thejas Nair <thejas@hortonworks.com>
>>>>> wrote:
>>>>>
>>>>> :)
>>>>>
>>>>>
>>>>> On Mon, Dec 2, 2013 at 6:18 PM, Lefty Leverenz <
>>>>> leftyleverenz@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Easy as 3.14159....  (I can take a hint.)
>>>>>
>>>>> -- Lefty
>>>>>
>>>>>
>>>>> On Mon, Dec 2, 2013 at 5:34 PM, Thejas Nair <thejas@hortonworks.com>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> FYI, Harish has a written a very nice doc describing predicate push
>>>>> down rules for join. I have attached it to the design doc page. It
>>>>> will be very useful for anyone looking at joins.
>>>>>
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/download/attachments/27362075/OuterJoinBehavior.html
>>>>>
>>>>>
>>>>> (any help converting it to wiki format from html is welcome!).
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>
>>>>> entity to
>>>>>
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>
>>>>> reader
>>>>>
>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>
>>>>> that
>>>>>
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>>>
>>>>> immediately
>>>>>
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable
law.
>>>>> If the reader of this message is not the intended recipient, you are
hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited.
If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message