hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish Butani <hbut...@hortonworks.com>
Subject Re: doc on predicate pushdown in joins
Date Wed, 11 Dec 2013 15:59:02 GMT
getQualifiedAliases is a private method in JoinPPD.

Maybe we should remove the section on Hive Implementation here. It is in the Design doc; this
information only concerns developers.

regards,
Harish.


On Dec 11, 2013, at 3:05 AM, Lefty Leverenz <leftyleverenz@gmail.com> wrote:

> Happy to fix the sentence and the link.  I pointed out the name change just so you would
review it, so please don't apologize!
> 
> One more question:  why am I not finding getQualifiedAliases() in the SemanticAnalyzer
class?  It turns up in OpProcFactory.java with javadoc comments, but I can't find it anywhere
in the API docs -- not even in the index (Hive 0.12.0 API): 
> 
> getQMap() - Method in class org.apache.hadoop.hive.ql.QTestUtil
>  
> getQualifiedName() - Method in class org.apache.hadoop.hive.serde2.typeinfo.TypeInfo
> String representing the qualified type name.
> getQualifiers() - Method in class org.apache.hive.service.cli.thrift.TTypeQualifiers
>  
> getQualifiersSize() - Method in class org.apache.hive.service.cli.thrift.TTypeQualifiers
> 
> Most mysterious.
> 
> -- Lefty
> 
> 
> On Tue, Dec 10, 2013 at 2:35 PM, Harish Butani <hbutani@hortonworks.com> wrote:
> I can see why you would rename.
> 
> But this sentence is not correct:
> 'Hive enforces the predicate pushdown rules by these methods in the SemanticAnalyzer
and JoinPPD classes:'
> 
> It should be:
> Hive enforces the rules by these methods in the SemanticAnalyzer and JoinPPD classes:
> 
> (The implementation involves both predicate pushdown and analyzing join conditions)
> Sorry about this.
> 
> So the link should say 'Hive Outer Join Behavior'
> 
> regards,
> Harish.
> 
> 
> On Dec 10, 2013, at 2:01 PM, Lefty Leverenz <leftyleverenz@gmail.com> wrote:
> 
>> How's this?  Hive Implementation
>> 
>> Also, I moved the link on the Design Docs page from Proposed to Other.  (It's called
SQL Outer Join Predicate Pushdown Rules which doesn't match the title, but seems okay because
it's more descriptive.)
>> 
>> -- Lefty
>> 
>> 
>> On Tue, Dec 10, 2013 at 7:27 AM, Harish Butani <hbutani@hortonworks.com> wrote:
>> You are correct, it is plural.
>> 
>> regards,
>> Harish.
>> 
>> On Dec 10, 2013, at 4:03 AM, Lefty Leverenz <leftyleverenz@gmail.com> wrote:
>> 
>>> Okay, then monospace with "()" after the method name is a good way to show them:
 parseJoinCondition() and getQualifiedAlias() ... but I only found the latter pluralized,
instead of singular, so should it be getQualifiedAliases() or am I missing something?
>>> 
>>> trunk> grep -nr 'getQualifiedAlias' ./ql/src/java/* | grep -v 'svn'
>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:221:   * the comments
for getQualifiedAliases function.
>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:230:      Set<String>
aliases = getQualifiedAliases((JoinOperator) nd, owi
>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:242:        //
be pushed down per getQualifiedAliases
>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:471:    private
Set<String> getQualifiedAliases(JoinOperator op, RowResolver rr) {
>>> 
>>> 
>>> -- Lefty
>>> 
>>> 
>>> On Mon, Dec 9, 2013 at 2:12 PM, Harish Butani <hbutani@hortonworks.com>
wrote:
>>> Looks good.  Thanks for doing this.
>>> 
>>> Minor point:
>>> 
>>> Rule 1: During QBJoinTree construction in Plan Gen, the parse Join Condition
logic applies this rule.
>>> Rule 2: During JoinPPD (Join Predicate Pushdown) the get Qualified Alias logic
applies this rule.
>>> 
>>> FYI 'parseJoinCondition' and 'getQualifiedAlias' are methods in the SemanticAnalyzer
and JoinPPD classes respectively. 
>>> Writing these as separate words maybe confusing. You are better judge of how
to represent this(quoted/bold etc.)
>>> 
>>> regards,
>>> Harish.
>>> 
>>> 
>>> On Dec 9, 2013, at 1:52 AM, Lefty Leverenz <leftyleverenz@gmail.com> wrote:
>>> 
>>>> The Outer Join Behavior
>>>> wikidoc<https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior>is
>>>> 
>>>> done, with links from the Design
>>>> Docs <https://cwiki.apache.org/confluence/display/Hive/DesignDocs>
page and
>>>> the Joins doc<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Joins#LanguageManualJoins-JoinOptimization>
>>>> .
>>>> 
>>>> Harish (or anyone else) would you please review the changes I made to
>>>> the definition
>>>> for "Null Supplying
>>>> table"<https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior#OuterJoinBehavior-Definitions>
>>>> 
>>>> ?
>>>> 
>>>> -- Lefty
>>>> 
>>>> 
>>>> On Mon, Dec 2, 2013 at 6:46 PM, Thejas Nair <thejas@hortonworks.com>
wrote:
>>>> 
>>>>> :)
>>>>> 
>>>>> 
>>>>> On Mon, Dec 2, 2013 at 6:18 PM, Lefty Leverenz <leftyleverenz@gmail.com>
>>>>> wrote:
>>>>>> Easy as 3.14159....  (I can take a hint.)
>>>>>> 
>>>>>> -- Lefty
>>>>>> 
>>>>>> 
>>>>>> On Mon, Dec 2, 2013 at 5:34 PM, Thejas Nair <thejas@hortonworks.com>
>>>>> wrote:
>>>>>> 
>>>>>>> FYI, Harish has a written a very nice doc describing predicate
push
>>>>>>> down rules for join. I have attached it to the design doc page.
It
>>>>>>> will be very useful for anyone looking at joins.
>>>>>>> 
>>>>>>> 
>>>>> https://cwiki.apache.org/confluence/download/attachments/27362075/OuterJoinBehavior.html
>>>>>>> 
>>>>>>> (any help converting it to wiki format from html is welcome!).
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual
or
>>>>> entity to
>>>>>>> which it is addressed and may contain information that is confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If
the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
notified
>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure
or
>>>>>>> forwarding of this communication is strictly prohibited. If you
have
>>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity
to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified
that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>>> 
>>> 
>>> 
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to which
it is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.
>>> 
>> 
>> 
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which
it is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.
>> 
> 
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message