Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DDBBA100A4 for ; Thu, 12 Dec 2013 01:17:46 +0000 (UTC) Received: (qmail 90966 invoked by uid 500); 12 Dec 2013 01:17:46 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 90913 invoked by uid 500); 12 Dec 2013 01:17:46 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 90903 invoked by uid 99); 12 Dec 2013 01:17:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Dec 2013 01:17:46 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of leftyleverenz@gmail.com designates 209.85.214.51 as permitted sender) Received: from [209.85.214.51] (HELO mail-bk0-f51.google.com) (209.85.214.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Dec 2013 01:17:39 +0000 Received: by mail-bk0-f51.google.com with SMTP id 6so638137bkj.10 for ; Wed, 11 Dec 2013 17:17:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=aa/UUgnPrja+kw9TMTTDPAVTZXkI94qK/shniqs28Xo=; b=zpK70Zrz+4+WNuiXkxBfNE/JH0un/HAMIlBv8UQTb8xyktT2Her9DqYdfLfuLqwivd 2tUJbZ4J0sAqHoCc+s8UE14VEAYmQTNENfrjNYszFXTmxojUtUvhIDvC/xGOy1USIthC 5gtN8LanBXnc2coR0BEQLKst5exIzA5g/Nxpfd3FghTJzXxtr3KjdoQXBk2vibw6WUV2 Z2TeHhbO/YVWvwWBzYrdGGkWwI3LU5u9NyvGDqybc3F/f0b1kUfkzFNs4ENxJ8pFbw5b VHxkwr2JuuQxVsaG81HwC6EOModkpKZXUVXIRdvsS0T2acGDt1rHAPI2KxSblaH/R27E zrzw== MIME-Version: 1.0 X-Received: by 10.205.36.81 with SMTP id sz17mr2081623bkb.29.1386811038845; Wed, 11 Dec 2013 17:17:18 -0800 (PST) Received: by 10.205.90.66 with HTTP; Wed, 11 Dec 2013 17:17:18 -0800 (PST) In-Reply-To: <1461A627-5B52-47FB-B025-CE56A7FF6D6A@hortonworks.com> References: <1461A627-5B52-47FB-B025-CE56A7FF6D6A@hortonworks.com> Date: Wed, 11 Dec 2013 17:17:18 -0800 Message-ID: Subject: Re: doc on predicate pushdown in joins From: Lefty Leverenz To: Harish Butani Cc: dev@hive.apache.org Content-Type: multipart/alternative; boundary=bcaec52c677557784704ed4c1b4e X-Virus-Checked: Checked by ClamAV on apache.org --bcaec52c677557784704ed4c1b4e Content-Type: text/plain; charset=ISO-8859-1 > Maybe we should remove the section on Hive Implementation here. > It is in the Design doc; this information only concerns developers. But this is the Design doc (unless there's another one somewhere -- maybe attached to a JIRA ticket?) and it's in the Resources for Contributors part of the wiki, so it seems appropriate to me. I'll delete the implementation section if that's your preference. Here are the links again, with fixes: - Design Docs (bottom of list) - Predicate Pushdown Rules Speaking of JIRA tickets, is there one for this and should I add any version information? -- Lefty On Wed, Dec 11, 2013 at 7:59 AM, Harish Butani wrote: > getQualifiedAliases is a private method in JoinPPD. > > Maybe we should remove the section on Hive Implementation here. It is in > the Design doc; this information only concerns developers. > > regards, > Harish. > > > On Dec 11, 2013, at 3:05 AM, Lefty Leverenz > wrote: > > Happy to fix the sentence and the link. I pointed out the name change > just so you would review it, so please don't apologize! > > One more question: why am I not finding getQualifiedAliases() in the > SemanticAnalyzer class? It turns up in OpProcFactory.java with javadoc > comments, but I can't find it anywhere in the API docs -- not even in the > index (Hive 0.12.0 API ): > > *getQMap()* - > Method in class org.apache.hadoop.hive.ql.QTestUtil > *getQualifiedName()* - > Method in class org.apache.hadoop.hive.serde2.typeinfo.TypeInfo String > representing the qualified type name.*getQualifiers()* - > Method in class org.apache.hive.service.cli.thrift.TTypeQualifiers > *getQualifiersSize()* - > Method in class org.apache.hive.service.cli.thrift.TTypeQualifiers > Most mysterious. > > -- Lefty > > > On Tue, Dec 10, 2013 at 2:35 PM, Harish Butani wrote: > >> I can see why you would rename. >> >> But this sentence is not correct: >> 'Hive enforces the predicate pushdown rules by these methods in the >> SemanticAnalyzer and JoinPPD classes:' >> >> It should be: >> Hive enforces the rules by these methods in the SemanticAnalyzer and >> JoinPPD classes: >> >> (The implementation involves both predicate pushdown and analyzing join >> conditions) >> Sorry about this. >> >> So the link should say 'Hive Outer Join Behavior' >> >> regards, >> Harish. >> >> >> On Dec 10, 2013, at 2:01 PM, Lefty Leverenz >> wrote: >> >> How's this? Hive Implementation >> >> Also, I moved the link on the Design Docs pagefrom >> *Proposed* to *Other*. (It's called SQL Outer Join Predicate Pushdown >> Rules which >> doesn't match the title, but seems okay because it's more descriptive.) >> >> -- Lefty >> >> >> On Tue, Dec 10, 2013 at 7:27 AM, Harish Butani wrote: >> >>> You are correct, it is plural. >>> >>> regards, >>> Harish. >>> >>> On Dec 10, 2013, at 4:03 AM, Lefty Leverenz >>> wrote: >>> >>> Okay, then monospace with "()" after the method name is a good way to >>> show them: parseJoinCondition() and getQualifiedAlias() ... but I only >>> found the latter pluralized, instead of singular, so should it be >>> getQualifiedAliases() or am I missing something? >>> >>> trunk> *grep -nr 'getQualifiedAlias' ./ql/src/java/* | grep -v 'svn'* >>>> >>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:221: * >>>>> the comments for getQualifiedAliases function. >>>> >>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:230: >>>>> Set aliases = getQualifiedAliases((JoinOperator) nd, owi >>>> >>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:242: >>>>> // be pushed down per getQualifiedAliases >>>> >>>> ./ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java:471: >>>>> private Set getQualifiedAliases(JoinOperator op, RowResolver >>>>> rr) { >>>> >>>> >>>> >>> -- Lefty >>> >>> >>> On Mon, Dec 9, 2013 at 2:12 PM, Harish Butani wrote: >>> >>>> Looks good. Thanks for doing this. >>>> >>>> Minor point: >>>> >>>> *Rule 1:* During *QBJoinTree* construction in Plan Gen, the parse Join >>>> Condition logic applies this rule. >>>> *Rule 2:* During *JoinPPD* (Join Predicate Pushdown) the get Qualified >>>> Alias logic applies this rule. >>>> >>>> FYI 'parseJoinCondition' and 'getQualifiedAlias' are methods in the >>>> SemanticAnalyzer and JoinPPD classes respectively. >>>> Writing these as separate words maybe confusing. You are better judge >>>> of how to represent this(quoted/bold etc.) >>>> >>>> regards, >>>> Harish. >>>> >>>> >>>> On Dec 9, 2013, at 1:52 AM, Lefty Leverenz >>>> wrote: >>>> >>>> The Outer Join Behavior >>>> wikidoc< >>>> https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior>is >>>> >>>> done, with links from the Design >>>> Docs >>>> page and >>>> the Joins doc< >>>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Joins#LanguageManualJoins-JoinOptimization >>>> > >>>> . >>>> >>>> Harish (or anyone else) would you please review the changes I made to >>>> the definition >>>> for "Null Supplying >>>> table"< >>>> https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior#OuterJoinBehavior-Definitions >>>> > >>>> >>>> ? >>>> >>>> -- Lefty >>>> >>>> >>>> On Mon, Dec 2, 2013 at 6:46 PM, Thejas Nair >>>> wrote: >>>> >>>> :) >>>> >>>> >>>> On Mon, Dec 2, 2013 at 6:18 PM, Lefty Leverenz >>> > >>>> wrote: >>>> >>>> Easy as 3.14159.... (I can take a hint.) >>>> >>>> -- Lefty >>>> >>>> >>>> On Mon, Dec 2, 2013 at 5:34 PM, Thejas Nair >>>> >>>> wrote: >>>> >>>> >>>> FYI, Harish has a written a very nice doc describing predicate push >>>> down rules for join. I have attached it to the design doc page. It >>>> will be very useful for anyone looking at joins. >>>> >>>> >>>> >>>> https://cwiki.apache.org/confluence/download/attachments/27362075/OuterJoinBehavior.html >>>> >>>> >>>> (any help converting it to wiki format from html is welcome!). >>>> >>>> -- >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> >>>> entity to >>>> >>>> which it is addressed and may contain information that is confidential, >>>> privileged and exempt from disclosure under applicable law. If the >>>> >>>> reader >>>> >>>> of this message is not the intended recipient, you are hereby notified >>>> >>>> that >>>> >>>> any printing, copying, dissemination, distribution, disclosure or >>>> forwarding of this communication is strictly prohibited. If you have >>>> received this communication in error, please contact the sender >>>> >>>> immediately >>>> >>>> and delete it from your system. Thank You. >>>> >>>> >>>> -- >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to >>>> which it is addressed and may contain information that is confidential, >>>> privileged and exempt from disclosure under applicable law. If the >>>> reader >>>> of this message is not the intended recipient, you are hereby notified >>>> that >>>> any printing, copying, dissemination, distribution, disclosure or >>>> forwarding of this communication is strictly prohibited. If you have >>>> received this communication in error, please contact the sender >>>> immediately >>>> and delete it from your system. Thank You. >>>> >>>> >>>> >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to which it is addressed and may contain information that is >>>> confidential, privileged and exempt from disclosure under applicable law. >>>> If the reader of this message is not the intended recipient, you are hereby >>>> notified that any printing, copying, dissemination, distribution, >>>> disclosure or forwarding of this communication is strictly prohibited. If >>>> you have received this communication in error, please contact the sender >>>> immediately and delete it from your system. Thank You. >>>> >>> >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >>> >> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > --bcaec52c677557784704ed4c1b4e--