Return-Path: X-Original-To: apmail-pig-commits-archive@www.apache.org Delivered-To: apmail-pig-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9F2E910A7E for ; Fri, 21 Mar 2014 03:58:27 +0000 (UTC) Received: (qmail 59933 invoked by uid 500); 21 Mar 2014 03:58:27 -0000 Delivered-To: apmail-pig-commits-archive@pig.apache.org Received: (qmail 59819 invoked by uid 500); 21 Mar 2014 03:58:23 -0000 Mailing-List: contact commits-help@pig.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@pig.apache.org Delivered-To: mailing list commits@pig.apache.org Received: (qmail 59812 invoked by uid 99); 21 Mar 2014 03:58:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Mar 2014 03:58:21 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Mar 2014 03:58:19 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id A1CBE23888E4; Fri, 21 Mar 2014 03:57:56 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1579872 - in /pig/branches/branch-0.12: CHANGES.txt src/org/apache/pig/newplan/logical/rules/PushDownForEachFlatten.java test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java Date: Fri, 21 Mar 2014 03:57:56 -0000 To: commits@pig.apache.org From: daijy@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140321035756.A1CBE23888E4@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: daijy Date: Fri Mar 21 03:57:56 2014 New Revision: 1579872 URL: http://svn.apache.org/r1579872 Log: PIG-3826: Outer join with PushDownForEachFlatten generates wrong result Modified: pig/branches/branch-0.12/CHANGES.txt pig/branches/branch-0.12/src/org/apache/pig/newplan/logical/rules/PushDownForEachFlatten.java pig/branches/branch-0.12/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java Modified: pig/branches/branch-0.12/CHANGES.txt URL: http://svn.apache.org/viewvc/pig/branches/branch-0.12/CHANGES.txt?rev=1579872&r1=1579871&r2=1579872&view=diff ============================================================================== --- pig/branches/branch-0.12/CHANGES.txt (original) +++ pig/branches/branch-0.12/CHANGES.txt Fri Mar 21 03:57:56 2014 @@ -34,6 +34,8 @@ PIG-3480: TFile-based tmpfile compressio BUG FIXES +PIG-3826: Outer join with PushDownForEachFlatten generates wrong result (daijy) + PIG-3820: TestAvroStorage fail on some OS (daijy) PIG-3818: PIG-2499 is accidently reverted (daijy) Modified: pig/branches/branch-0.12/src/org/apache/pig/newplan/logical/rules/PushDownForEachFlatten.java URL: http://svn.apache.org/viewvc/pig/branches/branch-0.12/src/org/apache/pig/newplan/logical/rules/PushDownForEachFlatten.java?rev=1579872&r1=1579871&r2=1579872&view=diff ============================================================================== --- pig/branches/branch-0.12/src/org/apache/pig/newplan/logical/rules/PushDownForEachFlatten.java (original) +++ pig/branches/branch-0.12/src/org/apache/pig/newplan/logical/rules/PushDownForEachFlatten.java Fri Mar 21 03:57:56 2014 @@ -159,6 +159,11 @@ public class PushDownForEachFlatten exte for( int i = 0; i < preds.size(); i++ ) { Operator op = preds.get( i ); if( op == foreach ) { + // Don't optimize if the flattened side is outer side of an outer join + // See PIG-3826 + if (join.getInnerFlags()[i]==false) { + return false; + } Collection exprs = join.getJoinPlan( i ); for( LogicalExpressionPlan expr : exprs ) { List projs = getProjectExpressions( expr ); Modified: pig/branches/branch-0.12/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java URL: http://svn.apache.org/viewvc/pig/branches/branch-0.12/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java?rev=1579872&r1=1579871&r2=1579872&view=diff ============================================================================== --- pig/branches/branch-0.12/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java (original) +++ pig/branches/branch-0.12/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java Fri Mar 21 03:57:56 2014 @@ -1215,5 +1215,21 @@ public class TestNewPlanPushDownForeachF Assert.assertTrue( sort instanceof LOSort ); } + + @Test + // See PIG-3826 + public void testOuterJoin() throws Exception { + String query = "A = load 'A.txt' as (id:chararray, value:double);" + + "B = load 'B.txt' as (id:chararray, name:chararray);" + + "t1 = group A by id;" + + "t2 = foreach t1 { r1 = filter $1 by (value>1); r2 = limit r1 1; generate group as id, FLATTEN(r2.value) as value; }" + + "t3 = join B by id LEFT OUTER, t2 by id;" + + "store t3 into 'output';"; + LogicalPlan newLogicalPlan = migrateAndOptimizePlan( query ); + + Operator store = newLogicalPlan.getSinks().get( 0 ); + Operator join = newLogicalPlan.getPredecessors(store).get(0); + Assert.assertTrue( join instanceof LOJoin ); + } }