Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
Date: Tue, 17 Mar 2015 19:48:42 +0000 (UTC)
From: "William Watson (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Message-ID: <JIRA.12781576.1426184534000.121650.1426621722483@Atlassian.JIRA>
In-Reply-To: <JIRA.12781576.1426184534000@Atlassian.JIRA>
References: <JIRA.12781576.1426184534000@Atlassian.JIRA>
 <JIRA.12781576.1426184534811@arcas>
Subject: [jira] [Commented] (PIG-4458) Support UDFs in a FOREACH Before a
 Merge Join
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365899#comment-14365899 ] 

William Watson commented on PIG-4458:
-------------------------------------

Okay posted a new patch with a (too?) verbose comment when the merge join fails on one side.

> Support UDFs in a FOREACH Before a Merge Join
> ---------------------------------------------
>
>                 Key: PIG-4458
>                 URL: https://issues.apache.org/jira/browse/PIG-4458
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: William Watson
>         Attachments: PIG-4458.03.patch, remove_merge_join_udf_restriction.patch, remove_merge_join_udf_restriction.patch
>
>
> Right now, the MapSideMergeValidator outright rejects any foreach that has a UDF in it:
> {code}
> private boolean isAcceptableForEachOp(Operator lo) throws LogicalToPhysicalTranslatorException {
>         if (lo instanceof LOForEach) {
>             OperatorPlan innerPlan = ((LOForEach) lo).getInnerPlan();
>             validateMapSideMerge(innerPlan.getSinks(), innerPlan);
>             return !containsUDFs((LOForEach) lo);
>         } else {
>             return false;
>         }
>     }
> {code}
> There is a TODO for this later on in that same class (inside containsUDFs):
> {code}
> // TODO (dvryaboy): in the future we could relax this rule by tracing what fields
> // are being passed into the UDF, and only refusing if the UDF is working on the
> // join key. Transforms of other fields should be ok.
> {code}
> We should do the TODO and relax this requirement or just remove it altogether


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)