pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4662) New optimizer rule: filter nulls before inner joins
Date Fri, 26 May 2017 16:51:04 GMT

    [ https://issues.apache.org/jira/browse/PIG-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026494#comment-16026494

Rohini Palaniswamy commented on PIG-4662:

bq. I'd like to see a modular design rather than intermingle different concept together.
  In that case we can extend and add a POJoinLocalRearrange that handles join specific conditions
like this. It would not be mixing up POLocalRearrange then.

bq. I don't feel it is hard to find the join key in the logical optimizer and adding a filter
on it.
  Adding a extra filter operator for 3 lines of check will definitely impact performance when
we are dealing with billions of records. We recently had a user who added is null bincond
checks for lot of columns in his foreach which dealt with 10+billions of records and it took
extra 40+ minutes. Filter and foreach are what we are trying to optimize in PIG-3764 with
bytecode generation. As we are trying to improve performance everywhere and trying to save
milliseconds we should not be doing this unless it is a major or complicated change in which
case it will be cleaner to keep it separate.

> New optimizer rule: filter nulls before inner joins
> ---------------------------------------------------
>                 Key: PIG-4662
>                 URL: https://issues.apache.org/jira/browse/PIG-4662
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Ido Hadanny
>            Assignee: Satish Subhashrao Saley
>            Priority: Minor
>              Labels: Performance
>             Fix For: 0.18.0
> As stated in the docs, rewriting an inner join and filtering nulls from inputs can be
a big performance gain: http://pig.apache.org/docs/r0.14.0/perf.html#nulls
> We would like to add an optimizer rule which detects inner joins, and filters nulls in
all inputs:
> A = filter A by t is not null;
> B = filter B by x is not null;
> C = join A by t, B by x;
> see also: http://stackoverflow.com/questions/32088389/is-the-pig-optimizer-filtering-nulls-before-joining

This message was sent by Atlassian JIRA

View raw message