pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Min Zhou (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-1270) Push limit into loader
Date Wed, 30 Nov 2011 04:47:40 GMT

    [ https://issues.apache.org/jira/browse/PIG-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159819#comment-13159819
] 

Min Zhou commented on PIG-1270:
-------------------------------

We are using a modified version of 0.19.1. However, that internal version provide new MR API
and is compatible with both hadoop clients under the versions of 0.19.x and 0.20.2. Our version
doesn't change any logic of map phase from the community version, so this patch should improves
the latter as well.


That's a good attempt if we can address more cases like limit optimization on LOFilter.

                
> Push limit into loader
> ----------------------
>
>                 Key: PIG-1270
>                 URL: https://issues.apache.org/jira/browse/PIG-1270
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.7.0
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>         Attachments: PIG-1270-1.patch, PIG-1270-2.patch, PIG-1270-3.patch
>
>
> We can optimize limit operation by stopping early in PigRecordReader. In general, we
need a way to communicate between PigRecordReader and execution pipeline. POLimit could instruct
PigRecordReader that we have already had enough records and stop feeding more data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message